Splitting a model across machines
Eldric cannot yet run one model across several machines. Every model is served whole by one node: on its graphics card, partly in system memory, or on the processor. This page says what you can do today with a model too large for any one card, and what we measured about splitting one.
What to do today
- Run the model on a node whose card and system memory together hold it. A model that does not fit the card loads partly on the processor, and a Mixture-of-Experts model keeps its layers on the card and moves only expert weights out (Mixture-of-experts models).
- Use a smaller quantisation of the same model.
- Serve the model from a cloud provider through the cloud module (Cloud provider keys).
Several nodes can each serve their own models to the whole cluster; that is described on Models across the cluster, and it is a different thing from splitting one model.
What we measured
This is a measurement, not a feature. On 2026-10-30 RESEARCH split one model by layers across two of our nodes with llama.cpp's own tools, outside Eldric, to find out whether such a split is fast enough to chat with.
| Run | Model | Where | Prompt, tokens/s | Answer, tokens/s |
|---|---|---|---|---|
| control | Qwen2.5-7B Q4_K_M | one node, 12 GB card | 7001 | 100.0 |
| split by layers | same | two nodes, half the layers each | 1376 | 81.6 |
| split inside each layer | same | two nodes | 67.6 | 6.9 |
| split by layers | Qwen2.5-14B Q8_0, 15.7 GB | two nodes, 12 GB and 8 GB cards | 914 | 27.8 |
- The 14B model fits on neither card alone. Split by layers it answered at 27.8 tokens per second, and the first token of an answer to a 512-token prompt came after 0.56 s (2.1 s for 2048 tokens).
- Splitting costs most when reading the prompt: the 7B model read prompts at a fifth of its single-node speed and answered at 82 % of it.
- The nodes were joined by 1 Gbit. The 14B split used about 15 Mbit/s of it; what limits it is the number of round trips, not the bandwidth.
- Splitting inside every layer instead of between layers was 12 times slower when answering. Over 1 Gbit it is not usable.
What the measurement does not say
It says that a 14B model split over two consumer cards on 1 Gbit was fast enough to chat with, on that pair of machines, with that build. It says nothing about more than two nodes or about several requests at once. And it does not mean Eldric can do it: no Eldric release can split a model, and we do not give a date. The upstream project calls the transport these tools use insecure, so for the measurement it was reachable only from the two test machines and was removed afterwards.