Mixture-of-experts models

Guide · Administrators · Applies to 5.0.194

A Mixture-of-Experts (MoE) model has many expert blocks per layer but uses only a few of them for each token. That makes it large on disk and fast to run, if the part it uses most stays on the graphics card. This page says what Eldric does with such a model beyond what llama.cpp itself offers, and how it shares a card between several models.

What Eldric adds

llama.cpp can already keep the expert weights of chosen layers in system memory while everything else runs on the card. You have to say how many layers. Eldric, from 5.0.193, works that number out at load time and acts on it:

• It recognises an MoE model from the expert count the GGUF declares, and adds up the size of each layer's expert weights from the file.
• It picks the fewest leading layers whose experts must leave the card for the rest, plus room for the conversation memory, to fit. A model that fits whole stays whole.
• If even that does not fit and unloading is allowed, it unloads idle models first (see Loading and memory), then plans again.
• It reports the result in the model's row in /v1/models: placement is gpu, moe_experts_cpu:<n> or cpu, and vram_bytes is the planned graphics memory for the weights.

Everything else, including which experts a token uses, is llama.cpp's own MoE support. Eldric does not spread a model over several nodes; see Splitting a model across machines.

How the split is chosen

The room on the card is its free memory, minus the conversation memory the model will need, minus a margin of a tenth of the free memory and 256 MiB per parallel conversation. For the conversation memory the plan assumes one long conversation of the model's trained context, at most 16,384 tokens (or the administrator's ctx_length), and 8192 tokens for each further one. Eldric then moves the experts of layer 0, then layers 0 and 1, and so on, until the weights left on the card fit that room.

The plan errs towards not loading: in both measured runs it estimated the card's share of the weights about 600 MiB higher than llama.cpp then used. Treat vram_bytes as a plan, not a reading.

Measured

gpt-oss-20b (MXFP4, 12.1 GB file), one chat each, on two Eldric nodes with an NVIDIA card. The 8 GB run used a 5.0.193 development build that is part of the release; the 12 GB run used a 5.0.194 build.

Card	Setting	Placement	Answer speed
RTX 2080, 8 GB	automatic	moe_experts_cpu:13	9.8 tokens/s
RTX 2080, 8 GB	experts never moved (0)	out of memory, refused	none
RTX 2080, 8 GB	processor only	cpu	5.1 tokens/s
RTX 4070 Ti, 12 GB	automatic	moe_experts_cpu:4	15.6 tokens/s
RTX 4070 Ti, 12 GB	experts never moved (0)	gpu (fits whole)	21.2 tokens/s
RTX 4070 Ti, 12 GB, another model loaded, unloading off	automatic	moe_experts_cpu:15	11.6 tokens/s
RTX 4070 Ti, 12 GB	processor only	cpu	5.9 tokens/s

So on a small card the model answers about twice as fast as on the processor alone, and on a card it fits whole it is fastest with nothing moved. In these runs gpt-oss-20b answered in text and did not call the tool it was offered.

Settings

Set with PUT /api/v1/inference/config (administrators); both apply at the next load.

Setting	Default	Meaning
cpu_moe_layers	-1	-1 chooses the fewest layers that fit, 0 never moves experts, a number moves exactly that many leading layers whether or not it fits.
gpu_layers	999	0 keeps the whole model on the processor; the MoE plan is then not used.

Context on a node with several models

How much context a model gets is set out on Local models. From 5.0.194 a node with more than one chat model in its model directory keeps room free for the smallest of the others: its weights plus 10 %, plus the memory of one 2048-token conversation. The idea is that switching between two models in a chat should not push the first one out. A node with one chat model plans as before, and an administrator's ctx_length turns the reserve off. Embedding models and vision projectors are not counted. If the reserve would leave the model less than 8192 tokens, it is not taken, and unloading decides instead.

Measured on 2026-10-29 on a node with a 12 GB card and 5.0.194-1: llama3.1-8b got a long conversation of 8960 tokens (plus three of 8192), with 1.52 GB kept for the smallest other chat model, a 1.2B router model. On 5.0.193 the same node gave it 19,200 tokens. The row in /v1/models shows the 8960; the reason is in the service log, as "limited by multi_model_reserve".