Local models

Guide · Administrators · Applies to 5.0.196

Eldric can serve large language models (LLMs) itself, from GGUF files on your own nodes, with no Ollama, vLLM or other runtime installed. Eldric ships no models: an administrator puts the files on the node. This page explains where the files go, how a model gets into the model list, how much context it gets and what happens when it does not fit.

Which node serves them

Local models are served by the inference role, which a node has by default. It runs inside the Eldric service on port 8880; there is no separate inference daemon any more (eldric-inferenced left the GPU package in 5.0.182).

• CPU package (eldric-aios): runs GGUF models on the processor. On x86_64 the processor needs AVX2; without it local models are refused and the rest of Eldric keeps running. The reason shows as native_unavailable_reason on GET /api/v1/inference/engine.
• GPU package (eldric-aios-cuda): replaces the CPU package on a node with an NVIDIA card and driver, and runs models on the card. It falls back to the processor when no card is present.

Adding a model

Put the .gguf file into the node's model directory. On a packaged node that is /data/eldric/models; ELDRIC_INFERENCE_MODEL_DIR in /etc/eldric/eldric-aios.env sets another one. Only files directly in the directory are read, not subfolders.

• The model's name is the file name without .gguf: qwen3-8b.gguf appears as qwen3-8b.
• From 5.0.191 the directory is checked every 30 seconds, so a file you add or remove by hand appears in or leaves the model list without a restart. ELDRIC_INFERENCE_RESCAN_SECONDS changes the interval, and 0 switches the check off. (The service may log that this variable "has no effect"; that message is wrong, it does.) An administrator can also rescan at once with POST /api/v1/models/rescan.
• Files that cannot be loaded are left out of the list and named in the log, once per scan, as model file <name> not listed: <reason>: for example an empty file, an unreadable one, or one that is not a GGUF model.
• A vision model's projector file is paired with its model by name: call it <model>.mmproj.gguf or mmproj-<model>.gguf.

The model list

Every node lists its own local models and those of the other nodes in the cluster, so a user can choose any of them from any entry point. When several nodes offer the same model, a node prefers its own copy. /v1/models shows, for each local model, whether it is loaded and its context_length; a loaded model also states its placement (gpu, moe_experts_cpu:<n> or cpu) and vram_bytes, the graphics memory it is planned to use (from 5.0.193; a plan, so it can read a few hundred MiB more than nvidia-smi). GET /api/v1/cluster/models-by-node on a controller shows which node holds which model.

A model behind a local server such as Ollama that cannot answer chat (for example an embedding model) is marked no_chat_backend and served for embeddings only (from 5.0.188). From 5.0.196 each row also says what the model is for: kind (chat, embedding or rerank) and supports_chat, read from the GGUF file itself, and the chat's model picker offers only models that can chat; see Embeddings. What every field of a row means is on the Inference overview.

Context length

From 5.0.190 a model gets the context it was trained for, as far as the node's memory allows after the model itself is loaded. Eldric keeps one long conversation slot per model and gives further parallel conversations 8192 tokens each; no slot gets fewer than 2048. When the node cannot tell how much memory is free, the long slot is capped at 32768 tokens. An administrator can set a lower cap with ctx_length (see Settings). From 5.0.194 a node with several chat models keeps room free for the smallest of the others; see Context on a node with several models.

A conversation longer than the context is refused with 400 context_length_exceeded; the message gives the prompt's size and the model's context. Shorten the conversation or choose a model with a longer context.

Loading and memory

• A model is loaded when the first request for it arrives, unless an administrator preloads it (preload_models). It stays loaded until it is unloaded or the service restarts.
• Before loading, Eldric checks that the model fits. One that fits system memory but not the graphics card loads partly on the processor. One that fits neither is refused. From 5.0.193 the check counts the memory the operating system can free (on Linux MemAvailable), not only unused memory.
• From 5.0.193 a Mixture-of-Experts model that does not fit the graphics card keeps all its layers on the card and moves the expert weights of as few layers as needed into system memory (cpu_moe_layers). It answers more slowly than a model that fits whole, but faster than on the processor alone. Measured speeds are on Mixture-of-experts models.
• From 5.0.193 a graphics-card node unloads the least recently used models when a new one does not fit (evict_on_memory_pressure). Pinned models and models answering a request are never unloaded. In 5.0.193 the room a new model needs is estimated from its file size plus 10 %, not from all the graphics memory it will use. Tested on a card with 12 GB: a loaded 3B model was unloaded so that an 8B model could load and answer; with the setting off the 8B request was refused with 503 model_load_failed and nothing was unloaded.

Settings

GET|PUT /api/v1/inference/config (administrators) reads and changes the settings of the node you ask. The answer to a PUT says which values took effect and which need a restart.

Setting	Default	Meaning
gpu_layers	999 (all)	How many layers go on the graphics card; 0 keeps the model on the processor. Applies at the next load.
cpu_moe_layers	-1 (automatic)	Mixture-of-Experts models only: how many leading layers keep their expert weights in system memory. -1 chooses the fewest that fit, 0 never moves experts. From 5.0.193; applies at the next load.
evict_on_memory_pressure	auto	Unload idle models when a new one does not fit: auto (on a node with a graphics card, off otherwise), on or off. From 5.0.193.
pinned_models	none	Models that are never unloaded to make room. From 5.0.193.
ctx_length	0 (automatic)	A cap on the context. Applies at the next load.
preload_models	none	Models to load at start. Takes effect at the next start.
max_contexts	4	Parallel conversations per model. Takes effect at the next start.

Ollama and other local servers

From 5.0.187 Eldric no longer connects to a local Ollama by default. If chat models were served through a local Ollama, they stop answering after the update. Either put their GGUF files into the model directory, so Eldric serves them itself, or set ELDRIC_INFERENCE_OLLAMA_URL to the Ollama address in the node's environment file and restart. GET /api/v1/inference/local-backends (administrators) shows, for each local server, which of its models can chat and which cannot.

If memories were stored with an embedding model that only Ollama served, rebuild them with the new embedding model before switching Ollama off: POST /api/v1/memory/rebuild (administrators) runs as a dry run first.

Errors you may see

Answer	Meaning
400 context_length_exceeded	The conversation is longer than the model's context.
503 model_load_failed	GPU package: the model could not be loaded. The detail names the free graphics memory, the model's size and the models already loaded. Unload a model or choose another; the request can be retried.
502 native_dispatch_failed	CPU package: the model could not be loaded or did not answer; the reason is in detail.

The model-health check that hides a model after repeated failures (from 5.0.185) applies to models behind another server, such as a cloud provider or Ollama, not to models Eldric serves itself.