Inference
Eldric runs large language models (LLMs) itself, with no Ollama, vLLM or other runtime installed. A node serves open-weight models in GGUF form, for example Qwen, Llama 3, Mistral, Gemma 4, gpt-oss and Hermes-style models, whose tool-call formats Eldric recognises. Eldric ships no models: an administrator adds the files. The same API also reaches cloud LLMs through your own provider accounts. This section says what is served, where it runs and what each kind of model can do in the shipped release.
Where models run
Local models run in the inference module, inside the Eldric service on port 8880 of every node that has the inference role, which a node has by default. The engine is llama.cpp, built into Eldric, and it reads GGUF files.
- CPU package (
eldric-aios): runs models on the processor. On x86_64 it needs AVX2. - GPU package (
eldric-aios-cuda): runs models on an NVIDIA card and falls back to the processor without one. - A build on a Mac uses Metal. We do not offer a macOS server package; the macOS download is the app and the command-line client.
GET /api/v1/inference/engine on a node reports what it has: engine, cuda, metal, native_ready and, when the engine cannot run, native_unavailable_reason, plus the model directory and the models loaded. Up to 5.0.181 the GPU package also carried a separate inference daemon, eldric-inferenced; it is no longer installed.
Cloud models are served by the cloud module on a node that has it, with the provider keys an administrator entered there (Cloud provider keys). Which node answers which model, and what happens when one is down, is on Models across the cluster.
The API
| Route | What it does |
|---|---|
POST /v1/chat/completions | Chat, OpenAI-compatible. With "stream": true the answer arrives as server-sent events. |
POST /v1/embeddings | Text to vectors (Embeddings). |
GET /v1/models | Every model the cluster offers, local and cloud. |
The same routes answer for local and cloud models. For a local model the request is handed to the inference module inside the service; for a cloud model the cloud module forwards it to the provider. Clients reach them through the edge with an API key (API reference).
What the model list tells you
A local model's row in /v1/models carries:
| Field | Meaning |
|---|---|
id | The file name without .gguf. |
loaded, serving_state | Whether it is in memory now; ok when loaded, unknown otherwise. |
kind, supports_chat | What the model is for: chat, embedding or rerank, and whether it can chat (from 5.0.196). |
context_length | The context it gets on this node (Context length). |
tool_format, tool_call_supported | The tool format its template teaches, or none (Tool calling). |
vision_supported | Whether a projector is paired with it (Vision). |
placement, vram_bytes | For a loaded model: gpu, moe_experts_cpu:<n> or cpu, and the planned graphics memory (Mixture-of-experts models). |
pinned | Never unloaded to make room. |
The row does not say whether a model is for chat or for embeddings; see Embeddings.
The pages in this section
- Model types: chat, tool calling, vision, embeddings, the small workloads, forecasting, world models and the router's tool index.
- Local models: adding GGUF files, context length, memory, settings and errors.
- Models across the cluster: the catalogue, how a node is chosen, a node going down.
- Mixture-of-experts models: expert offload and context on a node with several models.
- EMM and EXMM: Eldric's memory and model container formats.
- Splitting a model across machines: not available yet, and what we measured.
What Eldric knows and remembers, as opposed to which model answers, is under Knowledge and memory.