Cloud provider keys

Guide · Administrators · Applies to 5.0.173

Cloud models (OpenAI, Anthropic, HuggingFace and other providers) need a provider API key. This page describes how Eldric stores that key today, why a model can show up in the model list and still not answer, and how to set keys up so every request is served. Changes that are only planned are marked as such at the end.

How keys work today

• You add a cloud backend in the admin console (cloud dashboard) or with POST /api/v1/cloud/backends. Only administrators can do this.
• The key is stored encrypted, on the node that received the request, and only there.
• The other nodes of the cluster learn about the backend and its model list, but never receive the key. This is deliberate: the key is not shared with other nodes.
• The API never returns a key. It reports only key_present: true or false.

So a cloud model appears in the model list of the whole cluster, but only the node that holds the key can serve it.

Listed, but not answering

If a request for a cloud model reaches a node that does not hold the key, the answer is model_not_ready, usually within a fraction of a second and without contacting the provider. That speed is the clue: the provider was never asked.

It happens most often when the key was entered on a different node than the one that runs the cloud role, for example on a controller, or after the cloud role moved to another node.

Recommended setup

Use one of these two ways, on the node that runs the cloud role:

1. Enter the key in the admin console while connected to that node, or send POST /api/v1/cloud/backends to that node.
2. Keep the key in that node's environment file and register the backend with api_key_env set to the variable's name, instead of sending the key itself. The key then never enters Eldric's backend store. After changing a key in the environment file, restart the Eldric service on that node, because the environment is read at start. Backup copies of the environment file still contain the old key; delete them.

Several cloud nodes: repeat the setup on each of them. A key entered once applies to one node.

A key without a backend: GET /api/v1/cloud/backends/unconfigured-keys (from 5.0.173, administrators only) lists provider keys found in a node's environment that no backend uses yet. It names the variable, never its value. Ask the node you are checking; each node reports its own environment.

Check a backend

1. GET /api/v1/cloud/backends on each cloud node: the node that holds the key reports key_present: true. Entries copied from other nodes are named id@host and never carry a key.
2. Send a short test request (max_tokens: 1) once to the node holding the key and once through your normal entry point. If the two answers differ, the problem is routing, not the provider.
3. An error from the provider itself (for example missing credit, or a model the provider does not offer) proves the key reached the provider. A model_not_ready from Eldric means it did not.

Whose key is used

Today there are no per-tenant or per-user provider keys. Every user who can use a cloud backend uses the key the administrator entered, and the usage is billed to that provider account.

Planned

• A cluster-wide key store: enter a key once, and it is valid on every node.
• Keys per tenant as well as per cluster, also for single-company installations, so that a department or project can use its own provider account.
• Optional keys per user.