Models across the cluster

Guide · Administrators · Applies to 5.0.196

In a cluster, each node serves the models it holds, and every node can reach every model. This page explains how a node learns what the others serve, how the node that answers is chosen, and what happens when a node stops answering. Each node always serves a model whole. Spreading one model over several nodes is a different thing and is not available (Splitting a model across machines).

One catalogue for the cluster

Every node reports to a controller every 30 seconds. The controller answers with a module map: for each node, its address and which modules it runs. A node keeps the latest map, also on disk (<data>/cluster/module-map.json), so it knows its peers again after a restart before the first answer arrives. No list of peers has to be written into a configuration file.

From that map, each node asks the nodes running the inference or cloud module which models they serve, and merges the answers with its own into one list: GET /v1/models. The list is refreshed in the background every 30 seconds. When two nodes offer the same model, it appears once.

Which node answers

For a chat request, the node that handles it first looks at itself: if its own inference module serves the model, it answers. Otherwise it picks among the nodes that serve the model. It leaves out nodes whose row says they cannot serve it and nodes that recently failed to answer, prefers nodes whose model is known to work, and then applies the routing strategy.

Strategy	Picks
load_based (default)	The node with the fewest requests in progress, then the shortest recent answer time.
round_robin	Each node in turn.
least_connections	The fewest requests in progress.
least_latency	The shortest recent answer time.
random	Any of them.

An administrator changes it with PUT /api/v1/router/strategy. From 5.0.196 the choice is stored on the router node and kept across restarts; if it cannot be stored, the PUT answers 500 persist_failed and nothing changes. A strategy not in the table is refused with 400 invalid_strategy. Up to 5.0.195 the choice was held in memory only.

When a node stops answering

• If the node a chat is forwarded to cannot be reached at all (connection refused, name not resolved, or no connection within 10 seconds), the chat is sent once to another node that lists the same model, as long as nothing has been sent to the client yet. A node that accepted the connection is never asked twice, whatever it answers: it may already have run tools. This applies to /v1/chat/completions, streaming or not, and to the agent's chats. Up to 5.0.195 such a request failed with upstream_unreachable, after up to 5 minutes, and was not sent to another node.
• The unreachable node is then skipped for 30 seconds, so the next requests go to the other nodes that serve the model. If every node that serves it is marked, they are tried anyway.
• A node that is down does not answer the catalogue's questions either, so its models leave the list at the next refresh, within about 30 seconds. They come back when it answers again.
• GET /api/v1/cloud/peers (administrators, optionally ?model=) lists the nodes this node forwards each model to, with serving_state, down_for_s and in_flight.
• A node that answered with an error, even a server error, is not skipped. Its answer goes to the client unchanged.

So a model that two nodes serve stays available when one of them goes down; a model that only the failed node had is gone from the list until the node is back. The controller shows how long ago each node last reported: last_seen_age_s in the module map, and marks a node stale after 90 seconds and offline after 5 minutes without a report (the defaults).

Cloud models

Cloud models are served by the cloud module. From 5.0.196 the provider keys are kept in the controllers' credential store, and the node that runs the module reads them from there (Cloud provider keys). A request for a cloud model on another node is forwarded to that node's cloud module, which calls the provider. A forwarded request is never forwarded a second time.

The edge

The edge is the entry point for clients. It checks the caller and passes chat requests to a router: the one configured in router_url, or any node in the module map that runs the router. If the first router cannot be reached it tries a second. It only retries when the connection failed, never after the router may already have taken the request, so a retry cannot run a request twice. Addresses of internal nodes are removed from what the edge sends back.

The edge serves no model of its own. On a single node that also runs the cloud or inference module and has no router, it hands the request to those modules directly. With none of them it answers 503 no_chat_upstream.

Dedicated inference nodes

A node loads only the modules of its roles. Without the inference role it does not load the inference module, and it sends requests for local models to a node that has one. Running inference on its own nodes keeps a model that fills a card, or a node that fails under load, away from the controller, the data store and the edge. When such a node goes down, the steps in When a node stops answering apply.

From 5.0.196 the roles are set once for the whole cluster, in the admin console under Cluster setup or with GET|PUT /api/v1/cluster/setup: a controller tier that every controller runs, and a role list per node. A node follows its list with its next heartbeat; a module that was not running starts at once, and one switched off stops and is listed as stopped until the node restarts. On a node the setup names, other ways of switching a module are refused with 409 managed_by_cluster_setup. A node the setup does not name keeps its own roles, and a node with no roles set runs controller, inference, router, edge and data.

Traffic between nodes

Nodes call each other over plain HTTP on port 8880 unless an administrator gives a node an https:// address to advertise. Keep the nodes on a network you control, and keep port 8880 closed to the outside; clients come in through the edge.