Multi-step orchestration across providers
A combo is an ordered pipeline of upstream provider calls. Each step can target a specific credential, a provider (any credential), or a routing strategy. The five combo types cover the common patterns: graceful fallback, racing, refinement chains, and per-step routing. The engine lives at src/lib/server/combos/engine.ts with one executor per type under src/lib/server/combos/executors/.
Combo shape
A combo definition has a type (one of the five below), an array ofsteps, an enabled flag, and org-scoping metadata. Each step has anid and one of credentialId, providerId, orroutingStrategy to identify which credential to use, plus optionalmodel and params overrides and a fallback condition.
interface ComboStep {
id: string;
credentialId?: string; // specific credential id
providerId?: string; // any credential of this provider
routingStrategy?: string; // pick a credential via this strategy
model?: string; // model override (sent upstream)
params?: Record<string, unknown>; // override temperature, max_tokens, etc.
condition?: {
onStatus?: number[]; // fallback trigger (statuses)
onLatencyAboveMs?: number;
onTokensAbove?: number;
};
mergeStrategy?: 'first' | 'concat' | 'best-of-n';
}The 5 combo types
fallbackFallbackTry step 1. On failure or condition match, fall to step 2; on a further failure, fall to step 3; and so on. The first successful step wins.
parallelParallelSend to all steps concurrently; return the first successful response. For streaming, return the first stream that produces a chunk. Abort every other step via AbortController.
speculativeSpeculativeA specialization of parallel for low-latency racing. Send to multiple steps and return the first streaming chunk that arrives. Distinct combo type so callers can mark intent.
chainChainPipe step 1's output into step 2's input. For chat, step 1's assistant text is appended as a new user message and step 2 refines it. Returns the final step's response.
routingRoutingLike fallback, but each step uses its own routingStrategy to pick a credential from the pool. The first step that succeeds is returned.
Wiring a combo to a routing rule
A combo by itself does nothing until a routing rule references it. Use thecombo strategy with a comboId:
{
"name": "Use the chat-with-fallback combo",
"enabled": true,
"priority": 5,
"match": {"endpoint": "/v1/chat/completions"},
"strategy": "combo",
"strategyConfig": {"comboId": "combo_fallback_chat"},
"credentialPool": []
}When the router matches this rule, it loads the combo definition from Lightbase, builds aComboExecutionContext with the org's enabled credentials, and delegates to the matching executor. Each step records its own usage row with the combo id.
Per-step params override
The params field on a step is merged into the request body before the upstream call. Use it to override temperature, max_tokens, top_p, or any provider-specific field on a per-step basis.
{
"id": "s2",
"credentialId": "cred_openai",
"model": "gpt-4o",
"params": {"temperature": 0.7, "max_tokens": 1024, "top_p": 0.95}
}Stream semantics
- fallback: if the request is streaming, only the winning step's stream is piped to the client. Failed steps do not produce visible output.
- parallel and speculative: the first upstream to produce a streaming chunk wins. Loser streams are aborted; their underlying
fetchis cancelled viaAbortController. - chain: intermediate steps run non-streaming (the engine needs the full text to feed the next step). Only the final step honours the client's
stream: trueflag. - routing: same as fallback — the winning step's stream is piped to the client.
Audit and usage
Every step in a combo produces a ComboStepResult (stepId, credentialId, providerId, status, latencyMs, success, error). The engine returns aComboExecutionResult containing the winning credential id, the per-step results, the final response or stream, and the usage from the winning step. Each step writes a usage_records row tagged with the combo id.