5 combo types

Multi-step orchestration across providers

A combo is an ordered pipeline of upstream provider calls. Each step can target a specific credential, a provider (any credential), or a routing strategy. The five combo types cover the common patterns: graceful fallback, racing, refinement chains, and per-step routing. The engine lives at src/lib/server/combos/engine.ts with one executor per type under src/lib/server/combos/executors/.

Combo shape

A combo definition has a type (one of the five below), an array ofsteps, an enabled flag, and org-scoping metadata. Each step has anid and one of credentialId, providerId, orroutingStrategy to identify which credential to use, plus optionalmodel and params overrides and a fallback condition.

interface ComboStep {
  id: string;
  credentialId?: string;       // specific credential id
  providerId?: string;          // any credential of this provider
  routingStrategy?: string;     // pick a credential via this strategy
  model?: string;               // model override (sent upstream)
  params?: Record<string, unknown>;  // override temperature, max_tokens, etc.
  condition?: {
    onStatus?: number[];         // fallback trigger (statuses)
    onLatencyAboveMs?: number;
    onTokensAbove?: number;
  };
  mergeStrategy?: 'first' | 'concat' | 'best-of-n';
}

The 5 combo types

1fallbackFallback

Try step 1. On failure or condition match, fall to step 2; on a further failure, fall to step 3; and so on. The first successful step wins.

When to use
You have a preferred provider but want a graceful degradation path. Use conditions to fall over even on success if the response is too slow or too expensive.
Example
{
  "id": "combo_fallback_chat",
  "orgId": "org_abc",
  "name": "Chat with fallback",
  "type": "fallback",
  "enabled": true,
  "steps": [
    {
      "id": "s1",
      "credentialId": "cred_openai_primary",
      "model": "gpt-4o-mini",
      "condition": {"onStatus": [429, 500, 502, 503, 504], "onLatencyAboveMs": 5000}
    },
    {
      "id": "s2",
      "providerId": "openrouter",
      "model": "openai/gpt-4o-mini"
    },
    {
      "id": "s3",
      "providerId": "duckduckgo"
    }
  ]
}
3-step example
{
  "type": "fallback",
  "steps": [
    {"id": "s1", "credentialId": "cred_openai_primary", "condition": {"onStatus": [429, 500, 502, 503, 504]}},
    {"id": "s2", "providerId": "openrouter", "model": "openai/gpt-4o-mini"},
    {"id": "s3", "providerId": "duckduckgo"}
  ]
}
Behavior
Step 1 runs first. If it returns a status in onStatus, takes longer than onLatencyAboveMs, or fails entirely, the engine advances to step 2 and so on. The first step that succeeds (and does not trigger its own condition) is returned. If every step fails, the last error is returned with status 502 and a combo.fallback.exhausted error code.
2parallelParallel

Send to all steps concurrently; return the first successful response. For streaming, return the first stream that produces a chunk. Abort every other step via AbortController.

When to use
You want the fastest possible response and are willing to pay for parallel upstream calls. Best for non-streaming workloads or streaming workloads where the first upstream to start talking wins.
Example
{
  "id": "combo_parallel_chat",
  "orgId": "org_abc",
  "name": "Race 3 chat providers",
  "type": "parallel",
  "enabled": true,
  "steps": [
    {"id": "s1", "credentialId": "cred_openrouter", "model": "openai/gpt-4o-mini"},
    {"id": "s2", "credentialId": "cred_groq", "model": "llama-3.3-70b-versatile"},
    {"id": "s3", "credentialId": "cred_cerebras", "model": "llama-3.3-70b"}
  ]
}
3-step example
{
  "type": "parallel",
  "steps": [
    {"id": "s1", "credentialId": "cred_openrouter", "model": "openai/gpt-4o-mini"},
    {"id": "s2", "credentialId": "cred_groq", "model": "llama-3.3-70b-versatile"},
    {"id": "s3", "credentialId": "cred_cerebras", "model": "llama-3.3-70b"}
  ]
}
Behavior
All steps start at the same time. For non-streaming, the first to resolve with HTTP 2xx wins; losers are aborted via their AbortController. For streaming, each upstream stream is peeked for its first chunk; the first upstream to produce a chunk wins, and losers are aborted (their fetch is actually cancelled). Every step's outcome (success / aborted / failed) is recorded for audit.
3speculativeSpeculative

A specialization of parallel for low-latency racing. Send to multiple steps and return the first streaming chunk that arrives. Distinct combo type so callers can mark intent.

When to use
You have several providers that all serve the same model and you want the fastest first-token. The losers are cancelled as soon as the first chunk arrives.
Example
{
  "id": "combo_speculative_stream",
  "orgId": "org_abc",
  "name": "Speculative first-token race",
  "type": "speculative",
  "enabled": true,
  "steps": [
    {"id": "s1", "credentialId": "cred_groq", "model": "llama-3.3-70b-versatile"},
    {"id": "s2", "credentialId": "cred_cerebras", "model": "llama-3.3-70b"},
    {"id": "s3", "credentialId": "cred_openrouter", "model": "openai/gpt-4o-mini"}
  ]
}
3-step example
{
  "type": "speculative",
  "steps": [
    {"id": "s1", "credentialId": "cred_groq", "model": "llama-3.3-70b-versatile"},
    {"id": "s2", "credentialId": "cred_cerebras", "model": "llama-3.3-70b"},
    {"id": "s3", "credentialId": "cred_openrouter", "model": "openai/gpt-4o-mini"}
  ]
}
Behavior
Semantically identical to parallel. The engine delegates to the parallel executor; the first streaming chunk wins and losers are aborted. Use this type to communicate to future readers / audit logs that the intent is speculative latency racing rather than throughput.
4chainChain

Pipe step 1's output into step 2's input. For chat, step 1's assistant text is appended as a new user message and step 2 refines it. Returns the final step's response.

When to use
You want a multi-turn refinement pipeline: summarize with a small model, then refine with a large model. Or: extract entities with a structured model, then format with a fluent model.
Example
{
  "id": "combo_chain_summarize_refine",
  "orgId": "org_abc",
  "name": "Summarize then refine",
  "type": "chain",
  "enabled": true,
  "steps": [
    {"id": "s1", "credentialId": "cred_groq", "model": "llama-3.3-70b-versatile", "params": {"temperature": 0.2}},
    {"id": "s2", "credentialId": "cred_openai", "model": "gpt-4o", "params": {"temperature": 0.5}},
    {"id": "s3", "credentialId": "cred_anthropic", "model": "claude-3.5-sonnet", "params": {"temperature": 0.4}}
  ]
}
3-step example
{
  "type": "chain",
  "steps": [
    {"id": "s1", "credentialId": "cred_groq", "model": "llama-3.3-70b-versatile"},
    {"id": "s2", "credentialId": "cred_openai", "model": "gpt-4o"},
    {"id": "s3", "credentialId": "cred_anthropic", "model": "claude-3.5-sonnet"}
  ]
}
Behavior
Each intermediate step runs non-streaming (the engine needs the full text). The assistant text from step 1 is appended to the request body as a new user message before step 2 runs. The final step honours the original request's streaming flag. If an intermediate step returns empty text or fails, the chain short-circuits and returns the last successful step's output.
5routingRouting

Like fallback, but each step uses its own routingStrategy to pick a credential from the pool. The first step that succeeds is returned.

When to use
You want a fallback chain where each step prefers a different selection criterion — e.g., try the cheapest credential first, then the fastest, then any available.
Example
{
  "id": "combo_routing_multistrat",
  "orgId": "org_abc",
  "name": "Cost then latency then any",
  "type": "routing",
  "enabled": true,
  "steps": [
    {"id": "s1", "routingStrategy": "lowest-cost"},
    {"id": "s2", "routingStrategy": "least-latency"},
    {"id": "s3", "routingStrategy": "priority-failover"}
  ]
}
3-step example
{
  "type": "routing",
  "steps": [
    {"id": "s1", "routingStrategy": "lowest-cost"},
    {"id": "s2", "routingStrategy": "least-latency"},
    {"id": "s3", "routingStrategy": "priority-failover"}
  ]
}
Behavior
Step 1 uses the lowest-cost strategy to pick a credential from the pool; step 2 uses least-latency; step 3 uses priority-failover. Steps are tried in order; the first success wins. Each step's fallback chain (if the strategy returns one) is also followed before advancing to the next step.

Wiring a combo to a routing rule

A combo by itself does nothing until a routing rule references it. Use thecombo strategy with a comboId:

{
  "name": "Use the chat-with-fallback combo",
  "enabled": true,
  "priority": 5,
  "match": {"endpoint": "/v1/chat/completions"},
  "strategy": "combo",
  "strategyConfig": {"comboId": "combo_fallback_chat"},
  "credentialPool": []
}

When the router matches this rule, it loads the combo definition from Lightbase, builds aComboExecutionContext with the org's enabled credentials, and delegates to the matching executor. Each step records its own usage row with the combo id.

Per-step params override

The params field on a step is merged into the request body before the upstream call. Use it to override temperature, max_tokens, top_p, or any provider-specific field on a per-step basis.

{
  "id": "s2",
  "credentialId": "cred_openai",
  "model": "gpt-4o",
  "params": {"temperature": 0.7, "max_tokens": 1024, "top_p": 0.95}
}

Stream semantics

  • fallback: if the request is streaming, only the winning step's stream is piped to the client. Failed steps do not produce visible output.
  • parallel and speculative: the first upstream to produce a streaming chunk wins. Loser streams are aborted; their underlying fetch is cancelled via AbortController.
  • chain: intermediate steps run non-streaming (the engine needs the full text to feed the next step). Only the final step honours the client's stream: true flag.
  • routing: same as fallback — the winning step's stream is piped to the client.

Audit and usage

Every step in a combo produces a ComboStepResult (stepId, credentialId, providerId, status, latencyMs, success, error). The engine returns aComboExecutionResult containing the winning credential id, the per-step results, the final response or stream, and the usage from the winning step. Each step writes a usage_records row tagged with the combo id.