Skip to main content

ServiceProvider

ServiceProvider is the org-scoped management-plane object for a backend that can serve model, agent, or compound-agent traffic. It lets Responses Proxy load providers from SQL and make them routable without redeploying.

Why it exists​

Teams may need to register:

  • multiple OpenAI accounts or projects
  • OpenAI-compatible gateways such as Kimi or OpenRouter
  • internal agent services behind a Responses-compatible endpoint
  • Anthropic or other heterogeneous providers
  • future custom gateways once a custom adapter exists

The record answers: where is the service, what wire protocol does it speak, how does auth work, what credential should be used, and what metadata should dashboards and operators see?

Provider example​

{
  "id": "interactive-main",
  "name": "Interactive Main",
  "provider_type": "openai",
  "base_url": "https://api.openai.com/v1",
  "wire_api": "responses",
  "auth_type": "bearer",
  "auth_header": "authorization",
  "api_key": "replace-with-provider-key",
  "health_path": "models",
  "metadata": {
    "owner": "platform",
    "purpose": "codex-interactive"
  }
}

Then register a model alias:

{
  "alias": "codex/interactive",
  "display_name": "Codex Interactive",
  "kind": "model",
  "provider_id": "interactive-main",
  "upstream_id": "gpt-4.1-mini",
  "route": {
    "type": "responses",
    "model": "gpt-4.1-mini"
  },
  "capabilities": {
    "streaming": true,
    "tools": true,
    "claude_code": {
      "enabled": true,
      "profile": "auto",
      "images": true,
      "documents": true,
      "client_tools": true,
      "parallel_tools": true,
      "hosted_tools": []
    }
  }
}

A request with model: "codex/interactive" routes to the provider resource, rewrites the upstream model, and records provider_id plus model_alias in telemetry.

The nested claude_code object also makes the alias available through the Messages-compatible gateway. profile="auto" resolves from the provider wire_api; use enabled=false or narrower media/tool flags when the upstream model cannot support the defaults. Chat Completions aliases cannot declare hosted tools.

The request must also include a proxy key for the same org:

Authorization: Bearer rprx-...

Routing path​

  1. Resolve the proxy key to its owning org.
  2. Use the current in-process routing snapshot built from SQL at startup, mutation, manual reload, or periodic refresh.
  3. Use active resources derived from provider credential rows or provider-level fallback credentials.
  4. Keep live resource health, counters, and active-resource ids in Redis.
  5. Match POST /v1/responses model to a registered alias.
  6. Pin the provider pool and rewrite the upstream model.
  7. Select a healthy resource with affinity or weighted least-connections.
  8. Use the provider adapter to passthrough or translate the request.
  9. Persist request and decision rows inline, then publish raw telemetry events for worker replay.

Operational tradeoffs​

Durable config drives live routing:

  • Benefit: providers and aliases take effect without redeploying.
  • Cost: data-plane availability depends on registry correctness and routing refresh on every replica.

Provider credential rows support BYOS:

  • Benefit: users can add, validate, rotate, disable, or delete credentials without editing provider or alias records.
  • Cost: credential-row precedence can hide provider-level env or inline credentials while any enabled row exists.

Inline credentials are easy to set up:

  • Benefit: the dashboard can create a working provider quickly.
  • Cost: encrypted credential material lives in SQL, so PROXY_SECRET_ENCRYPTION_KEY must stay stable.

Env-sourced key pools improve throughput and fault isolation:

  • Benefit: one provider row can expand into resources from OPENAI_API_KEY, OPENAI_API_KEY_1, and later slots.
  • Cost: env vars are deployment-global and only resolve for the bootstrap org by default, or orgs listed in PROXY_ENV_CREDENTIAL_OWNER_ORG_IDS; orgs that need separate physical credentials should use distinct provider ids.

External credential references keep ownership elsewhere:

  • Benefit: secret ownership remains outside the proxy database.
  • Cost: the runtime environment must provide the referenced value.

Proxy-managed response state supports stateless providers:

  • Benefit: Chat Completions and Anthropic can support Codex previous_response_id continuations.
  • Cost: reconstructed conversation state is stored in SQL.

Refresh behavior​

After a successful provider or alias mutation, the current process refreshes routing immediately. Other replicas refresh from SQL within PROXY_ROUTING_REFRESH_INTERVAL_SECONDS.

Use POST /admin/routing/reload when you need to trigger a reload explicitly.

Deleting a provider or alias removes the registry row. Use enabled=false only for a reversible runtime pause while the row exists.