Skip to main content

ServiceProvider

ServiceProvider is the management-plane object for a backend that can serve model, agent, or compound-agent traffic. It lets Responses Proxy load providers from SQL and make them routable without redeploying.

Why it exists​

Teams may need to register:

  • multiple OpenAI accounts or projects
  • OpenAI-compatible gateways such as Kimi or OpenRouter
  • internal agent services behind a Responses-compatible endpoint
  • Anthropic or other heterogeneous providers
  • future custom gateways once a custom adapter exists

The record answers: where is the service, what wire protocol does it speak, how does auth work, what credential should be used, and what metadata should dashboards and operators see?

Provider example​

{
  "id": "interactive-main",
  "name": "Interactive Main",
  "provider_type": "openai",
  "base_url": "https://api.openai.com/v1",
  "wire_api": "responses",
  "auth_type": "bearer",
  "auth_header": "authorization",
  "api_key": "replace-with-provider-key",
  "health_path": "models",
  "metadata": {
    "owner": "platform",
    "purpose": "codex-interactive"
  }
}

Then register a model alias:

{
  "alias": "codex/interactive",
  "display_name": "Codex Interactive",
  "kind": "model",
  "provider_id": "interactive-main",
  "upstream_id": "gpt-4.1-mini",
  "route": {
    "type": "responses",
    "model": "gpt-4.1-mini"
  },
  "capabilities": {
    "streaming": true,
    "tools": true
  }
}

A request with model: "codex/interactive" routes to the provider resource, rewrites the upstream model, and records provider_id plus model_alias in telemetry.

Routing path​

  1. Load enabled providers and aliases from SQL.
  2. Resolve inline or referenced credentials.
  3. Convert adapter-compatible providers into active resources.
  4. Match POST /v1/responses model to a registered alias.
  5. Pin the provider pool and rewrite the upstream model.
  6. Select a healthy resource with affinity or weighted least-connections.
  7. Use the provider adapter to passthrough or translate the request.
  8. Emit durable request and decision telemetry.

Operational tradeoffs​

Durable config drives live routing:

  • Benefit: providers and aliases take effect without redeploying.
  • Cost: data-plane availability depends on registry correctness.

Inline credentials are easy to set up:

  • Benefit: the dashboard can create a working provider quickly.
  • Cost: encrypted credential material lives in SQL, so PROXY_SECRET_ENCRYPTION_KEY must stay stable.

External credential references keep ownership elsewhere:

  • Benefit: secret ownership remains outside the proxy database.
  • Cost: the runtime environment must provide the referenced value.

Proxy-managed response state supports stateless providers:

  • Benefit: Chat Completions and Anthropic can support Codex previous_response_id continuations.
  • Cost: reconstructed conversation state is stored in SQL.

Refresh behavior​

After a successful provider or alias mutation, the current process refreshes routing immediately. Other replicas refresh from SQL within PROXY_ROUTING_REFRESH_INTERVAL_SECONDS.

Use POST /admin/routing/reload when you need to trigger a reload explicitly.