Skip to main content

Responses Proxy troubleshooting

Start with health, admin auth, routing state, and a registered alias before debugging deeper provider behavior.

Quick health checklist​

Set the service URL and admin header:

export RESPONSES_PROXY_BASE_URL="https://agent-responses-proxy.svc.cloud.morph.so"
export ADMIN_HEADER="Authorization: Bearer $SEELE_ORG_ADMIN_MORPH_API_KEY"
export RESPONSES_PROXY_API_KEY="rprx-..."

Check public health and OpenAPI:

curl -fsS "$RESPONSES_PROXY_BASE_URL/healthz"
curl -fsS "$RESPONSES_PROXY_BASE_URL/readyz"
curl -fsS "$RESPONSES_PROXY_BASE_URL/openapi.json" >/tmp/openapi.json

Check authenticated admin state:

curl -fsS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/state"
curl -fsS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/routing"

Send a minimal request with a registered alias:

curl -sS "$RESPONSES_PROXY_BASE_URL/v1/responses" \
  -H "Authorization: Bearer $RESPONSES_PROXY_API_KEY" \
  -H 'content-type: application/json' \
  -d '{
    "model": "openai/gpt-mini",
    "input": "Reply with exactly: smoke ok",
    "metadata": {
      "session_id": "troubleshooting-smoke-1",
      "session_name": "troubleshooting smoke"
    }
  }'

401 on /v1/responses​

Cause: the request is missing a valid org-scoped proxy key.

Fix: mint a key in the Responses console or with POST /admin/keys, then send the returned secret as:

Authorization: Bearer <key>

Keys are org-scoped, so the key must belong to the org that owns the target provider and alias.

503 on /v1/responses​

Cause: the registry has no routable providers, or all compatible providers were skipped.

Fix:

  • Register a provider with wire_api set to responses, chat_completions, or anthropic_messages.
  • Ensure the provider is enabled.
  • Ensure inline credentials can decrypt, or the referenced environment value exists.
  • Confirm resolved_key_count and pool_size are greater than zero.
  • Register a model alias pointing at the provider.
  • Check the routing view:
curl -sS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/routing"

400 missing model​

POST /v1/responses create requests require model. Use a registered alias, for example:

{"model": "openai/gpt-mini", "input": "hello"}

404 unknown model​

The model value is not a registered alias. Check aliases:

curl -sS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/models"

Admin returns 401​

Cause: missing or invalid Morph org token or service delegation token.

Fix:

curl -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/providers"

If this fails through the console, confirm the frontend server can obtain a service delegation token for agent-responses-proxy with responses:admin.

Admin returns 403​

Cause: the bearer token is valid, but the principal is not a member of the allowed organization or does not have an allowed role.

Fix: check PROXY_ADMIN_ALLOWED_ORG_ID, PROXY_ADMIN_ALLOWED_ORG_SLUG, and PROXY_ADMIN_ALLOWED_ROLES.

Admin returns 503​

Cause: admin auth is misconfigured or a Morph auth dependency is unavailable.

Fix: check PROXY_ADMIN_AUTH_MODE, Morph API base URLs, PROXY_ADMIN_DELEGATION_SERVICE_API_KEY, and allowed org settings.

Records disappear after redeploy​

Usually the service is using container-local SQLite, or you were viewing Redis live state.

Fix:

  • Set PROXY_DATABASE_URL to the same Neon/Postgres database for every deploy.
  • Confirm API and worker receive the same database URL.
  • Expect Redis counters, health, queue depth, and affinity to reset with container lifecycle.

Requests succeed but metrics are empty​

Likely causes:

  • PROXY_DATABASE_URL is unset, so the service is using local SQLite
  • API and frontend/BFF are pointing at different deployments or organizations
  • the request has not completed yet; streaming rows are written when the stream closes
  • the request used a proxy key from a different org than the dashboard org

Fix:

curl -sS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/overview?limit=5"
curl -sS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/requests?limit=5"
curl -sS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/metrics"

Current backend builds write request and decision rows inline on the request path, then publish raw telemetry to Redis for worker persistence and replay. If /admin/requests is empty after a completed request, check the database URL, proxy-key org, and active frontend org before debugging the worker.

If telemetry_queue_depth grows while request rows are fresh, the data plane is working but the worker is lagging on raw telemetry events. Restart the worker or inspect worker logs.

Provider is listed but does not route​

Check:

  • enabled is true
  • wire_api is responses, chat_completions, or anthropic_messages
  • credential_resolved is true
  • resolved_key_count and pool_size are greater than zero
  • Env-sourced credentials are enabled for PROXY_BOOTSTRAP_ORG_ID, or for an org listed in PROXY_ENV_CREDENTIAL_OWNER_ORG_IDS
  • Production provider URLs are HTTPS public hosts
  • model alias has provider_id set to that provider
  • /admin/routing includes the alias and resource

Use validation endpoints before trying Codex:

curl -sS -X POST "$RESPONSES_PROXY_BASE_URL/admin/providers/test" \
  -H "$ADMIN_HEADER" \
  -H 'content-type: application/json' \
  -d '{"id":"openai-main","name":"OpenAI","provider_type":"openai","base_url":"https://api.openai.com/v1","wire_api":"responses","auth_type":"bearer","api_key":"replace-with-provider-key"}'

curl -sS -X POST "$RESPONSES_PROXY_BASE_URL/admin/models/test" \
  -H "$ADMIN_HEADER" \
  -H 'content-type: application/json' \
  -d '{"provider_id":"openai-main","upstream_id":"gpt-4.1-mini"}'

Inline credential rejected​

Cause: inline provider credentials require PROXY_SECRET_ENCRYPTION_KEY.

Fix: set the encryption value, restart, and retry. Keep it stable across deploys.

Secret reference does not resolve​

Cause: a provider uses credential_ref=env:NAME, but NAME is not present in the proxy runtime environment.

Fix: add the variable to the proxy deployment. In the console, $NAME is validated against org secrets by the BFF, but the backend can only resolve it if the runtime env contains the same name, matches *_API_KEY or *_API_KEY_<number> unless allowlisted, and is not a blocked sensitive name.

Token columns are empty​

Token usage is extracted only when response bodies include supported usage fields:

  • usage.input_tokens
  • usage.output_tokens
  • usage.total_tokens
  • usage.prompt_tokens
  • usage.completion_tokens
  • SSE payloads with response.usage

If usage is absent or malformed, token columns remain empty. Request and response body columns may also be empty because PROXY_CAPTURE_REQUEST_BODY and PROXY_CAPTURE_RESPONSE_BODY default to false.

For streaming responses, token usage is written after the stream closes. OpenAI Responses-backed Claude Code streaming also records usage from the converted stream state when available.

Revoked proxy key still works briefly​

Cause: positive proxy-key org resolution is cached in each running API process for PROXY_PROXY_KEY_CACHE_TTL_SECONDS, which defaults to 30 seconds.

Fix: wait for the cache window to expire, restart the API process, or lower the TTL. Set PROXY_PROXY_KEY_CACHE_TTL_SECONDS=0 only when immediate revocation matters more than hot-path efficiency. Invalid keys are cached for at most 5 seconds.

Metrics update but raw telemetry queue grows​

Cause: request and decision rows are persisted inline, but Redis stream events still need the telemetry worker for raw event history and replay.

Fix:

uv run responses-proxy-worker
curl -sS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/overview?limit=5"

Check worker logs for Redis group, database, or schema errors. API and worker should share PROXY_REDIS_URL and PROXY_DATABASE_URL.

Codex model metadata warning​

If Codex prints Model metadata for <alias> not found, the proxy alias can still work, but Codex does not have local catalog metadata for that alias.

Add the alias to codex-model-catalog.json, or point MODEL_CATALOG_JSON at a custom catalog. For one-off tuning, set model-specific limits to match the upstream model:

MODEL_CONTEXT_WINDOW=200000
MODEL_AUTO_COMPACT_TOKEN_LIMIT=160000
TOOL_OUTPUT_TOKEN_LIMIT=12000

Codex posts to /responses​

If Codex reports unexpected status 404 Not Found for /responses, the configured provider base URL is missing /v1.

Use:

https://agent-responses-proxy.svc.cloud.morph.so/v1

Claude Code posts to /v1/v1/messages​

If server logs show POST /v1/v1/messages, ANTHROPIC_BASE_URL already includes /v1 and Claude Code is appending another /v1.

Use the proxy base URL that matches your Claude Code version. Current tested setups use:

export ANTHROPIC_BASE_URL="https://agent-responses-proxy.svc.cloud.morph.so/v1"

If that produces /v1/v1/messages, remove the suffix:

export ANTHROPIC_BASE_URL="https://agent-responses-proxy.svc.cloud.morph.so"

Claude Code returns 401 asking for login​

Cause: the proxy key is missing from the configured Claude Code custom headers.

Fix:

export ANTHROPIC_CUSTOM_HEADERS="X-Responses-Proxy-Key: rprx-..."

The proxy key is not an Anthropic key. It must be minted from the Responses Proxy dashboard or /admin/keys.

Claude Code says model may not exist​

Likely causes:

  • the model name is not registered as an alias in the proxy-key org
  • GET /v1/models does not list the alias
  • the alias points at a provider protocol unsupported by the Claude Code gateway
  • the provider has no routable credential rows or fallback credentials

Fix:

curl -sS "$RESPONSES_PROXY_BASE_URL/v1/models" \
  -H "X-Responses-Proxy-Key: $RESPONSES_PROXY_API_KEY"
curl -sS -H "$ADMIN_HEADER" "$RESPONSES_PROXY_BASE_URL/admin/routing"

Claude Code aliases support wire_api="anthropic_messages", wire_api="responses", and wire_api="chat_completions" providers. Chat aliases also require PROXY_CLAUDE_GATEWAY_CHAT_COMPLETIONS_ENABLED=true. Check capabilities.claude_code.enabled and ensure an explicit profile matches the provider protocol.

Claude Code alias is missing from /v1/models​

Check these conditions:

  • the alias and provider are enabled and belong to the proxy key's org
  • the provider has a resolved credential and active resource
  • capabilities.claude_code.enabled is not false
  • profile is auto or matches the provider wire_api
  • Chat Completions routing is enabled by deployment

Existing aliases without a claude_code object use the compatibility defaults and are eligible. Use the console alias editor to make the intended capabilities explicit.

Claude Code request returns 413​

The complete JSON request, including base64 media expansion, exceeds PROXY_CLAUDE_GATEWAY_MAX_BODY_BYTES. The default is 33554432 bytes.

Reduce or resize attached media, split the request, or raise the limit deliberately. URL images avoid base64 expansion; Chat Completions document URLs are not portable and remain unsupported.

Claude Code media or tool request returns 422​

The gateway rejects capabilities it cannot preserve. Common causes:

  • images, documents, or client tools are disabled on the alias
  • a Claude image uses a provider file id instead of base64 or URL input
  • a document uses a provider-scoped file id
  • a Chat Completions alias receives a document URL instead of base64 file data
  • a translated request contains a server tool other than supported web search
  • hosted web search is not enabled on a Responses alias, or is configured on Chat Completions

Do not enable a toggle merely to suppress the error. Confirm the upstream model and provider implement that input shape, then test the complete flow.

X-Responses-Proxy-Normalized-Fields appears​

For translated Responses and Chat aliases, this response header lists Anthropic-only fields the gateway removed or normalized, including prompt-cache, context-management, top_k, and tool-level cache/eager-streaming hints. It is diagnostic, not an upstream error.

Use an anthropic_native alias when the workflow requires provider-native Messages semantics. Translated aliases intentionally reject or report fields that cannot be represented exactly.

Claude Code OAuth setup fails​

Check:

  • PROXY_ENABLE_CLAUDE_CODE_TOKEN_ESCROW=true
  • PROXY_SECRET_ENCRYPTION_KEY is stable and configured
  • the provider is first-party Anthropic: provider_type="anthropic" and base_url="https://api.anthropic.com/v1"
  • the provider uses wire_api="anthropic_messages", auth_type="bearer", and auth_header="authorization"
  • the setup token has not expired or already completed

Use stdin for setup tokens:

printf '%s\n' "$SETUP_TOKEN" | responses-proxy-claude-code teleport \
  --base-url "$RESPONSES_PROXY_BASE_URL" \
  --session-id "$SESSION_ID" \
  --setup-token-stdin

In fish, use the printf ... | form rather than bash-style heredocs.

Tool adapter errors​

If a Chat Completions or Anthropic model reports that a tool type is unsupported, confirm the backend build includes Codex tool adapter support.

Current adapter behavior:

  • Codex custom tools map to upstream JSON tools with an input string.
  • Freeform apply_patch tool arguments tolerate keys such as input, patch, diff, content, cmd, and command.
  • Upstream calls are converted back to Responses custom_tool_call items.
  • Nested namespace tool containers are flattened into provider-native JSON tools.
  • Non-forwardable hosted Responses tools, such as web_search, are omitted for Chat Completions and Anthropic providers.

That omission rule applies to the Codex-facing /v1/responses adapter. On the Claude Code surface, translated server tools are validated strictly: only web_search can map to a Responses alias, and Chat Completions has no hosted-tool mapping. Unsupported server tools return an error instead of being omitted.

Docker health check fails​

Diagnose:

docker ps -a
docker logs --tail 200 agent-responses-proxy-server
curl -fsS "$RESPONSES_PROXY_BASE_URL/healthz"

Common causes: Redis failed to start, port 8000 is in use, or the API/worker exited.

GitHub Actions deploy fails​

The deploy dry-run requires Morph builder settings plus:

  • PROXY_DATABASE_URL
  • PROXY_BOOTSTRAP_ORG_ID
  • PROXY_ADMIN_DELEGATION_SERVICE_API_KEY when using delegation auth
  • PROXY_ADMIN_ALLOWED_ORG_ID or PROXY_ADMIN_ALLOWED_ORG_SLUG when using org auth
  • PROXY_SECRET_ENCRYPTION_KEY

Add them as GitHub environment secrets without printing values in logs.