Skip to main content

Responses Proxy troubleshooting

Start with health, state, and a simple smoke request before debugging deeper routing behavior.

Quick health checklist​

Set the service URL:

export RESPONSES_PROXY_BASE_URL="https://agent-responses-proxy.svc.cloud.morph.so"

Check service health and OpenAPI:

curl -fsS "$RESPONSES_PROXY_BASE_URL/healthz"
curl -fsS "$RESPONSES_PROXY_BASE_URL/readyz"
curl -fsS "$RESPONSES_PROXY_BASE_URL/openapi.json" >/tmp/openapi.json
curl -fsS "$RESPONSES_PROXY_BASE_URL/admin/state"

Send a minimal Responses request:

curl -sS "$RESPONSES_PROXY_BASE_URL/v1/responses" \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4.1-mini",
    "input": "Reply with exactly: smoke ok",
    "metadata": {
      "session_id": "troubleshooting-smoke-1",
      "session_name": "troubleshooting smoke"
    }
  }'

Check durable records:

curl "$RESPONSES_PROXY_BASE_URL/admin/requests?limit=5"
curl "$RESPONSES_PROXY_BASE_URL/admin/decisions?limit=5"

No provider resources configured​

Cause: no runtime OpenAI resources were parsed.

Set one of:

OPENAI_API_KEY=sk-...
OPENAI_API_KEYS=sk-1,sk-2
OPENAI_API_KEY_1=sk-...

Registered providers from /admin/providers do not yet feed the live load balancer.

Records disappear after redeploy​

You may be looking at Redis-backed live counters, or the service may be using container-local SQLite.

Check:

echo "${PROXY_DATABASE_URL:+set}"
curl "$RESPONSES_PROXY_BASE_URL/admin/requests?limit=5"

Fix:

  • Set PROXY_DATABASE_URL to the same Postgres or Neon database for every deploy.
  • Confirm worker and API receive the same database URL.
  • Expect live counters, affinity count, and queue depth to reset after redeploy.

Requests succeed but a 403 appears​

Possible causes:

  • one configured API key is invalid or unauthorized
  • health checks call /v1/models and one key receives 403
  • a request is routed to a key without access to the selected model

Diagnose:

curl -X POST "$RESPONSES_PROXY_BASE_URL/admin/health/check"
curl "$RESPONSES_PROXY_BASE_URL/admin/state"
curl "$RESPONSES_PROXY_BASE_URL/admin/requests?limit=20"

Fix:

  • Replace invalid keys.
  • Confirm every key has access to the model used by clients.
  • Temporarily reduce the pool to known-good keys.

Dashboard shows resources but no recent requests​

Likely causes:

  • worker is not running
  • worker and API point at different Redis URLs
  • worker and API point at different database URLs
  • the request has not completed yet

Fix:

uv run responses-proxy-worker
curl "$RESPONSES_PROXY_BASE_URL/admin/requests?limit=5"

For Docker service, inspect logs:

docker logs --tail 200 agent-responses-proxy-server

Token columns are empty​

Token usage is extracted only when upstream response bodies include supported usage fields:

  • usage.input_tokens
  • usage.output_tokens
  • usage.total_tokens
  • usage.prompt_tokens
  • usage.completion_tokens
  • SSE payloads with response.usage

If streaming usage is missing, or PROXY_MAX_CAPTURE_BYTES truncates captured data before usage appears, token columns can remain empty.

Codex config TOML error​

Symptom:

invalid type: string ..., expected struct ModelProviderInfo

Cause: provider config was passed as one inline TOML string.

Use helper scripts or separate -c overrides:

codex \
  -c model_provider='"proxy"' \
  -c model='"gpt-4.1-mini"' \
  -c model_providers.proxy.name='"Responses Proxy"' \
  -c model_providers.proxy.base_url='"https://agent-responses-proxy.svc.cloud.morph.so/v1"' \
  -c model_providers.proxy.wire_api='"responses"'

Registry returns 401​

Cause: PROXY_ADMIN_TOKEN is configured and the request did not include the bearer token.

Fix:

curl -H "Authorization: Bearer $PROXY_ADMIN_TOKEN" "$RESPONSES_PROXY_BASE_URL/admin/providers"

Inline provider API key is rejected​

Symptom:

Set PROXY_SECRET_ENCRYPTION_KEY before registering inline API keys

Fix:

export PROXY_SECRET_ENCRYPTION_KEY="$(openssl rand -base64 32)"

Restart the API and retry. For production, store the value as a stable GitHub environment secret. Alternatively, use credential_ref and omit api_key.

Registered provider does not receive traffic​

This is expected in the current implementation. The registry stores future provider, model, and agent configuration. Live routing still uses resources parsed from:

OPENAI_API_KEY_1=...
OPENAI_API_KEY_2=...

No available resources are registered​

Causes:

  • requested pool has no resources
  • all resources are unhealthy
  • no keys were configured

Diagnose:

curl "$RESPONSES_PROXY_BASE_URL/admin/state"

Fix:

  • Check metadata.pool_id or X-Proxy-Pool-ID.
  • Confirm OPENAI_RESOURCE_POOL_IDS matches the requested pool.
  • Run /admin/health/check.
  • Replace bad keys.

Docker health check fails​

Diagnose:

docker ps -a
docker logs --tail 200 agent-responses-proxy-server
curl -fsS "$RESPONSES_PROXY_BASE_URL/healthz"

Common causes:

  • container env is missing OpenAI keys
  • Redis failed to start inside the container
  • port 8000 is already in use
  • worker or API process exited