Skip to main content

Dedicated Gateway Service

Run the API Builder gateway as its own service, next to Studio, on Docker Swarm or Kubernetes with autoscaling.

The API Gateway normally runs inside Studio. For heavy API traffic, run it as a service of its own: the same image and code, started with HRIDAAI_SERVICE=gateway. API calls then get their own replicas, CPU and memory. A traffic spike no longer slows Studio's pages, chat and agents, and on Kubernetes the gateway scales itself.

Use it when:

  • partners or other systems call your published APIs at high volume, or in bursts;
  • you want to scale, deploy or limit API traffic separately from Studio;
  • you want the gateway to scale automatically on Kubernetes.

For light traffic, Studio's built-in gateway is enough. The per-call speed-ups (pooled connections, caching) apply there too.

How it works​

  • Same gateway, nothing else. The service answers only the gateway's URLs: /gw/…, /api/v1/api-definitions/<id>/proxy/… and /api/v1/api-definitions/<id>/env/<stage>/proxy/…, plus health checks. Every other path returns 404. It runs from a slim image of its own and starts in 2 to 4 seconds; a worker uses about 125 MB of memory idle and 160 MB under load, against about 1 GB for Studio.
  • Shared with Studio:
    • the database and Redis, with the same REDIS_KEY_PREFIX;
    • HRIDAAI_SECRET_KEY, which verifies Studio's caller tokens and decrypts upstream credentials, and HRIDAAI_ENCRYPTION_KEY / HRIDAAI_ENCRYPTION_KEYS_PREVIOUS when Studio sets them (see Rotating the encryption key).
  • Stays in Studio:
    • managing APIs and the database migrations;
    • the jobs that read the call log: analytics, alerts and retention. The service writes its calls to the same call log.
  • Changes arrive at once. Each process caches API definitions, versions, mappers, consumer keys and settings for API_GATEWAY_CACHE_SECONDS (15 by default). Any change made in Studio clears them in every process right away, through Redis: publishing, a new version, revoking a key, suspending a space.
Use Redis

Without Redis, a change made in Studio reaches the service only when its cache period ends, and rate limits are counted separately in each process. The service logs a warning at startup when REDIS_URL isn't set.

The gateway image​

The gateway has an image of its own, the hrida-ai-gateway package in GitHub's container registry, built from Studio's Dockerfile (docker build --target gateway). It holds only the gateway and the packages it needs, without the web interface, document tools or machine-learning libraries. It is about 490 MB, starts in 2 to 4 seconds, and runs as an unprivileged user (UID 10001).

Its tags are Studio's, so one HRIDAAI_DOCKER_TAG pins both:

Studio imageGateway image
ghcr.io/hrida-ai/hrida-ai-studio:1.4.2 (a release)ghcr.io/hrida-ai/hrida-ai-gateway:1.4.2
…/hrida-ai-studio:latest…/hrida-ai-gateway:latest
…/hrida-ai-studio:main…/hrida-ai-gateway:main

CI builds it for linux/amd64 and linux/arm64 with every Studio image, and each release's notes list both images. Releases before the package existed published the gateway as Studio's image with a -gateway tag (hrida-ai-studio:1.0.25-gateway); CI keeps adding those tags, so deployments that pull them still get updates. Studio's own image can also run the gateway (HRIDAAI_SERVICE=gateway): set GATEWAY_IMAGE=ghcr.io/hrida-ai/hrida-ai-studio in .env to use it instead.

Container Images lists all of Studio's images, their tags and how to pin them.

Deploy on Docker Swarm​

The Swarm stack, docker-compose-studio-prodSwarm.yaml, includes the hrida-ai-gateway service. It runs the gateway image of Studio's release, with these settings in .env.

Docker Swarm doesn't scale services on load: it keeps GATEWAY_REPLICAS containers running. Size it for your peak traffic (see Sizing), or change it by hand (docker service scale <stack>_hrida-ai-gateway=4). For scaling on load, run the gateway on Kubernetes. Pin a release in HRIDAAI_DOCKER_TAG: Swarm resolves the tag once per deploy, so every replica runs the same build.

VariableDefaultWhat it sets
GATEWAY_REPLICAS2Containers, spread across nodes
GATEWAY_UVICORN_WORKERS2Gateway processes per container
GATEWAY_DATABASE_POOL_SIZE5Database connections per process (most lookups are cached)
GATEWAY_DATABASE_POOL_MAX_OVERFLOW5Extra connections per process at peaks
GATEWAY_MEM_LIMIT1gMemory limit per container
GATEWAY_GRACEFUL_SHUTDOWN_SECONDS40On stop, how long calls in progress may finish (the stack allows 60 s)
HRIDA_AI_GATEWAY_PUBLISHED_PORT3014Port on each node for the load balancer
GATEWAY_IMAGEghcr.io/hrida-ai/hrida-ai-gatewayThe image, with Studio's tag; ghcr.io/hrida-ai/hrida-ai-studio runs Studio's own image
GATEWAY_DATABASE_HOST, GATEWAY_DATABASE_PORThrida-postgres, 5432Where the gateway reaches the database, e.g. PgBouncer
GATEWAY_ACCESS_LOGfalseUvicorn's line per request; the call log records each call anyway
GATEWAY_OTEL_TRACES_SAMPLER_ARG0.05Share of calls traced, when tracing is on

Deploy as usual:

bash
set -a; . ./.env; set +a
docker stack deploy --with-registry-auth -c docker-compose-studio-prodSwarm.yaml hrida-studio

During an upgrade, new gateway containers wait until Studio has migrated the database. They report healthy only then, for up to 5 minutes, so the old ones keep serving until the new ones are ready.

Firewall

Like every port Swarm publishes, 3014 is open on each node's public IP. Allow only the load balancer to reach it, as for Grafana and OnCall.

On one host, with Docker Compose​

Add the service next to Studio in your compose file, with the same .env:

yaml
services:
  hrida-ai-studio:
    # … as before, plus:
    environment:
      API_BUILDER_SELF_URL: http://hrida-ai-gateway:8080

  hrida-ai-gateway:
    image: ghcr.io/hrida-ai/hrida-ai-gateway:latest # Studio's tag
    env_file: .env # Studio's DATABASE_URL, REDIS_URL, REDIS_KEY_PREFIX, HRIDAAI_SECRET_KEY
    environment:
      HRIDAAI_SERVICE: gateway
      ENABLE_DB_MIGRATIONS: "false"
      UVICORN_WORKERS: "2"
    ports:
      - "3014:8080"
    restart: unless-stopped

Give it its own host name​

Outside callers reach the service on a host name of its own, for example api.example.com:

  1. DNS — point api.example.com at your load balancer.
  2. Load balancer — an HTTPS service for that host, forwarding to port 3014 on each node, with the health check GET /ready.
  3. Studio — in Admin › API Platform › Gateways, set the Public base URL of the gateway that serves production to https://api.example.com.

The API pages then show https://api.example.com/gw/… URLs. The spec served at the gateway URL names that host too, so client generators send calls to the service.

Existing https://hrida.example.com/gw/… URLs keep working, because Studio still serves them, so callers can move at their own pace.

One host instead

If your load balancer or proxy can route by path, you can keep a single host. Send /gw/ and the …/proxy/ paths to the service and everything else to Studio. For nginx:

nginx
location /gw/ {
    proxy_pass http://gateway_nodes;
}
location ~ ^/api/v1/api-definitions/[^/]+/(env/[^/]+/)?proxy/ {
    proxy_pass http://gateway_nodes;
}
location / {
    proxy_pass http://studio_nodes;
}

Studio's agents​

Studio sends its agents' API calls to API_BUILDER_SELF_URL. The Swarm stack sets it to http://hrida-ai-gateway:8080, so agent calls go to the service over the internal network.

  • When Studio starts, it moves existing tool-server registrations to the current URL.
  • To send agent calls back to Studio, unset the variable and restart Studio.

Kubernetes, with autoscaling​

The Studio repository has the manifests in kubernetes/gateway/:

FileWhat it is
deployment.yamlThe gateway pods: one worker each, probes, a graceful stop
service.yamlThe in-cluster address, hrida-ai-gateway:8080
hpa.yamlThe HorizontalPodAutoscaler
pdb.yamlKeeps at least one pod running through node drains
configmap.yamlSettings
secret.example.yamlThe secrets to create: DATABASE_URL, REDIS_URL, HRIDAAI_SECRET_KEY (and the encryption keys, if Studio sets them), and the image pull secret
ingress.example.yamlOptional: your API host name, for ingress-nginx

To deploy:

  1. Secret — create it with Studio's values:

    bash
    kubectl create secret generic hrida-ai-gateway -n hrida \
  --from-literal=DATABASE_URL='postgresql://user:password@postgres:5432/hridaai' \
  --from-literal=REDIS_URL='redis://redis:6379/0' \
  --from-literal=HRIDAAI_SECRET_KEY='<the same key as Studio>'

    Add HRIDAAI_ENCRYPTION_KEY and HRIDAAI_ENCRYPTION_KEYS_PREVIOUS the same way if Studio sets them.

  2. Image pull secret — the image is private. Create ghcr-hrida with a GitHub user and a token that can read packages (read:packages):

    bash
    kubectl create secret docker-registry ghcr-hrida -n hrida \
  --docker-server=ghcr.io --docker-username=<github-user> --docker-password=<token>

  3. Settings — check configmap.yaml. REDIS_KEY_PREFIX must match Studio's.

  4. Image — pin the release Studio runs, or its digest, in kustomization.yaml:

    bash
    cd kubernetes/gateway && kustomize edit set image ghcr.io/hrida-ai/hrida-ai-gateway:1.4.2

    With a moving tag such as latest, pods the autoscaler adds later can start a newer build than the pods already running.

  5. Apply:

    bash
    kubectl apply -k kubernetes/gateway -n hrida

  6. Studio — if it runs in the same cluster, set API_BUILDER_SELF_URL=http://hrida-ai-gateway.hrida.svc:8080.

The pods run as the image's unprivileged user, with a read-only root filesystem, no Linux capabilities and the RuntimeDefault seccomp profile; only /tmp and the data directory are writable, on small emptyDir volumes. Each pod asks for 500m CPU and 256 MiB of memory, with a 1 GiB limit.

How it scales​

The HorizontalPodAutoscaler keeps CPU use around 65% of each pod's request (500m). It needs the metrics-server add-on.

  • Range: 2 to 10 pods.
  • Up: fast, doubling or adding 4 pods every 30 seconds.
  • Down: slowly, after 5 quiet minutes and by at most half the pods a minute.

Pods can come and go freely: rate limits, circuit breakers, the response cache and cache invalidation are shared through Redis, and calls are logged to the database.

When a pod stops:

  1. It keeps serving for 10 seconds while it is removed from the Service.
  2. It lets calls in progress finish, for up to 40 seconds.
  3. It writes the calls still in its buffer to the call log.

Kubernetes allows 60 seconds for all of it. Node drains and cluster upgrades evict one gateway pod at a time (the PodDisruptionBudget allows one unavailable pod), however many the autoscaler is running.

Database connections

Each pod opens up to DATABASE_POOL_SIZE + DATABASE_POOL_MAX_OVERFLOW connections, 5 by default: with warm caches a call needs no query, and the buffered call log and key bookkeeping need one only now and then. At the maximum of 10 pods that is 50 connections, plus Studio's, and the total must stay below Postgres's max_connections (100 by default). Raise maxReplicas only as far as that allows, or put PgBouncer in front of Postgres.

Scaling on calls in progress​

The gateway mostly waits on upstream APIs. When they slow down, calls pile up while CPU stays low, so CPU alone scales too late. kubernetes/gateway/keda-scaledobject.example.yaml scales with KEDA on CPU and on calls in progress, from the gateway's OpenTelemetry metrics in Prometheus (ENABLE_OTEL=true, ENABLE_OTEL_METRICS=true):

  • The hridaai.api_gateway.in_flight gauge counts the calls in progress in each worker; KEDA adds pods to keep about 40 per pod.
  • KEDA creates and owns the HorizontalPodAutoscaler, so it replaces hpa.yaml: in kustomization.yaml, list the example instead of hpa.yaml.
  • Check the metric's name in Prometheus first: depending on the collector it may carry a _ratio suffix, and the service may be job or service_name.

The hridaai.api_gateway.calls counter (hridaai_api_gateway_calls_total in Prometheus) can drive scaling on calls per second the same way.

Who can reach the pods​

FORWARDED_ALLOW_IPS in configmap.yaml trusts X-Forwarded-For only from the cluster's private networks, where the ingress controller runs; consumer-key address allow-lists, the failed-login limit and the call log rely on it. Add the ingress's addresses if they are public, and never set * (the gateway would then take the left-most address, which the caller writes). To also keep everything but the ingress controller (and Studio's agents) away from the pods, use kubernetes/gateway/networkpolicy.example.yaml: adjust its two selectors and add it under resources: in kustomization.yaml.

Connection pooling with PgBouncer​

Each gateway worker keeps its own database connections, so replicas × workers × (pool + overflow) can pass Postgres's max_connections as the gateway grows. PgBouncer accepts all of those connections and shares a few dozen real ones between them.

On Docker Swarm, docker-compose-pgbouncer-prodSwarm.yaml runs it: 2 replicas, transaction pooling, PgBouncer 1.26.

bash
set -a; . ./.env; set +a
docker stack deploy -c docker-compose-pgbouncer-prodSwarm.yaml hrida-pgbouncer

Then point the gateway at it in .env, and redeploy the Studio stack:

bash
GATEWAY_DATABASE_HOST=hrida-pgbouncer
GATEWAY_DATABASE_PORT=6432
VariableDefaultWhat it sets
PGBOUNCER_REPLICAS2PgBouncer containers
PGBOUNCER_POOL_SIZE20Real Postgres connections per PgBouncer container
PGBOUNCER_MAX_CLIENT_CONN2000Client connections each container accepts
PGBOUNCER_QUERY_WAIT_TIMEOUT30Seconds a client may wait for a connection before an error

Keep PGBOUNCER_REPLICAS × PGBOUNCER_POOL_SIZE, plus Studio's and the worker's pools, below max_connections. Studio itself stays on Postgres directly: it runs the database migrations, which need a session of their own.

On Kubernetes, run PgBouncer the way you usually do, or use your managed database's pooler, and put its address in the gateway's DATABASE_URL.

Prepared statements work through PgBouncer 1.21 and later with max_prepared_statements set, as this stack does. Behind a pooler that can't follow them (an older PgBouncer, some managed database proxies), set DATABASE_ENABLE_PREPARED_STATEMENTS=false on the gateway.

High availability​

The gateway keeps no state of its own, so running two or more replicas makes it highly available. What it depends on needs the same care:

  • Postgres. While it's down, APIs the gateway has cached keep answering (definitions, versions and keys stay cached for API_GATEWAY_CACHE_SECONDS after they were last loaded), but call log rows that can't be written are lost; the gateway logs it. Run Postgres with a standby: a managed database with automatic failover, or a streaming replica you promote. Point DATABASE_URL at an address that always reaches the primary, such as the managed database's endpoint or a virtual IP.
  • Redis. Rate limits, quotas, Idempotency-Keys, the circuit breaker, the response cache and cache invalidation live there. If Redis can't be reached, each process falls back to its own memory, so calls keep working, with limits counted per process. For failover, use Redis Sentinel: set REDIS_SENTINEL_HOSTS (and REDIS_SENTINEL_PORT) on Studio and the gateway, with the master's name as the host in REDIS_URL, for example redis://mymaster:6379/0.
  • Valkey's memory. The Swarm data stack caps Valkey at REDIS_MAXMEMORY (512 MB, under its 768 MB container limit, REDIS_MEM_LIMIT) with the volatile-lru policy: when it's full, keys with an expiry, such as rate-limit counters and caches, are evicted first, never Studio's settings or queues. Raise both together for heavy traffic.

Metrics​

With OpenTelemetry metrics on (ENABLE_OTEL_METRICS), the gateway exports, besides calls, latency and breaker openings:

MetricWhat it measures
hridaai.api_gateway.shedCalls refused to protect the gateway, by reason: overloaded, upstream_busy, auth_throttled
hridaai.api_gateway.in_flightGateway calls in progress, per worker
hridaai.api_gateway.event_loop.lagHow late the worker's event loop runs: high values mean a CPU-bound worker
hridaai.api_gateway.cache.lookupsHits and misses of the per-process caches
hridaai.api_gateway.redis.durationTime spent on Redis per call
hridaai.api_gateway.call_log.buffered, hridaai.api_gateway.call_log.droppedCall log rows waiting to be written, and rows lost

Traces are sampled: the Swarm stack keeps 5% of the gateway's traces (GATEWAY_OTEL_TRACES_SAMPLER_ARG), and always keeps a trace the caller sampled. Uvicorn's access log is off in the gateway (GATEWAY_ACCESS_LOG), since the call log records every call.

Health checks​

PathAnswers 200 whenUse it for
/liveThe process answersKubernetes liveness
/healthThe database answersLoad balancers and Swarm, as Studio's /health
/readyStarted, and the database schema has reached this versionReadiness, load balancer health checks

While Studio hasn't yet migrated the database for a new version, /ready answers 503 with {"status": "waiting for Studio to migrate the database"}. A database outage doesn't make the service unready: APIs it has cached keep working.

Sizing and load testing​

A reference measurement, with the load test below on one machine. Each call was a GET through the gateway with a consumer key, to an HTTPS upstream that answers at once, with PostgreSQL and Redis, and the server limited to 2 CPUs:

Server1 caller32 callers at onceMemory
Studio (1 worker)167 calls/s, 5.6 ms per call245 calls/s, p95 166 msabout 1.1 GB
Gateway service, gateway image (1 worker)372 calls/s, 2.5 ms per call667 calls/s, p95 114 msabout 165 MB

As a starting point, plan one gateway worker, which uses at most one CPU, per 500 or so simple calls per second. Mappers, validation, masking, large responses and slow upstreams all change that.

Measure with your own APIs before going live. scripts/bench_api_gateway.py in the Studio repository runs a stand-in upstream and sends load to a gateway URL:

bash
python scripts/bench_api_gateway.py load --url https://api.example.com/gw/_/numbers/hello \
  --key hak_… --concurrency 32 --duration 60

It prints calls per second, latency percentiles and the status codes seen. Run it from another machine than the gateway. With k6:

javascript
import http from 'k6/http';
import { check } from 'k6';

export const options = { vus: 50, duration: '2m' };

export default function () {
  const r = http.get('https://api.example.com/gw/_/numbers/hello', {
    headers: { Authorization: `Bearer ${__ENV.KEY}` },
  });
  check(r, { ok: (res) => res.status === 200 });
}

Raise the API's rate limits for the test first, or its own limits will answer 429.

Settings​

VariableDefaultDescription
HRIDAAI_SERVICE—gateway runs only the API gateway
API_BUILDER_SELF_URLhttp://localhost:<PORT>Where Studio's agents send API calls; the service's internal URL when it runs
API_GATEWAY_CACHE_SECONDS15How long each process caches API definitions, versions, mappers, keys and settings; 0 turns caching off
API_GATEWAY_POOL_CONNECTIONS200Open upstream connections per process
API_GATEWAY_POOL_CONNECTIONS_PER_HOST50Per upstream host and port
API_GATEWAY_DNS_CACHE_SECONDS60How long an upstream's DNS answer is reused
API_GATEWAY_KEEPALIVE_SECONDS4How long an idle upstream connection is kept open
UVICORN_TIMEOUT_GRACEFUL_SHUTDOWN—On stop, how long calls in progress may finish

The service also needs Studio's DATABASE_URL, REDIS_URL, REDIS_KEY_PREFIX and HRIDAAI_SECRET_KEY, with ENABLE_DB_MIGRATIONS=false. The gateway's own behaviour (overload limits, the breaker, partner tokens, failed logins) is set with the variables in Gateway Policy & Security.

Troubleshooting​

  • /ready answers "waiting for Studio to migrate the database". Studio hasn't started on the new version yet. Upgrade Studio, or wait for it.
  • Every path answers 404. The service serves only gateway URLs. Studio's pages and APIs are on Studio's host.
  • A revoked key keeps working for a few seconds, or limits seem higher than set. The service isn't using Redis, or uses another REDIS_KEY_PREFIX than Studio.
  • Agent calls fail after enabling the service. Check that Studio can reach API_BUILDER_SELF_URL and that Studio was restarted, so registrations moved there.
  • Pods stay in ImagePullBackOff. The ghcr-hrida image pull secret is missing in the namespace, or its token can't read packages.
  • 503 overloaded or upstream_busy under load. A worker is at API_GATEWAY_MAX_IN_FLIGHT, or the API at its Calls at once: an upstream is slow, or the gateway needs more replicas. Watch hridaai.api_gateway.shed and event_loop.lag.
  • 502 upstream_blocked for an upstream that worked before. Its host name resolves to a loopback, link-local or metadata address, which the gateway refuses. See Gateway Policy & Security.
Hrida.ai is proprietary software of Zlabs Innovation. See the license for terms. © 2026 Zlabs Innovation.