CYGNABYTE Let’s build →

Scaling an AI SaaS backend with Go

Model calls are slow.
Build a backend that waits well.

What breaks when an AI SaaS product meets real traffic, and the Go, Kafka, Redis and PostgreSQL patterns that keep latency, model costs and noisy tenants under control. Written for founders and technical leads.

Published

Go concurrencyKafka backpressurep99 budgets
Cover: Scaling an AI SaaS backend with Go
Part 01

Why AI SaaS backends fail at scale

Most backends are sized for fast requests: read a row, write a row, respond in milliseconds. An AI feature changes the shape of the load. A model call that writes a long answer can hold its request open for seconds, along with a connection, memory for the context and sometimes a database connection nobody released.

Queueing math does the rest. Requests in flight equal the arrival rate multiplied by the time each one spends in the system (Little’s law). Suppose, purely as an example, that 50 AI requests arrive each second and each call takes 8 seconds: about 400 are open at once. If the provider slows to 20 seconds, the same traffic means 1,000, with no change on your side.

How it usually breaks

  • Open requests exhaust connections and memory, so even fast endpoints time out.
  • Clients retry the timeouts and multiply the load at the worst moment.
  • The provider returns rate-limit errors, and naive retries turn them into a retry storm.
  • One tenant’s batch job takes the shared capacity, and every other customer waits.
Key point: model latency, not CPU, is usually what piles up. The backend has to be designed to wait well.
Part 02

Stateless Go services and autoscaling

Go suits this workload because waiting is light. A goroutine blocked on network I/O uses a few kilobytes of stack, not an operating-system thread, which is why one Go node can hold hundreds of thousands of open connections. Python still fits document processing and data work; on the hot path, lightweight waiting is what matters.

Deadlines and limits on every call

Pass a context with a deadline through every handler. A request past its deadline stops waiting, and when a user closes the tab, the canceled context aborts the model call instead of letting it generate tokens nobody will read. Cap in-flight calls per dependency with a semaphore, so a slow provider fills its own lane and leaves room for sign-in, billing and everything else.

Stateless pods, scaled on the right signal

Keep sessions, counters and conversation state in Redis and PostgreSQL, never in a pod’s memory, so any pod can serve any request and Kubernetes autoscaling can add pods as load climbs and remove them as it falls. Scale AI-heavy services on in-flight requests or queue depth: a pod waiting on a model looks idle on a CPU graph while its users wait. On scale-down, drain open streams first, or answers get cut off mid-sentence.

Part 03

Queues and backpressure with Kafka

Split work by how long a user will wait. Interactive answers stream back on the request. Long jobs, such as document processing, bulk summarization or multi-step agent runs, go on a queue: the API records the job, publishes an event and returns a job ID at once, and the client follows progress over its live connection.

With Kafka, every write is an event and each consumer pulls work at its own pace, so services scale independently and a slow consumer never takes down a fast one. When workers fall behind, the backlog waits in the log and shows up as consumer lag, not as failed requests. That is the first half of backpressure; the second is deciding what producers do when lag keeps growing.

Decisions to make early

  • Key events by tenant so each tenant’s events stay in order, and watch for hot partitions from one large tenant.
  • Assume at-least-once delivery: give every event an ID and make consumers idempotent.
  • Write a database change and its event together, for example through an outbox table, so a crash cannot lose either.
  • Decide what happens at the limit. A clear “try again shortly” beats an unbounded queue nobody watches.
Part 04

Rate limits per provider and per tenant

Two sets of limits apply. Providers limit you, usually in requests and tokens per minute. You limit tenants, so one customer cannot degrade the product for everyone else. Both belong in Redis, where every pod reads and updates the same counters atomically.

Provider limits

Keep a token bucket per provider and model that counts tokens as well as requests, because two requests can differ enormously in tokens: estimate input tokens before the call, then reconcile with reported usage. On a rate-limit error, honor the retry-after hint and back off with jitter. A circuit breaker per provider stops traffic to a provider that keeps failing and routes it to a fallback model behind your model-agnostic layer, a cached answer or the queue.

Tenant limits

Per-tenant limits follow the customer’s plan: requests per minute, tokens per day, concurrent jobs. They also stop runaway loops, such as an agent calling the same tool over and over. Check the limit before the model call and return an error the interface can explain, not a timeout.

Watch for: a limit enforced in only one place. If the API checks it but the job workers do not, queued work bypasses it.
Part 05

Streaming over WebSockets, and caching

Streaming leaves the model’s speed unchanged, but users read the first words while the rest is generated. Track time to first token apart from total response time: users judge the first, and your capacity depends on the second.

A gateway built for connections

A one-way HTTP stream is enough when the client only listens. WebSockets earn their place when the client also talks back on the same connection, for example to cancel a generation or send the next message, while answers, notifications and live updates flow the other way. Run them through a dedicated Go gateway with pub/sub fan-out: the worker producing an answer is rarely on the pod holding the user’s socket, so it publishes chunks to a Redis channel and the gateway forwards them. Add heartbeats and a per-connection buffer limit, so a slow client cannot hold unbounded memory. Redis pub/sub only reaches current subscribers, so keep recent chunks in a short-lived store, and a client that reconnects can resume where it left off.

Caching that stays safe

  • Cache repeated, non-personal answers under a key built from the normalized input, model, prompt version and tenant.
  • Cache expensive steps, such as retrieval results and parsed documents, separately from final answers.
  • Put the stable part of a prompt first, so providers that support prompt caching can reuse it.
  • Never share an entry across tenants, and expire entries when source data or prompts change.
Part 06

Scaling PostgreSQL in stages

PostgreSQL carries an AI SaaS product a long way when it is set up for growth from day one. Each stage buys headroom; prepare the next before you need it.

Pooling and replicas

Autoscaled pods multiply database connections. Size pools per pod, put a connection pooler in front of the database, and never hold a connection while waiting on a model: read, release, call the model, then write in a short transaction. Send history, dashboards and search to read replicas, but route reads that must reflect a user’s own recent write to the primary, because replication lag is real.

Partitioning and a sharding path

Messages, usage events and audit logs grow fastest. Partition them by time so indexes stay small and retention means dropping a partition, not running a long delete. Put tenant_id on every table, unique index and query from the first migration; if volume or one large tenant ever demands sharding, it becomes largely a data move instead of a rewrite. Keep analytics on a store built for it, not on the production database.

Your next action: check that every query in your codebase can answer one question: which tenant is this for?
Part 07

Multi-tenancy and idempotent usage metering

Tenant isolation has to hold beyond the database. Take the tenant from the authenticated session, never from the request body, enforce it in the data layer (PostgreSQL row-level security is one option), and carry it into cache keys, retrieval filters, queue messages and logs.

Count each model call once, even when events repeat

Each model call emits a usage event with its own call ID, plus tenant, feature, model, input and output tokens, cost and the request it belongs to. Delivery is at-least-once, so consumers deduplicate on the call ID, then roll events up into per-tenant counters for limits, dashboards and billing. Keying on the request ID instead would undercount: one request can make several billable model calls, such as agent steps or retries.

Bill without double-charging

When usage flows to Stripe, send each report with an idempotency key derived from your own usage record, so Stripe discards a retried report instead of counting the usage twice. Verify webhook signatures, store processed event IDs and treat duplicates as no-ops. Reconcile usage totals against invoices on a schedule, so a gap shows up in your report before a customer finds it.

Part 08

Observability and load testing at target concurrency

Metrics, traces and alerts ship with the first release, not after the first incident. Measure at least:

  • p50, p95 and p99 latency per endpoint and per provider, plus time to first token.
  • In-flight requests, queue lag and open connections per pod.
  • Rate-limit errors and circuit-breaker state for each provider.
  • Tokens and cost per request, cache hit rate and cost per successful task, by tenant and feature.

Trace each request from the gateway to the provider with prompts and personal data redacted, and alert on what users feel, such as p99 over budget or a growing queue, not on CPU alone.

Load-test against p99 budgets

Write the workload down first: active users, simultaneous requests, model latency, data volume and cost limits. Then load-test at that target concurrency, with the p99 budgets set as hard thresholds in the harness, so a regression fails the run. A stub provider with realistic latency, streaming and rate-limit errors lets you test at full scale without paying for tokens; a smaller run against the real providers confirms their limits. Include spike, soak and failover scenarios. Capacity claims need measurements; there is no universal user-count guarantee.

Part 09

How Fusebox keeps it shippable

All of this adds moving parts, and moving parts are where builds stall. We build these backends with Fusebox: the system is split into small modules, or fuses, each with a written contract and a harness that proves it works. Engineers build fuses in parallel, and nothing is wired in until it passes.

The fuses in an AI SaaS backend

  • API gateway (authentication, tenant resolution, limits): contract tests and limit tests.
  • Model gateway (provider adapters, token buckets, breakers, fallbacks): a stub provider that injects latency, rate limits and outages.
  • WebSocket gateway: simulated clients that disconnect, reconnect and read slowly.
  • Job workers and usage metering: replayed and duplicated events that must leave results and totals unchanged.

Passing fuses are wired together, load-tested at target concurrency, deployed to your cloud on AWS or GCP and monitored for cost, latency and quality, and what you learn goes back into the contracts. Because each engineer works to a contract rather than waiting on someone else’s code, adding people speeds the build up, and integration is the calm part.

Where to go next

See how we approach AI SaaS development and our Golang development services. After the two-week AI Proof, a Go backend typically takes 6–12 weeks and an AI SaaS platform v1 10–16 weeks, confirmed in a scoped proposal. Planning a first release? Read how long it takes to build an AI MVP.

Explore AI SaaS development →

Start with the idea

Plan the backend before the traffic arrives.

Tell us about your product, the AI features on your roadmap and the load you expect. We’ll help you define a useful next step.