Building AI API Infrastructure: Gateways, Streaming, Tool Calls, Rate Limits, and Observability

Building AI API Infrastructure: Gateways, Streaming, Tool Calls, Rate Limits, and Observability

Connecting an application to one AI model is easy.

Operating AI APIs as production infrastructure is not.

As soon as multiple applications, teams, tenants, models, or providers share the same AI layer, you need more than an SDK call. You need authentication, quotas, routing, streaming, tool-call handling, usage accounting, retries, observability, capability checks, and controls that prevent one expensive workload from degrading everything else.

This is the difference between calling an AI API and building AI API infrastructure.

A Useful High-Level Architecture

A production AI API layer often looks like:

Applications / Agents
        ↓
AI Gateway / Router
        ↓
┌─────────────────────────────┐
│ Auth + API keys             │
│ Quotas / budgets            │
│ Capability validation       │
│ Provider routing            │
│ Streaming normalization     │
│ Usage accounting            │
│ Retry / fallback policy     │
│ Observability               │
└─────────────────────────────┘
        ↓
┌──────────┬──────────┬──────────┐
│Provider A│Provider B│Provider C│
└──────────┴──────────┴──────────┘

The gateway should not become a giant layer that hides every provider difference. Its job is to create a stable contract where stability is possible and expose differences where semantics genuinely differ.

Start With a Clear API Contract

Decide what your internal API promises.

For a text/agent API, the contract may include:

• text input/output
• multimodal input
• streaming
• function/tool calls
• structured output
• reasoning configuration
• usage information
• cancellation
• model selection
• metadata

The mistake is pretending every model supports every capability identically.

A better architecture keeps a capability registry:

model_x:
  streaming: true
  tools: true
  parallel_tools: true
  structured_output: true
  images: true

model_y:
  streaming: true
  tools: false
  structured_output: limited

Validate requests before sending them upstream.

Failing fast with a useful 400 is better than silently dropping a requested feature.

Normalization vs Pass-Through

There are two extremes when building a multi-provider API.

Full pass-through

Expose every provider's native request and response format.

Benefits:

• full provider feature access
• minimal translation risk

Costs:

• every application becomes provider-aware
• switching models requires application changes
• observability and accounting become fragmented

Full normalization

Force every provider into one universal schema.

Benefits:

• stable application contract
• easier routing

Costs:

• lowest-common-denominator behavior
• provider features get lost
• subtle semantic differences are hidden

A practical design is usually in the middle:

stable common core
+
capability-aware extensions
+
explicit provider escape hatch when needed

Do not normalize away information that clients need for correctness.

Streaming Is a Protocol, Not a Boolean

It is tempting to model streaming as:

{"stream": true}

But production streaming contains multiple event types.

Modern AI APIs can stream:

• text deltas
• tool-call arguments
• reasoning/thought metadata where exposed
• citations
• status transitions
• usage
• errors
• completion events

OpenAI's Responses API, for example, uses typed Server-Sent Events such as response lifecycle and content-delta events. Google's current Interactions API similarly uses step-oriented SSE events for text, function calls, tool results, and status transitions.

Your gateway should model streaming as a typed event stream.

Bad abstraction:

upstream bytes → concatenate → send to client

Better abstraction:

upstream event
    ↓
parse provider event
    ↓
normalize semantic event
    ↓
stream to client

This makes cancellation, tool calls, usage accounting, and errors much safer.

Never Assume Streaming Errors Arrive as HTTP Errors

Before a stream starts, an upstream provider may return a normal HTTP error.

After the stream has started, an error may arrive inside the SSE stream.

That distinction matters because the gateway can no longer change the already-sent HTTP status code.

Your stream protocol therefore needs an error event that clients understand.

Conceptually:

HTTP 200 stream opened
      ↓
text.delta
      ↓
tool.delta
      ↓
error { code, message }
      ↓
stream closes

Clients must not treat “HTTP connection opened successfully” as “the AI request succeeded.”

Tool Calls Need a State Machine

Tool calling is not merely parsing JSON from a model.

A production tool workflow looks like:

model request
   ↓
tool call(s)
   ↓
validate arguments
   ↓
authorize tool
   ↓
execute outside model
   ↓
record tool result
   ↓
return result to model
   ↓
continue generation

The model proposes an action. Your application executes it.

This separation is fundamental.

For every tool call preserve:

• tool/function name
• unique call ID
• argument payload
• lifecycle state
• execution result
• error result
• parent AI request/trace ID

When providers stream tool arguments incrementally, aggregate deltas until the call is complete before executing it.

Never execute half-parsed streamed JSON.

Tool Authorization Must Be Independent of Model Choice

Do not encode permissions in the prompt alone.

A model being told:

"never delete production data"

is not an authorization system.

Enforce tool policy in trusted code:

model requests delete_database
        ↓
policy engine checks user + tenant + tool + resource
        ↓
allow / require approval / deny

The provider should be replaceable without changing the security boundary.

Structured Output Requires Capability Checks

Structured output is increasingly useful for workflows that need predictable machine-readable responses.

But providers and models differ in:

• JSON Schema support
• supported schema subset
• strictness guarantees
• streaming representation
• interaction with tool calls

Do not translate every response_format into whatever looks approximately similar upstream.

Instead:

1. Validate whether the selected model supports the requested mode.
2. Validate schema restrictions.
3. Send the provider-native representation.
4. Validate the final result again before trusted downstream use.

Model-generated JSON should still be treated as untrusted input to your application.

Rate Limits Are Multi-Dimensional

Traditional APIs often think in requests per second.

AI workloads add token and cost dimensions.

A useful quota system may track:

requests / minute
tokens / minute
concurrent requests
input tokens
output tokens
tool calls
model-specific spend
daily/monthly budget

Current Gemini documentation, for example, describes limits across request and token dimensions, with some limits also dependent on model, project, tier, or spend.

Your internal quota layer should therefore avoid assuming:

1 request = 1 unit of load

A 500-token classification request and a 200,000-token agent run are not equivalent.

Put Admission Control Before the Expensive Work

Do not discover that a tenant exceeded quota after the model has already generated the answer.

A better flow:

request
  ↓
authenticate
  ↓
validate model/capability
  ↓
check quota / concurrency / budget
  ↓
reserve estimated capacity
  ↓
call provider
  ↓
reconcile actual usage

Some token usage is only known after completion, so accounting usually needs both reservation/estimation and final reconciliation.

Backpressure Protects the Entire Platform

When upstream capacity is saturated, blindly accepting every request creates runaway queues and retry storms.

Use bounded controls:

• per-tenant concurrency
• global concurrency
• queue limits
• request deadlines
• overload responses
• circuit breakers

A fast 429 or 503 can be healthier than keeping an interactive request queued for several minutes.

Backpressure is part of reliability, not a user-experience failure.

Retries Must Be Error-Aware

Retries are useful for transient failures such as some rate-limit or server-side errors.

They are harmful for permanent failures such as:

• invalid schema
• unsupported capability
• bad credentials
• malformed tool definition
• context too large

A robust retry policy uses:

error classification
+
exponential backoff
+
jitter
+
maximum attempt count
+
overall request deadline

Also respect provider retry headers when they are available.

Without jitter, thousands of clients can retry at the same instant and recreate the outage wave.

Fallbacks Must Preserve Semantics

“Provider A failed, use Provider B” sounds simple.

It is not.

The fallback model might differ in:

• context window
• tool support
• structured-output support
• image support
• safety behavior
• reasoning controls
• tokenization
• output format
• price

A fallback is safe only when the replacement model satisfies the current request's capability contract.

Conceptually:

request requires:
  tools + strict schema + images

fallback candidates:
  model B: tools + schema + images ✅
  model C: text only ❌

Routing must be capability-aware.

Do Not Retry Non-Idempotent Tool Work

AI generation can often be retried.

External actions may not be.

Imagine:

model → charge_customer tool

The tool succeeds, but the response back to the model times out.

Blindly replaying the full agent step can charge the customer twice.

Use idempotency keys and durable tool-call state for side-effecting actions.

The AI gateway and agent runtime should be able to distinguish:

model inference retry

from:

business action replay

Usage Accounting Must Be Durable

If customers are billed or limited by AI usage, do not rely only on ephemeral logs.

Persist an immutable or append-oriented usage record containing fields such as:

request_id
tenant_id
api_key_id
provider
model
input_tokens
output_tokens
cached_tokens where applicable
request status
started_at
completed_at
estimated cost

The exact provider usage fields differ, so preserve native details where useful while also storing normalized totals.

Reconciliation is important when streams disconnect or provider responses arrive incompletely.

Cost Routing Should Not Destroy Quality

A router can select models based on:

• capability
• latency
• cost
• provider health
• tenant plan
• workload type

For example:

classification → small/fast model
complex coding → stronger reasoning model
embeddings → embedding model
image task → multimodal-capable model

But never route only by cheapest price.

A low-cost model that causes retries, bad tool calls, or poor task completion can be more expensive end-to-end.

Optimize cost per successful task, not only cost per token.

Observability Needs AI-Specific Dimensions

Traditional HTTP metrics are necessary but insufficient.

Track:

Request-level

• provider
• model
• latency
• status/error class
• retry count
• fallback count
• input/output tokens
• streaming vs buffered

Agent/tool-level

• tool calls per request
• tool failures
• approval latency
• tool execution duration
• agent turns/iterations

Platform-level

• requests per tenant
• token throughput
• concurrency
• queue depth
• provider error rate
• cost
• cache hit rate where applicable

OpenTelemetry provides standardized semantic-convention work across common telemetry domains, including GenAI-specific conventions maintained by the OpenTelemetry ecosystem.

Standard naming reduces the pain of instrumenting every provider differently.

Preserve Provider Request IDs

When an upstream provider returns a request identifier, capture it.

Your own gateway should also issue a stable request ID.

A useful trace chain is:

client_request_id
      ↓
gateway_request_id
      ↓
provider_request_id
      ↓
tool_call_id(s)

This makes production debugging dramatically easier when a user reports one failed interaction among millions of successful calls.

Be Careful Logging Prompts and Model Outputs

AI telemetry can contain:

• personal data
• API keys pasted by users
• source code
• private documents
• medical or financial information
• tool outputs

Do not make full prompt logging the default simply because it helps debugging.

Prefer:

• opt-in payload capture
• redaction
• short retention
• tenant-specific controls
• encrypted storage
• access auditing
• hashed or summarized diagnostics where possible

Observability should not create a second sensitive-data warehouse by accident.

Streaming Usage Needs Finalization Logic

With buffered responses, usage often arrives with the final response object.

With streaming, your client may disconnect early or the stream may fail before final usage is received.

Your accounting system needs states such as:

reserved
in_progress
completed
incomplete
failed
reconciled

Do not silently count every disconnected stream as zero usage.

Likewise, do not bill an estimate as exact usage without recording that distinction.

Version Your Internal AI Contract

Provider APIs evolve quickly.

New event types, tools, models, and output structures appear regularly.

Avoid forcing every application to upgrade simultaneously.

Version stable behaviors:

/api/ai/v1/...

and make breaking changes explicit.

Internally, use typed provider adapters so one provider's new field does not leak unpredictably into every consumer.

Test the Weird Cases

A happy-path text completion proves almost nothing about an AI gateway.

Test:

• stream disconnect mid-token
• stream error after HTTP 200
• malformed provider event
• tool call split across many deltas
• parallel tool calls
• unsupported tool model
• invalid structured-output schema
• rate limit with retry header
• provider timeout after tool side effect
• usage missing on interrupted stream
• fallback candidate lacking capability
• cancellation while upstream is running
• client reconnect behavior
• very large context

Production reliability lives in these edges.

A Practical Production Flow

1. Authenticate request
2. Resolve tenant / API key policy
3. Validate model + capabilities
4. Estimate and reserve quota
5. Apply concurrency/admission control
6. Route to provider adapter
7. Parse typed stream or buffered response
8. Execute authorized tools if required
9. Capture usage + provider request IDs
10. Persist accounting
11. Emit traces/metrics
12. Reconcile reservation with actual usage

Every stage should have an explicit failure mode.

Production Checklist

• Common API contract is versioned
• Provider capabilities are explicit
• Unsupported features fail fast
• Streaming uses typed semantic events
• Mid-stream errors are represented correctly
• Tool arguments are fully assembled before execution
• Tool authorization is enforced outside the model
• Side-effecting tools use idempotency controls
• Quotas include token/concurrency dimensions
• Admission control happens before provider execution
• Retries are bounded, jittered, and error-aware
• Fallbacks are capability-aware
• Usage accounting is durable
• Provider request IDs are preserved
• Prompt/output logging follows data-minimization rules
• Cost and task-quality metrics are monitored together
• Cancellation and interrupted streams are tested

Final Takeaway

AI API infrastructure is an execution control plane between applications and probabilistic, expensive, fast-changing model services.

The strongest systems do not pretend every model is identical. They provide a stable core while preserving capability differences, stream typed events correctly, enforce tool and tenant policy outside the model, account for usage durably, and apply backpressure before overload becomes an outage.

Start simple with one provider adapter and one stable internal contract. Add routing, fallbacks, and multi-provider normalization only when there is a real reliability, capability, or cost reason to do so.

Official References

• OpenAI Responses Streaming Events
• OpenAI API Quickstart
• Gemini API Rate Limits
• Gemini Function Calling
• Gemini Streaming
• Gemini API Errors
• OpenTelemetry Semantic Conventions