API Gateway Architecture: Routing, Authentication, Rate Limits, Aggregation, and Failure Control

API Gateway Architecture: Routing, Authentication, Rate Limits, Aggregation, and Failure Control

An API gateway is often introduced as a simple box in front of microservices:

Clients → API Gateway → Services

In production, that box can become one of the most important components in the request path.

It may terminate TLS, authenticate clients, enforce rate limits, route requests, normalize protocols, aggregate responses, attach trace context, apply tenant policy, and protect downstream services from overload.

Used well, a gateway reduces duplicated edge logic and hides internal topology from clients.

Used badly, it becomes a new monolith, a latency bottleneck, or a single point where every product team fights over business logic.

What an API Gateway Should Do

The gateway is best suited to cross-cutting edge concerns.

Typical responsibilities include:

• request routing
• TLS termination
• authentication integration
• coarse authorization/policy enforcement
• rate limiting and quotas
• request-size limits
• header normalization
• protocol translation where needed
• observability and correlation IDs
• API version routing
• caching for selected safe responses

Microsoft's current architecture guidance describes three common gateway patterns: routing, aggregation, and offloading.

Those are useful mental models because they separate what the gateway is good at from what belongs in backend services.

Pattern 1: Gateway Routing

The gateway exposes one public endpoint and routes requests to internal services.

api.example.com/orders/*   → Orders Service
api.example.com/users/*    → Identity Service
api.example.com/catalog/*  → Catalog Service

Benefits

• clients do not need to know internal service locations
• backend topology can change without changing public URLs
• one place can enforce edge security and traffic policy

Risk

A routing layer becomes dangerous when route rules encode business decisions rather than infrastructure decisions.

Good:

/orders/* → Orders Service

Bad:

if premium customer and order total > 5000 and inventory region X...

That logic belongs in the domain layer.

Pattern 2: Gateway Offloading

A gateway can remove repetitive infrastructure work from every service.

Examples:

TLS termination
auth token verification
rate limiting
request limits
CORS
standard headers
WAF integration
basic response caching

This keeps domain services focused on business behavior.

But offloading does not mean moving all security into the gateway.

A service may still need resource-level authorization such as:

Can this user modify this order?
Does this order belong to this tenant?

The gateway can authenticate who the caller is; the owning service often remains responsible for deciding whether that identity can perform a domain action.

Pattern 3: Gateway Aggregation

Mobile and web clients sometimes need data from several services to render one screen.

Without aggregation:

Client → Orders
Client → Customer
Client → Payments
Client → Delivery

With aggregation:

Client → Gateway/BFF
              ├→ Orders
              ├→ Customer
              ├→ Payments
              └→ Delivery
         ← combined response

Microsoft's Gateway Aggregation pattern notes that this can reduce client chattiness and network round trips.

This is particularly valuable for high-latency mobile networks.

But aggregation has a trap

If every screen-specific workflow is implemented directly in the shared gateway, it can become a giant business layer.

For substantial aggregation logic, consider a dedicated Backend for Frontend (BFF) or aggregation service behind the gateway.

Authentication: API Keys Are Not Automatically Identity

An API key may identify an application or subscription, but that does not necessarily make it a safe authorization mechanism.

AWS explicitly warns that API Gateway usage-plan API keys should not be treated as authentication or authorization by themselves.

For user or workload identity, use appropriate mechanisms such as:

• OAuth/OIDC access tokens
• JWT validation
• mTLS/workload identity
• signed service credentials
• cloud IAM mechanisms

Then pass verified identity context downstream safely.

Authorization Has Multiple Layers

A useful split is:

Gateway-level authorization

Questions like:

Is this token valid?
Can this API key call the billing API?
Is this tenant plan allowed to access this route?

Service/domain authorization

Questions like:

Can user 42 edit order 99?
Does order 99 belong to tenant 7?
Can this role approve refunds above ₹10,000?

Do not duplicate complex business authorization rules independently in several gateways.

Keep the resource owner responsible for resource-specific policy.

Rate Limiting Protects More Than the Gateway

Rate limiting protects:

• backend capacity
• databases
• expensive third-party APIs
• tenant fairness
• abuse-sensitive endpoints

Common dimensions include:

requests / second
requests / minute
burst size
per user
per API key
per tenant
per route

AWS API Gateway uses token-bucket-style throttling and explicitly describes configured limits as best-effort targets rather than absolute ceilings.

That is an important design lesson: rate limits should not be your only financial or security boundary.

Use durable quotas/budgets where exact accounting matters.

Global Rate Limits Matter in Horizontally Scaled Gateways

Suppose you have 20 gateway instances and each allows:

100 requests/sec per tenant

If the limit is purely local, the real aggregate allowance can become much larger depending on traffic distribution.

For limits that must apply globally, use shared/global coordination or a rate-limit service designed for the purpose.

Envoy's documentation distinguishes local circuit breaking from global rate limiting for exactly these distributed-capacity scenarios.

Rate Limit Before Expensive Work

The order matters.

Prefer:

request
  ↓
authenticate
  ↓
validate basic request
  ↓
rate/quota check
  ↓
route downstream

Do not execute expensive aggregation, body transformations, or backend work before discovering that the caller is over limit.

Request Size Limits Are an Important Safety Control

Gateways should enforce bounded request sizes where possible.

This protects services from:

• accidental huge payloads
• memory pressure
• oversized uploads
• abuse

For large files, a better architecture is often:

client → signed object-storage upload
client → API with asset ID

rather than proxying multi-gigabyte files through the application gateway.

Do Not Retry Every Request at the Gateway

Gateway retries can improve transient-failure handling, but they can also duplicate side effects or amplify outages.

Safe retry candidates are typically:

• idempotent requests
• transient connection failures
• selected upstream errors

Dangerous candidates include:

POST /charge-card
POST /create-order

unless the downstream operation has explicit idempotency semantics.

Use:

• retry budgets
• maximum attempts
• exponential backoff
• jitter
• request deadlines

Envoy specifically recommends bounding retry volume to avoid large-scale cascading failure.

Timeouts Must Reflect the End-to-End Budget

If a client has a 2-second timeout but the gateway waits 10 seconds for a backend, the work may continue after the user has already abandoned the request.

Set:

client deadline
→ gateway timeout
→ downstream timeout

so each layer respects the remaining budget.

For aggregation, the slowest downstream can dominate total latency, so consider partial/degraded results only where product semantics permit it.

Circuit Breaking Protects Unhealthy Upstreams

A circuit breaker limits concurrent pressure against a failing or saturated dependency.

Possible controls include:

• maximum active requests
• maximum pending requests
• connection limits
• retry limits

If an upstream is clearly failing, continuously sending more requests can delay recovery.

But circuit breaking should be based on measurable capacity and failure modes—not added as a magic reliability checkbox.

Bulkheads Prevent One Backend From Consuming Everything

Suppose one slow service causes thousands of pending gateway requests.

Without isolation, that can consume connection pools, memory, or worker capacity needed for unrelated APIs.

Use separate limits/pools where appropriate:

Orders upstream pool
Payments upstream pool
Search upstream pool

A failure in search should not necessarily exhaust every connection available to checkout.

API Versioning Belongs at a Stable Boundary

A gateway is a natural place to route versions:

/v1/orders → Orders v1
/v2/orders → Orders v2

But avoid permanent proliferation of stale versions.

Use versioning only for genuinely breaking contracts and define deprecation timelines.

Internally, prefer backward-compatible evolution whenever possible.

Protocol Translation Can Simplify Clients

A public client may speak REST/JSON while internal services use gRPC.

Mobile/Web → HTTPS JSON → Gateway → gRPC → Service

This can be useful when the gateway has mature protocol support.

But protocol translation increases operational/debugging complexity, so do not add it when both sides can use the same simple protocol comfortably.

Caching at the Gateway: Powerful but Easy to Get Wrong

Good cache candidates:

• public configuration
• read-heavy product/catalog responses
• static metadata

Dangerous candidates:

• personalized data
• authorization-sensitive responses
• rapidly changing transactional state

A cache key may need to include:

path
query parameters
tenant
locale
auth scope
API version

If a tenant or authorization dimension is missing, a performance optimization can become a data leak.

Multi-Tenant Gateways Need Explicit Tenant Context

The gateway may resolve tenant identity from:

• hostname/domain
• access token claims
• API key mapping
• explicit path only after authorization

Then it should pass a trusted tenant identity downstream.

Do not simply forward a user-controlled header such as:

X-Tenant-ID: 999

and assume services can trust it.

If internal identity headers are used, strip client-supplied versions and generate them only after authentication.

Observability Should Start at the Gateway

Because most external traffic crosses the gateway, it is an excellent location to establish trace context.

Track:

• request count
• response status
• latency percentiles
• route/upstream
• rate-limit decisions
• auth failures
• retries
• circuit-breaker rejections
• request/response sizes

For aggregation, use one parent trace with child spans for each backend call.

Microsoft's aggregation guidance explicitly recommends distributed tracing with correlation IDs so individual downstream calls remain visible.

Preserve Upstream Error Semantics Carefully

Do not map every backend failure into:

{"error":"Something went wrong"}

Create stable error classes for clients while preserving safe diagnostic detail internally.

Examples:

401 unauthenticated
403 forbidden
404 resource missing
409 conflict
429 throttled
503 temporary dependency failure

Avoid leaking internal hostnames, stack traces, SQL errors, or credentials through gateway responses.

Health Checks Must Not Create a Failure Cascade

A gateway depends on many upstream services.

If its liveness endpoint fails whenever one optional backend is down, the orchestrator may restart healthy gateway instances continuously.

Separate:

• process liveness
• gateway readiness
• upstream health used for routing

An optional degraded dependency should not necessarily mark the whole gateway dead.

High Availability Is Mandatory for a Central Gateway

If every client request goes through the gateway, one gateway instance becomes an obvious single point of failure.

Production deployments normally need:

• multiple replicas
• load balancing
• health checks
• multi-zone placement where appropriate
• stateless configuration or replicated state

Global/multi-region APIs may additionally need regional routing and failover design.

Do not centralize the request path without making the central component highly available.

Configuration Should Be Versioned and Safely Rolled Out

A bad routing rule can break the entire platform without changing one line of application code.

Treat gateway configuration like production code:

• version control
• validation
• automated tests
• staged rollout
• rollback
• audit history

Avoid manual console changes that cannot be reproduced.

The Gateway Must Not Become a Business Monolith

Warning signs:

• product teams add domain logic directly to routing middleware
• gateway deploys are required for every feature
• one huge codebase transforms every response
• gateway owns cross-service transactions
• gateway directly queries service databases

The gateway should reduce coupling between clients and services, not create a new source of coupling between all teams.

For client-specific behavior, a BFF can often isolate concerns better.

API Gateway vs BFF

Shared API gateway

Best for platform-wide concerns:

auth
routing
rate limits
TLS
observability

Backend for Frontend

Best for client-specific composition:

mobile-specific response
web dashboard aggregation
partner API adaptation

A common architecture is:

Clients
  ↓
Shared edge gateway
  ↓
BFF / domain services

Do not force one shared gateway to understand every UI screen.

When You May Not Need an API Gateway

A dedicated gateway can be unnecessary when:

• the system has one small backend service
• there are few external endpoints
• your platform/load balancer already provides required routing/TLS
• there is no duplicated edge policy yet

Adding a gateway introduces another hop and another system to operate.

Use it when centralized edge concerns materially simplify the architecture.

Production Checklist

• Gateway responsibilities are explicitly defined
• Business logic remains in domain services/BFFs
• Authentication uses appropriate identity mechanisms
• Resource-level authorization remains enforced where the resource is owned
• Tenant context cannot be spoofed by client headers
• Rate limits protect important downstream resources
• Global vs local limit semantics are understood
• Request-size limits exist
• Retries are bounded and idempotency-aware
• Upstream timeouts respect end-to-end deadlines
• Circuit/bulkhead limits protect unhealthy dependencies
• Aggregation is traced per downstream call
• Cache keys include all correctness/security dimensions
• Gateway configuration is versioned and rollbackable
• Multiple gateway replicas eliminate the obvious single point of failure
• Error responses do not leak internal details

Final Takeaway

A production API gateway is a traffic and policy boundary, not a second application backend.

Use it to centralize routing, identity verification, throttling, protocol concerns, observability, and carefully selected aggregation/offloading. Keep resource authorization and business behavior with the services that own those domains.

Most importantly, design the gateway as a critical distributed-system component: bound retries, apply deadlines, isolate unhealthy upstreams, make rate limits explicit, trace every hop, and deploy configuration safely.

The gateway should make your backend easier to evolve—not become the one service nobody can change without fear.

References

• Microsoft: API Gateways in Microservices
• Microsoft: Gateway Aggregation Pattern
• AWS: Managing APIs for Microservices
• AWS API Gateway Throttling
• Envoy Circuit Breaking
• Envoy Global Rate Limiting