Service Discovery in Microservices: DNS, Registries, Health Checks, and Load Balancing

Microservices rarely have permanent IP addresses.

Instances restart, containers move, autoscaling creates new replicas, unhealthy nodes disappear, and deployments replace old versions with new ones.

If one service hard-codes another service's address, the architecture stops being dynamic almost immediately.

Service discovery solves this by letting applications address a logical service name while infrastructure resolves that name to currently usable instances.

The Basic Problem

Imagine an Orders service needs Inventory.

Bad design:

http://10.0.4.27:8080

That address may disappear after a deployment.

What the caller really wants is:

inventory-service

and the platform should answer:

Which healthy Inventory instance should receive this request right now?

That is service discovery.

The Core Service Discovery Flow

A discovery system usually has three responsibilities:

1. Register service instances
2. Track which instances are healthy
3. Resolve a logical service name to usable endpoints

Conceptually:

Inventory instance starts
        ↓
registers as inventory-service
        ↓
health checks update status
        ↓
Orders asks for inventory-service
        ↓
resolver returns healthy endpoint(s)

The exact implementation may be DNS, an API-backed service registry, a load balancer, Kubernetes Services, a service mesh, or a combination.

DNS-Based Discovery

DNS is one of the simplest and most widely supported discovery mechanisms.

Instead of connecting to an IP address, applications connect to a stable hostname:

inventory.default.svc.cluster.local

Kubernetes automatically creates DNS records for Services and Pods so workloads can use consistent service names instead of ephemeral Pod IPs.

Why DNS is attractive

• virtually every application already knows how to resolve DNS
• application code does not need a special discovery SDK
• infrastructure can change endpoints behind a stable name

What DNS does not solve automatically

• stale resolver caches
• application connection pooling to removed endpoints
• advanced health/routing policy
• request-level load-balancing decisions
• topology-aware routing in every environment

DNS is simple, but you still need to understand how clients cache and reuse resolved addresses.

Kubernetes Service Discovery

Kubernetes provides service discovery as a built-in platform capability.

A Kubernetes Service gives a changing group of Pods a stable network identity.

Orders Pod
    ↓
inventory.default.svc.cluster.local
    ↓
Kubernetes Service
    ↓
healthy Inventory Pods selected by labels/endpoints

For many Kubernetes applications, this is enough.

You often do not need to deploy Consul, Eureka, ZooKeeper, or a custom service registry purely to discover services inside one cluster.

Use the platform capability before adding another distributed control plane.

ClusterIP Services

A normal Kubernetes Service commonly gets a stable virtual IP and DNS name.

Clients resolve the service name and connect to that logical service endpoint.

This is server-side discovery from the application's perspective:

client → stable Service address → platform chooses backend

The application does not need to know every Pod instance.

This keeps discovery logic out of business code.

Headless Services

Sometimes the client needs individual backend addresses rather than one virtual service address.

Kubernetes headless Services can expose endpoint DNS records directly.

This is useful for systems such as:

• stateful clusters
• databases
• custom client-side load balancing
• peer discovery

But exposing individual instances also pushes more responsibility to the client.

Use headless discovery because the protocol requires it—not as the default for ordinary HTTP services.

Service Registry-Based Discovery

Outside Kubernetes, or across heterogeneous infrastructure, a dedicated service registry can be useful.

A registry contains records like:

service: payments
instances:
  10.0.3.12:8080 healthy
  10.0.3.18:8080 healthy
  10.0.3.25:8080 unhealthy

HashiCorp Consul, for example, maintains a service catalog and can expose discovery through DNS or HTTP APIs.

AWS Cloud Map similarly maps logical names to cloud resources and allows discovery through DNS or APIs.

Registration: Who Adds Instances to the Registry?

There are two broad patterns.

Self-registration

The service instance registers itself when it starts and deregisters when it stops.

service process → registry

Advantages:

• service controls its own lifecycle

Risks:

• crashed services may fail to deregister
• application code is coupled to registry APIs

Third-party/platform registration

An orchestrator, sidecar, agent, or deployment system registers instances.

orchestrator/agent → registry

Advantages:

• application stays simpler
• infrastructure owns lifecycle

This is common in orchestrated platforms.

Health Checks Are Part of Discovery

A registry that returns dead endpoints is not useful discovery.

Consul supports health-aware service records and can use HTTP, TCP, and other checks to determine whether instances should remain eligible.

A health check should answer something concrete:

> Can this instance safely receive this class of traffic?

Do not make every health check perform deep dependency tests that create their own load or cause widespread flapping.

Liveness vs Readiness

These concepts are easy to mix up.

Liveness

Is the process stuck/dead and should it be restarted?

Readiness

Should this instance receive traffic right now?

A service can be alive but temporarily not ready while warming caches, loading configuration, or losing access to a required dependency.

Discovery/routing usually cares about readiness, not only process existence.

Client-Side Discovery

In client-side discovery, the caller queries a registry and chooses an instance itself.

Orders
  ↓
Registry → [Inventory A, B, C]
  ↓
Orders chooses B
  ↓
Inventory B

Benefits

• caller can implement custom load balancing
• topology/latency awareness can be sophisticated
• fewer intermediary proxies

Costs

• every language/client needs discovery logic
• health/caching/load-balancing behavior becomes application responsibility
• changing discovery systems may require client updates

This pattern is common in some RPC ecosystems but adds library complexity.

Server-Side Discovery

In server-side discovery, the client connects to a stable intermediary.

Orders
  ↓
Load Balancer / Service / Proxy
  ↓
Inventory replica

The intermediary handles backend selection.

Benefits

• clients stay simple
• discovery logic is centralized
• many client languages work without special SDKs

Costs

• adds another network component
• proxy/load-balancer capacity and availability matter

Kubernetes Services are a common example of this style from the application perspective.

Discovery and Load Balancing Are Related but Different

Service discovery answers:

> Which instances exist and are eligible?

Load balancing answers:

> Which eligible instance should receive this request?

A registry might return three healthy endpoints. A load balancer then chooses one based on:

• round robin
• least requests
• locality
• latency
• consistent hashing
• connection state

Many platforms combine these features, which is why the terms sometimes appear interchangeable.

DNS TTL and Caching Can Create Stale Endpoints

Suppose DNS says:

inventory.service → 10.0.3.12

That instance disappears, but an application/runtime caches the answer longer than expected.

The client may continue connecting to an invalid address.

Understand:

• DNS TTL
• language/runtime DNS cache behavior
• local resolver behavior
• connection-pool lifetime
• how quickly removed instances drain traffic

Service discovery is only as dynamic as the slowest cache in the resolution path.

Connection Pools Can Outlive Discovery Results

Even if DNS updates instantly, a client may reuse an existing HTTP/2, HTTP keep-alive, or database-style pooled connection.

That means discovery updates do not necessarily cause immediate traffic movement.

During deployments, use:

• graceful shutdown
• readiness removal before process termination
• connection draining
• termination grace periods

The correct sequence is often:

mark not ready
→ stop new traffic
→ drain in-flight work/connections
→ terminate

rather than killing the process first.

Avoid Registry Stampedes

If every request queries a central registry directly, the registry becomes part of the data path and may receive enormous load.

Use appropriate caching, DNS, local agents, sidecars, blocking queries, or platform abstractions.

Consul's design, for example, supports local agents and DNS/API discovery rather than requiring every application to query the server quorum on every business request.

A discovery control plane should not become a per-request database lookup.

Stale Discovery Can Be Better Than No Discovery

During a temporary registry outage, throwing away all previously known endpoints can make healthy services unreachable.

Some systems prefer cached/stale-but-recent discovery data when the control plane is unavailable.

This is a reliability trade-off:

stale endpoint risk
vs
complete service outage

The right policy depends on service churn and how quickly unhealthy endpoints are otherwise detected.

Multi-Zone and Multi-Region Discovery

At larger scale, discovery may include locality.

A caller might prefer:

same zone
→ same region
→ remote region only as fallback

Reasons include:

• latency
• cross-zone/region cost
• failure isolation
• data residency

Do not make every client implement this logic independently if the networking/service platform can provide it consistently.

Discovery Across Kubernetes and VMs

Hybrid environments are where dedicated registries become more interesting.

Example:

Kubernetes services
VM-based legacy services
cloud-managed databases
external internal APIs

Consul and AWS Cloud Map are examples of tools that can provide logical discovery outside a single Kubernetes-native environment.

The decision should come from topology complexity, not from wanting another infrastructure component.

Security Matters in Service Discovery

A service registry can reveal internal network topology.

Protect it with:

• authentication/ACLs
• private network exposure
• least-privilege read/write access
• TLS where needed
• controlled service registration

A compromised workload should not necessarily be able to register itself as:

payments-service

and receive sensitive traffic.

Identity and service registration need trustworthy ownership.

Do Not Treat Discovery as Authorization

Finding a service does not mean the caller is allowed to use it.

These are separate concerns:

Discovery → where is Payments?
Authentication → who is calling?
Authorization → may they call CapturePayment?

A private DNS name is not a security boundary.

Observe the Discovery Layer

Monitor:

• registration/deregistration rate
• unhealthy instance count
• DNS/query latency
• registry availability
• stale endpoint failures
• endpoint count per service
• load distribution
• connection failures during deployments

For incident debugging, you should be able to answer:

Which instances did the caller believe existed at that moment?

When You Do Not Need a Separate Service Discovery System

Do not install another registry automatically.

You may already have enough when:

• all services run inside one Kubernetes cluster
• Kubernetes DNS/Services satisfy routing requirements
• your managed platform already provides service names/load balancing
• there are few static services

A separate registry becomes more useful when:

• workloads span VMs, containers, regions, or orchestrators
• custom health/topology-aware discovery is required
• service mesh/networking architecture depends on it

Infrastructure should solve a problem you actually have.

Production Checklist

• Services use logical names rather than hard-coded instance IPs
• Registration/deregistration ownership is explicit
• Health/readiness determines endpoint eligibility
• DNS/client caching behavior is understood
• Connection draining happens during deployments
• Registry queries are not performed unnecessarily per request
• Discovery remains highly available
• Multi-zone/region routing policy is intentional
• Registry access is authenticated/authorized
• Discovery is not confused with service authorization
• Endpoint churn and resolution failures are observable
• A separate discovery system is introduced only if platform-native discovery is insufficient

Final Takeaway

Service discovery exists because service identity is stable while service instances are not.

In Kubernetes, built-in DNS and Services are often the simplest correct solution. In VM, multi-platform, or multi-region environments, a registry such as Consul or a managed discovery service such as AWS Cloud Map can provide a common service catalog and health-aware resolution.

The important part is not the product you choose. It is making registration, health, caching, load balancing, deployment draining, security, and failure behavior explicit.

A service should be able to move, scale, restart, and fail without forcing callers to know its IP address.

References

• Kubernetes: DNS for Services and Pods
• Consul: Service Discovery Overview
• Consul: DNS Discovery
• Consul: Health API
• AWS Cloud Map Documentation