
A service instance can be replaced five times during a deployment. If another service stores its IP address in configuration, every replacement becomes an outage. Service discovery gives callers a stable name and a way to find healthy, current instances without hard-coding ephemeral addresses.
This guide compares DNS-based, registry-based, client-side, and server-side discovery, then explains how Kubernetes Services fit the model. Continue from load balancing, health checks, and the Backend Interview Mastery series.
Table of Contents
Open Table of Contents
The Short Answer
Use DNS when a stable name and platform-managed routing are enough. Use a service registry when instances, metadata, health, zones, or versions must be queried dynamically. Client-side discovery makes the caller choose an instance; server-side discovery puts that choice in a load balancer, proxy, or service mesh.
Discovery is not load balancing, health checking, or authorization, although a production implementation usually coordinates with all three. The contract is: resolve a logical service to eligible endpoints, tolerate stale answers, and refresh when topology changes.
| Model | Selection happens in | Strength | Cost |
|---|---|---|---|
| DNS | Resolver/client or platform | Simple, ubiquitous | TTL and caching can be stale |
| Client-side registry | Calling service | Rich routing and low proxy overhead | Every client needs discovery logic |
| Server-side registry | Load balancer/proxy | Consistent policy and simpler clients | Extra hop and platform dependency |
| Kubernetes Service | Cluster networking/control plane | Stable virtual IP and DNS | Must understand selectors, endpoints, and readiness |
The Discovery Contract
flowchart TD
A[Service instance starts] --> B[Register name, address, zone, version]
B --> C[Health check or lease]
C --> D{Healthy?}
D -->|Yes| E[Eligible endpoint set]
D -->|No| F[Remove or quarantine endpoint]
G[Client] --> H[Resolve logical service name]
H --> E
E --> I[Connect to selected instance]
A useful record includes an address, port, protocol, zone, version, capacity hints, and expiration. Registration must be ephemeral or lease-based when instances can disappear without a clean shutdown. Deregistration on shutdown is helpful, but it cannot be the only safety mechanism; crashed processes never get to send the goodbye message.
DNS-Based Discovery
DNS maps a logical name such as payments.default.svc.cluster.local to a service address. Kubernetes publishes Service records through CoreDNS, and clients resolve the name instead of tracking pod IPs. DNS is operationally familiar, cacheable, and language-neutral.
The trade-off is staleness. A resolver, sidecar, or client library may cache a record until its TTL expires. Very short TTLs improve convergence but increase DNS load; long TTLs reduce lookup cost but send traffic to removed or overloaded instances. Retries should re-resolve when connection failures indicate that a cached endpoint is stale.
Service Registries
A registry such as Consul, etcd-backed control plane, or a platform registry stores live instances and metadata. Clients can query payments in us-east-1 with version or capability filters, then choose an endpoint using round robin, weighted routing, or least loaded selection.
Client-side discovery reduces data-plane hops, but every language needs correct caching, refresh, health, and fallback code. Server-side discovery centralizes that policy in a proxy or load balancer. The proxy becomes a critical platform dependency and must be scaled and observed like any other production service.
Kubernetes Services
Kubernetes separates the stable Service identity from changing Pods. A selector maps the Service to ready endpoints; kube-proxy or an eBPF dataplane routes traffic, while CoreDNS resolves the Service name. A ClusterIP is a virtual internal address, a headless Service returns pod addresses for clients that need their own balancing, and LoadBalancer integrates with an external provider.
apiVersion: v1
kind: Service
metadata:
name: payments
spec:
selector:
app: payments
ports:
- name: http
port: 80
targetPort: 8080
The selector is part of the discovery contract. A typo can produce a perfectly valid Service with zero endpoints. Readiness probes should remove an instance before it receives traffic; liveness probes should restart a process that cannot recover. The health-check guide explains why those signals must not be conflated.
Health, Staleness and Failure
Discovery can return an endpoint that becomes unhealthy immediately after the lookup. Clients still need connection timeouts, retries with a budget, and a circuit breaker. Never treat successful registration as proof that the application is ready; registration and readiness should be separate states.
Measure lookup latency, cache age, endpoint churn, empty result sets, connection failures by endpoint, and time from instance start to eligible traffic. During incidents, compare registry membership with actual health and load-balancer observations. A registry that says every instance is healthy while requests fail is itself a production dependency to debug.
Choosing an Architecture
For a small deployment, DNS plus a load balancer is often sufficient. Add a registry or mesh when you need zone-aware routing, version canaries, rapid membership updates, or consistent cross-language policies. Keep the discovery plane out of the synchronous request path where possible: cached answers and last-known-good endpoints let a brief control-plane outage avoid an immediate data-plane outage.
Interview Questions
1. Why is service discovery needed in microservices?
Instances are dynamic because of autoscaling, deployments, failures, and scheduling. Hard-coded addresses become invalid and force configuration changes for every replacement. Discovery gives callers a logical service name and a current set of endpoints.
2. What is the difference between service discovery and load balancing?
Discovery answers which endpoints exist and are eligible; load balancing chooses one for a request and distributes traffic. A DNS resolver can provide discovery while a client or proxy balances, and a Kubernetes Service provides both a stable virtual identity and dataplane routing. Keeping the responsibilities distinct makes failures easier to diagnose.
3. How do you handle stale DNS or registry entries?
Use bounded TTLs or leases, health checks, connection timeouts, and refresh on failed connections. Keep a last-known-good set for short control-plane outages, but do not keep sending traffic indefinitely to unhealthy instances. Record endpoint age and refresh failures so staleness is observable.
4. Why can a registry be a single point of failure?
If every request requires a live registry lookup, a registry outage can prevent otherwise healthy services from communicating. Cache answers, use replicated control-plane nodes, and keep the data plane able to use a bounded last-known-good set. This trades perfect convergence for availability during a short control-plane failure.
5. When would you use a headless Kubernetes Service?
Use it when clients need individual pod addresses, such as stateful databases, shard-aware clients, or systems with their own discovery and balancing. For ordinary stateless HTTP, a normal ClusterIP keeps clients insulated from pod churn and centralizes routing.
Conclusion
Service discovery replaces fragile instance addresses with a logical name, an endpoint set, and a convergence contract. DNS and Kubernetes Services are strong defaults; registries and client or server-side policies add richer routing when the platform needs them. The production design must combine leases, readiness, timeouts, caching, and observability so discovery failures do not become application-wide outages.
References
- Kubernetes: DNS for Services and Pods
- Kubernetes: Using CoreDNS for Service Discovery
- HashiCorp Consul: Service Discovery
YouTube Videos
- “DNS in Kubernetes: Service Discovery with CoreDNS Explained” - The Code Lucky [https://www.youtube.com/watch?v=eKLfqAwMMGM]