Production Kubernetes Setup: HA, Security, Networking, Storage, and Recovery

Production Kubernetes Setup: HA, Security, Networking, Storage, and Recovery

A production Kubernetes cluster is not defined by whether Pods start. It must survive failures, protect the control plane, isolate workloads, and recover state predictably.

Managed vs Self-Managed

Kubernetes's own production guidance recommends deciding how much control-plane work you really want to own.

Managed Kubernetes usually removes much of the control-plane patching and HA burden. Self-managed Kubernetes means you own API servers, etcd, certificates, upgrades, load balancing, and recovery.

If Kubernetes operations are not part of your product advantage, managed control planes are often the simpler default.

Make the Control Plane Highly Available

For critical self-managed clusters, do not depend on one control-plane node. Kubernetes production guidance recommends replicated control-plane components and load balancing across API servers. kubeadm's HA guidance uses three-member etcd examples and odd-member consensus topologies.

Back Up etcd—and Test Restore

etcd is Kubernetes' backing store for cluster state. Current Kubernetes docs explicitly recommend a backup plan.

A real recovery plan includes:

• scheduled snapshots
• protected off-node storage
• restore documentation
• periodic restore drills

A backup you have never restored is only an assumption.

Choose Networking Deliberately

Your CNI should match requirements for NetworkPolicy, routing, IPv4/IPv6, performance, and observability.

If you use NetworkPolicy, verify the selected network plugin actually enforces it. Kubernetes documents that NetworkPolicy depends on compatible network-plugin support.

A sensible security baseline often restricts unnecessary east-west traffic rather than allowing every Pod to talk to every other Pod.

Treat DNS as Critical Infrastructure

Service-to-service traffic often depends on CoreDNS. Monitor DNS latency, errors, and capacity. A cluster can have healthy application Pods and still look broken if service discovery is slow or unavailable.

Set Resource Requests and Limits

The scheduler uses requests to place Pods, while kubelet enforces configured limits. Without realistic requests, cluster capacity and autoscaling decisions become misleading.

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    memory: 512Mi

Use ResourceQuota and LimitRange in shared namespaces where one team or workload should not consume unlimited cluster resources.

Enforce Workload Security

Kubernetes recommends Pod Security Standards and least privilege. For normal applications, evaluate controls such as:

• run as non-root
• disable privilege escalation
• drop unnecessary capabilities
• seccomp
• avoid privileged/host namespace access
• read-only root filesystem where practical

Use dedicated ServiceAccounts and avoid automatically mounting API tokens when the workload does not need Kubernetes API access.

Protect Secrets and the API

The Kubernetes API is the control surface for the entire cluster. Use strong authentication, RBAC, limited admin access, TLS, and private/restricted endpoints where possible.

Also remember that Kubernetes Secrets require deliberate protection, including RBAC and encryption-at-rest strategy where appropriate.

Storage Is Not Backup

A PersistentVolume keeps data across Pod restarts; it does not automatically provide disaster recovery.

For stateful workloads, define:

• storage durability
• snapshots/backups
• restore procedure
• zone/failure-domain behavior
• application-level consistency requirements

Databases still need database-aware backup and recovery planning.

Make Deployments Failure-Aware

Use readiness, startup, and liveness probes for different purposes. Remove Pods from traffic before termination and allow in-flight work to drain.

Spread critical replicas across nodes/zones so three replicas do not accidentally live on one failure domain.

Observe and Upgrade the Cluster

Monitor:

• control-plane/API health
• etcd
• node pressure
• Pod restarts/OOMs
• scheduler failures
• DNS
• storage
• ingress/gateway
• workload SLOs

Have a tested Kubernetes upgrade path and validate compatibility of CNI, CSI, ingress/gateway, and admission components before major upgrades.

Production Checklist

• Control plane is HA or managed
• etcd backups and restores are tested
• CNI and NetworkPolicy behavior is verified
• CoreDNS is monitored
• Workloads have realistic resource requests
• Namespace quotas/limits exist where useful
• Pod Security and least privilege are enforced
• API access and Secrets are protected
• Stateful workloads have real backups
• Critical replicas span failure domains
• Cluster upgrades and rollback/recovery are documented

Final Takeaway

Production Kubernetes is mainly about failure boundaries and operational discipline. Make the control plane resilient, protect etcd, define workload resources and security, verify networking, back up stateful data, and test recovery before you need it.

References

• Kubernetes Production Environment
• Operating etcd Clusters
• Kubernetes Security
• Resource Management
• NetworkPolicy