KAI Road of Kubernetes 07 — StatefulSets: replicas are no longer interchangeable

The previous chapter separated a Pod’s lifetime from the lifetime of its data.

Pods can move, but data with business meaning should not disappear with them.

That only solves half of the problem.

Imagine three replicas: one performs leader duties, one owns shard A, and one owns shard B.

It is no longer enough to say: “just attach storage.”

The next questions are: Which replacement Pod should recover which data? How do peers find that exact member? Does startup order matter?

The sentence I would keep is this:

A Deployment manages interchangeable replicas. A StatefulSet manages named members with stable positions and their own data.

A StatefulSet does not create immortal Pods

StatefulSet Pods still crash, get evicted, move to another node, and get recreated during updates.

The controller does not make the process immortal. What it provides is a sticky identity.

The Pod can be replaced, but the member position it represents must remain recognizable.

If web-0 fails, the controller creates another web-0.

The new Pod may have a different IP and run on another node, but it can still use the name web-0, return to the same DNS identity, and mount the PVC assigned to web-0.

StatefulSet is not mainly about preventing change. It is about matching identity again after change.

Think of StatefulSet as numbered dorm rooms

A Deployment is like a team of event workers. Everyone wears the same uniform and performs the same job. If worker A leaves and worker B replaces them, clients normally do not care.

A StatefulSet is closer to a row of numbered dorm rooms:

• web-0 is room 0
• web-1 is room 1
• web-2 is room 2
• every room has its own mailbox
• every room has its own storage locker

The resident may change, but the room number cannot be shuffled.

The next resident in room 0 must recover room 0’s mailbox and locker, not take the belongings from room 2.

Dorm	StatefulSet
room number	Pod ordinal such as web-0
mailbox address	stable Pod DNS
private locker	one PVC per Pod
move in by room order	ordered creation and scaling
move out in reverse	reverse ordered scale-down

The real design question is:

Are the replicas interchangeable, or must each member be addressed by identity?

The three identities StatefulSet stabilizes

1. Stable ordinals

With N replicas, Pods receive ordinals from 0 through N-1 by default:

web-0
web-1
web-2

The names do not use random suffixes. That lets applications, monitoring, and operations refer to a specific member.

Do not over-interpret the number: web-0 does not automatically become the leader. The ordinal provides identity; leader election, replication, and quorum still belong to the application protocol.

2. Stable network identity

A StatefulSet uses a governing Headless Service:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - name: http
      port: 80

clusterIP: None means the Service does not provide the normal single virtual-IP load-balancing path.

It acts more like a discoverable member directory. Clients can resolve individual Pods such as:

web-0.web.default.svc.cluster.local
web-1.web.default.svc.cluster.local

A Headless Service is not a public traffic entry point, and StatefulSet does not create it for you. Its selector, name, and StatefulSet serviceName must agree.

Pod-specific DNS can also be affected by readiness and DNS negative caching. If peers must discover members before they become Ready, review publishNotReadyAddresses deliberately rather than assuming immediate discovery.

3. Stable dedicated storage

volumeClaimTemplates creates a PVC for each Pod:

volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteOncePod
      storageClassName: fast
      resources:
        requests:
          storage: 10Gi

With three replicas, the conceptual result is:

data-web-0 -> PVC/PV for web-0
data-web-1 -> PVC/PV for web-1
data-web-2 -> PVC/PV for web-2

When web-0 is recreated, it references data-web-0 again. That is how identity and data line up after replacement.

By default, deleting a Pod, scaling down, or deleting the StatefulSet does not automatically purge the associated PVCs.

Recent Kubernetes versions support .spec.persistentVolumeClaimRetentionPolicy for choosing whether PVCs are retained or deleted when a StatefulSet is deleted or scaled down. Review the cluster version, data-retention requirement, and PV reclaim policy before enabling deletion behavior.

A useful StatefulSet example

This is not a production database manifest. It only connects the pieces:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  clusterIP: None
  selector:
    app: web
  ports:
    - name: http
      port: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: web
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - name: http
              containerPort: 80
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOncePod
        storageClassName: fast
        resources:
          requests:
            storage: 10Gi

The responsibility split is:

• the Headless Service controls the member network domain
• serviceName connects the StatefulSet to that domain
• the ordinal gives each Pod a stable name
• volumeClaimTemplates gives each Pod its own claim
• the StorageClass and PV provisioner supply real storage
• the application handles replication, leadership, quorum, and data consistency

StatefulSet manages identity and orchestration. It does not invent a distributed-systems protocol for the application.

Ordering is not decoration

The default podManagementPolicy is OrderedReady.

Creation normally progresses in order:

web-0 Ready
  -> web-1 Ready
    -> web-2 Ready

Normal scale-down proceeds in reverse:

web-2
  -> web-1
    -> web-0

RollingUpdate also works from the highest ordinal toward the lowest, waiting for the updated Pod to become Running and Ready before continuing.

That is useful for workloads with bootstrap dependencies, but it has a cost:

If an earlier member is not Ready, later operations can queue behind it.

When rollout stalls, find the ordinal blocking the line.

If strict scaling order is unnecessary, podManagementPolicy: Parallel can relax creation and scale waits. Parallel does not make identities interchangeable.

One important boundary: directly deleting a StatefulSet does not guarantee ordered Pod termination. If ordered graceful shutdown matters, scale to zero first and verify the application state before deleting the object.

Deployment or StatefulSet?

Start with one question:

If one Pod disappears, can any new replica replace it without knowing which member it used to be?

If yes, start with Deployment.

If no—because the replacement needs its identity, dedicated data, or stable peer address—consider StatefulSet.

Decision	Deployment	StatefulSet
replicas interchangeable	usually yes	usually no
Pod name	random suffix	stable ordinal
individual Pod DNS	usually irrelevant	often part of the design
dedicated PVC per replica	requires extra design	native through volumeClaimTemplates
creation and scale order	not emphasized	ordered by default
typical workload	web/API, stateless worker	database, broker, identity-aware cluster

Having a PVC is not enough reason to select StatefulSet. A single-replica application can mount a PVC from Deployment.

StatefulSet is also not a magic safety wrapper for databases. It does not automatically provide backups, replication, correct failover, data consistency, quorum, or disaster recovery.

Common mistakes

1. Assuming StatefulSet Pods are never recreated

They are. Identity is stable; Pod UID, IP, node, and process are not.

2. Assuming web-0 automatically becomes leader

It does not. Leadership still comes from configuration, an operator, or an election protocol.

3. Forgetting the Headless Service

Without the governing Headless Service, the stable Pod DNS design is incomplete. Check serviceName, Service name, namespace, selector, and Pod labels.

4. Assuming StatefulSet deletion cleans all storage

Associated PVCs are retained by default. Before cleanup, inspect PVC retention policy, PV reclaim policy, backups, and the actual storage backend.

5. Force-deleting a stuck Pod too early

A StatefulSet identity must remain unique. If the old node state is unknown, force deletion can let two instances with the same identity run and create split-brain risk.

Confirm the old instance cannot return before force deletion.

6. Treating OrderedReady as application readiness

Kubernetes sees probes and Pod state. If readiness does not represent replication recovery or quorum safety, a Ready Pod may still be unsafe at the data layer.

How I inspect it

kubectl get statefulset -n <ns>
kubectl describe statefulset <name> -n <ns>
kubectl rollout status statefulset/<name> -n <ns>
kubectl get pods -n <ns> -l app=<label> -o wide
kubectl describe pod <name>-<ordinal> -n <ns>
kubectl get service <headless-service> -n <ns> -o yaml
kubectl get endpointslice -n <ns> -l kubernetes.io/service-name=<headless-service>
kubectl get pvc -n <ns>
kubectl describe pvc <claim-name> -n <ns>
kubectl get events -n <ns> --sort-by=.lastTimestamp

The signals I care about are:

• whether READY, CURRENT, and UPDATED replicas agree
• which ordinal first fails to become Ready
• whether Pod names, owner references, and labels match expectations
• whether the Headless Service has clusterIP: None
• whether serviceName, Service selectors, and Pod labels agree
• whether every ordinal received its own PVC and whether it is Bound
• whether Events show scheduling, attach, mount, probe, or termination failures
• whether update strategy, partition, or pod management policy is blocking progress

Short version:

Find the member that fell behind, then check whether its name, address, and data were matched again.

How I remember StatefulSet

I do not remember StatefulSet as “the Kubernetes object for running databases.” That makes the object sound safer than it is.

I remember it like this:

A StatefulSet is a fixed seating chart: people can change, but seat numbers, contact addresses, and storage lockers cannot be shuffled.

Deployment aims for: one replica is missing, so create any replica that can do the job.

StatefulSet aims for: member 1 is missing, so recreate member 1 with its identity and data.

That difference is the boundary between a stateless replica and a stateful member.

Keep these three things

1. StatefulSet preserves sticky identity; it does not make Pods immortal
2. Ordinals, a Headless Service, and volumeClaimTemplates connect names, network identities, and dedicated storage
3. StatefulSet owns Kubernetes-level identity and ordering; replication, leadership, quorum, and backups remain application responsibilities

The next chapter asks a different lifecycle question: What are Jobs and CronJobs? Not every workload should stay alive forever; some work should finish and formally stop.

Technical references:

• https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
• https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
• https://kubernetes.io/docs/concepts/storage/persistent-volumes/