The uploads vanished when the pod restarted. Kubernetes networking and storage: what actually breaks

2026-11-06来源 dev.to DockerEnglish

Sources: CD's devops wiki 03-kubernetes/03-network-fundamentals.md, 04-storage-fundamentals.md and 07-lab-storage-nfs.md, cross-checked against kubernetes.io, Calico docs and cloud storage price lists on 2026-10-28. Lab names only; screens are drawings.

Two tickets, one week, same cluster.

Monday. Contoso's support portal lets customers attach files to tickets. A routine deploy rolls the pods. Every attachment uploaded in the previous 9 days is gone: 1,240 files. The app wrote them to a folder inside the container. The folder died with the pod, exactly as designed.

Thursday. The orders API stops answering. Nothing was deployed. A node was drained for patching, the API pod moved to another node and got a new IP. The web tier had that IP in a config file, copied from kubectl get pods -o wide three months earlier because "the Service thing was confusing".

Neither is a Kubernetes bug. Both are the two parts of Kubernetes that behave nothing like a VM: the network and the disk.

TL;DR

Problem: pods are disposable. Their IP and their local disk are disposable with them. Anything that remembers a pod IP or writes to a pod's filesystem will lose data or connectivity on the next restart, drain or deploy.
Who it hurts: customers (lost files, broken checkout), support and legal (data you promised to keep is gone), the on-call engineer (outages with no deploy and no error), security (a flat network where every pod can reach the database).
Fix: talk to Services by DNS name, never pod IPs; write data to a PersistentVolumeClaim, never the container; add a default-deny NetworkPolicy so "every pod can talk to every pod" stops being the default.
Cost: a 128 GiB persistent disk is about $10 to $20 a month in Azure or AWS; an NFS share on a VM you already have is free. The 1,240 lost files cost two days and a customer apology.
Prove it: delete the pod. Files still there, DNS still answers, the blocked connection still times out.

1. The problem
2. Who gets hurt
3. Cost and value
4. Networking: the three things to know
5. Storage: the three things to know
6. Prove it
Gotchas
Checklist
What changed recently
Sources

1. The problem

On a VM, the IP address and the disk are the two most permanent things about the machine. In Kubernetes they are the two least permanent. That inversion is where most first-year incidents come from.

On a VM	In Kubernetes	What breaks if you forget
The IP lasts for years	Every pod gets its own IP; it changes on every restart or move	Hard-coded IPs, allowlists of pod IPs, "ping the pod" runbooks
The disk is the machine	The container filesystem is thrown away with the pod	Uploads, SQLite files, logs, anything written "locally"
A firewall sits between servers	All pods can reach all pods, no NAT, by default	Any compromised pod can reach the database
Storage is attached by an admin	Storage is requested by a claim and bound by the cluster	Pods stuck in `Pending` because no volume matched

The real world has the same shape. AdGuard VPN's public post-mortem (incident 2025-12-02) describes a Redis StatefulSet recreated on Kubernetes to fix a disk-size drift; the controller restarted the whole cluster at once, there were no startup or readiness probes to slow it down, and the service was down for about 50 minutes. The disk resize was fine. The parts that behave differently from a VM were not understood.

2. Who gets hurt

Who	What they lose
Customers	Files they uploaded, carts they built, sessions that vanish on every deploy
Support / legal	"We keep your attachments" was a promise. 1,240 of them are gone, with no backup because "the disk is on the server"
On-call engineer	Outages with no deploy and no error log: a node drain, a pod reschedule, a new IP
Security	A flat network. The web pod can open a connection to the database pod. So can the compromised one
Finance	Premium disks attached to pods that write 2 MB a day, because nobody chose a StorageClass

3. Cost and value

What persistent storage costs (list prices, checked 2026-10-28):

Option	Price	Fits
Azure Standard HDD S10, 128 GiB	$5.89 a month	Backups, cold files
Azure Standard SSD E10, 128 GiB	$9.60 a month	Uploads, small databases (this post)
Azure Premium SSD P10, 128 GiB	$19.71 a month (East US)	Databases that need IOPS
AWS EBS gp3	$0.08 per GB-month, so 128 GB is about $10.24; 3,000 IOPS included	Same as E10 on AWS
NFS share on a VM you already run (my lab)	$0 extra	Shared RWX storage for a few pods, labs, small teams
minikube default StorageClass `standard`	$0	Learning only: hostPath on the minikube node

What the incidents cost: the Monday ticket was 1,240 files, two days of engineering to confirm they were unrecoverable, and an apology email. The Thursday ticket was 35 minutes of failed orders. A $9.60 disk and a DNS name would have prevented both.

What networking costs: nothing. Services, CoreDNS and NetworkPolicy are in the cluster already. The only paid item is a cloud load balancer when you use type: LoadBalancer (roughly $18 to $25 a month per public IP on the big clouds), which is why the Gateway API with one shared load balancer is the cheaper way to expose many apps.

4. Networking: the three things to know

From my networking notes, in the order they bite:

1. Every pod has an IP, and all pods can reach each other without NAT. The CNI plugin (Calico, Flannel, Cilium) hands out the IPs and routes between nodes. That IP is not yours to keep. Never write it down.

2. A Service is the name you keep. It gets a stable ClusterIP and, through CoreDNS, a stable name: orders-api.contoso-shop.svc.cluster.local (or just orders-api from the same namespace). kube-proxy rewrites traffic to whichever pods match the selector right now.

Service type	Reachable from	Use
`ClusterIP` (default)	Inside the cluster	Everything internal: web → api → db
`NodePort`	`<node IP>:30000-32767`	Labs, or behind your own load balancer (my kubeadm lab used an external NGINX in front of three NodePorts)
`LoadBalancer`	A cloud load balancer with a public IP	One app, one public entry point; costs money per IP
Ingress / Gateway	HTTP by host name and path	Many apps behind one load balancer. Ingress NGINX is retired; use the Gateway API or a maintained controller

3. Flat is the default; you must say no. Kubernetes ships with no network restrictions. A NetworkPolicy with an empty podSelector and no rules is the "deny everything in this namespace" switch. It only works if the CNI enforces policies: Calico and Cilium do, Flannel and minikube's default bridge do not (start minikube with --cni=calico).

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: contoso-shop spec: podSelector: {} policyTypes: [Ingress, Egress] --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-api-to-db namespace: contoso-shop spec: podSelector: matchLabels: { app: db } ingress: - from: - podSelector: matchLabels: { app: orders-api } ports: - port: 27017

(You also need an egress rule for DNS to kube-system, or nothing resolves. See Gotchas.)

5. Storage: the three things to know

1. emptyDir and the container filesystem die with the pod. They are for cache and scratch. Monday's uploads were in the container filesystem.

2. A PersistentVolumeClaim is how a pod asks for a disk that outlives it. The claim names a size, an access mode and a StorageClass; the cluster binds it to a PersistentVolume, either one an admin created or one the StorageClass provisions on demand.

Piece	Who creates it	Analogy
StorageClass	Admin, once	The catalogue: "standard SSD", "premium", "nfs"
PersistentVolume (PV)	Admin by hand, or the StorageClass automatically	The actual disk
PersistentVolumeClaim (PVC)	The app team, in the namespace	The order form: "10 GiB, ReadWriteMany, class nfs-storage"
Pod `volumes` + `volumeMounts`	The app team	Plugging the disk into the pod at a path

3. Access modes decide how many pods can share it. ReadWriteOnce (one node, the normal cloud disk), ReadOnlyMany, ReadWriteMany (many nodes at once; NFS, Azure Files, EFS). Uploads served by three web pods on three nodes need RWX.

My lab does RWX with an NFS server on the database VM (nfs-lab-01), exported as /data, with nfs-common on every node:

apiVersion: v1 kind: PersistentVolume metadata: name: pv-uploads-nfs spec: capacity: storage: 10Gi accessModes: [ReadWriteMany] persistentVolumeReclaimPolicy: Retain # PVC deleted -> data kept storageClassName: nfs-storage nfs: server: nfs-lab-01.contoso.com path: /data/uploads --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: uploads namespace: contoso-shop spec: accessModes: [ReadWriteMany] storageClassName: nfs-storage resources: requests: storage: 5Gi

And in the Deployment from last week's post, two additions:

 containers: - name: web volumeMounts: - name: uploads mountPath: /app/uploads volumes: - name: uploads persistentVolumeClaim: claimName: uploads

On minikube without NFS, delete storageClassName and the PV; the built-in standard class provisions a hostPath volume for the claim (RWO only, fine for the test).

6. Prove it

kubectl get pv,pvc -n contoso-shop # STATUS Bound on both kubectl exec -n contoso-shop deploy/web -- sh -c 'echo hello > /app/uploads/test.txt; ls /app/uploads | wc -l' kubectl delete pod -n contoso-shop -l app=web # all three kubectl exec -n contoso-shop deploy/web -- ls /app/uploads | wc -l # same number

Networking, from a throwaway pod:

kubectl run probe -n contoso-shop --rm -it --image=busybox:1.36 -- sh # inside: wget -qO- http://orders-api:8080/health # by name: ok wget -qO- http://orders-api.contoso-shop.svc.cluster.local:8080/health nc -zv db 27017 # with default-deny + allow-api-to-db: times out from "probe"

Then label the probe pod app=orders-api and the same nc succeeds. That is the whole proof that the policy is enforced and not silently ignored.

Gotchas

Default deny also blocks DNS. Add an egress rule to kube-system on UDP/TCP 53, or every name lookup fails and it looks like the network is down.
NetworkPolicy with no enforcing CNI is a comment. kubectl apply succeeds, nothing happens. Test it (section 6) before trusting it.
RWO means one node, not one pod. Three replicas on three nodes with an RWO claim: two of them stay Pending forever. Use RWX (NFS, Azure Files, EFS) or a StatefulSet with one claim per pod.
Reclaim policy. Dynamically provisioned volumes default to Delete: delete the PVC and the disk is gone. Set Retain on anything you care about.
A PVC can only grow. And only if the StorageClass has allowVolumeExpansion: true. The AdGuard incident started here: the StatefulSet template could not be edited in place.
hostPath is for one-node labs. It pins the pod to one node and gives the container the node's disk. Never in production.
NodePort is not a public endpoint. It is a port on every node. Put a load balancer or a Gateway in front.
service/web has a name per namespace. web from contoso-shop is web.contoso-shop.svc.cluster.local from anywhere else.

Checklist

[ ] No pod IP in any config file, allowlist or runbook; Services by DNS name only
[ ] default-deny NetworkPolicy per namespace + explicit allow rules + DNS egress; CNI confirmed to enforce
[ ] Every path the app writes to is a PVC mount (or emptyDir on purpose, for cache only)
[ ] StorageClass chosen on purpose: standard SSD for uploads, premium for databases, RWX where many pods write
[ ] persistentVolumeReclaimPolicy: Retain and allowVolumeExpansion: true where it matters
[ ] Backup of the PV, because a volume is not a backup
[ ] Section 6 run after every cluster upgrade: files survive, DNS answers, the blocked port times out

What changed recently

Checked on 2026-10-28:

My notes list Calico, Flannel and Cilium as CNIs; that is still the list, and still only Calico and Cilium enforce NetworkPolicy by default. The kubeadm lab in my notes used Calico 3.27; current releases are 3.30+ and the manifest URL pattern is unchanged.
Ingress NGINX retired in March 2026 (Kubernetes blog statement 2026-01-29; no further security fixes). The "External to cluster" row in my notes now reads: NodePort or LoadBalancer, fronted by the Gateway API (v1.6, June 2026) or a maintained Ingress controller.
Storage prices (Azure Retail Prices API via a 2026-10 third-party guide, AWS EBS pricing page): E10 $9.60, P10 $19.71, gp3 $0.08/GB-month. Azure's own pricing page renders client-side; worth confirming in the pricing calculator.
AdGuard VPN post-mortem (published 2025-12-10) is the clearest recent public example of StatefulSet storage drift causing an outage; it is linked below.

Sources

Thanks for reading! I'm Cuong, and on CD Security Lab I rebuild real Microsoft security cases in a safe lab and explain them simply. Follow along:

▶️ YouTube (daily Shorts + explainers): CD Security Lab
💼 LinkedIn: Cuong Do
📘 Facebook: CD Security Lab
📸 Instagram: @cdsecuritylab
📝 More posts like this: Dev.to

Got a question about your own tenant? Drop it in the comments.

原文：https://www.a8q9.cn/en/article/the-uploads-vanished-when-the-pod-restarted-kubernetes-networking-and-storage-wh-57e380.html