Kubernetes DiskPressure: How to Find Which Pod Is Eating Your Node's Disk (Bottlerocket Included)

Your pager goes off. A pod got evicted. You run kubectl describe node and there it is: DiskPressure=True.

So you do what everyone does. You open a shell on the node and run df -h. It says 90% full. You run du -sh /* and add it up. It says 40%.

Where did the other half of the disk go?

If your nodes run Bottlerocket, it gets worse. There is no SSH, no package manager, and no crictl on the host. The usual playbook does not even start.

This post walks through a method that works on any Kubernetes node, and a read-only script that does the whole investigation for you. It tells you which namespace, pod and container is using the disk, what is growing right now, and what already got evicted in the past. It works even when the node is healthy today.

In this post

• What DiskPressure actually measures
• Why df and du disagree
• How to get onto a Bottlerocket node
• How the script attributes disk usage to pods
• How to read the results
• A real case: a database that never cleaned up after itself
• Fixes for each root cause
• FAQ

What does DiskPressure mean in Kubernetes?

DiskPressure is a node condition set by the kubelet. It turns True when free space or free inodes on the node drops below an eviction threshold. Once it does, the kubelet starts evicting pods to get space back.

The default hard thresholds are:

On a typical cloud node, the kubelet root and the container runtime share one disk. That means both signals read the same filesystem, and the stricter 15% rule is the one that fires first. People often assume they have 10% of headroom when they really have 15%.

Before evicting anything, the kubelet tries image garbage collection. It starts deleting unused images once disk usage crosses imageGCHighThresholdPercent (85% by default) and stops at imageGCLowThresholdPercent (80%). If those numbers sit too close to your eviction threshold, garbage collection starts too late to help.

Why do df and du show different numbers?

This is the question behind most DiskPressure investigations. df asks the filesystem how many blocks are in use. du walks directories and adds up file sizes. They only agree when every used block belongs to a file you can see.

The usual reasons for a gap:

1. Deleted files that a process still holds open. A log gets rotated or removed, but the application keeps writing to the old file handle. The space stays allocated until the process closes it or exits. du cannot see it because the file has no name anymore.
2. Directories du could not read. Permission and SELinux denials are silent if you hide stderr. Everything under that path quietly drops out of the total.
3. Data hidden under a mount point. Files written before something was mounted on top of a directory still use space, but no path leads to them.
4. Filesystem overhead and reserved blocks. Usually a percent or two, not half the disk.

I hit number 2 on a real node. The report said 75% of used space was "not visible to du". The cause had nothing to do with the workload. I had started my debug pod without enough privileges, so du was skipping most of the container runtime directory. The fix was one flag, covered in the next section.

How do you get onto a Bottlerocket node?

Bottlerocket is built to be locked down. The root filesystem is read-only, there is no shell login by default, and SELinux limits what unprivileged containers can touch. The container data you need lives on a separate data partition mounted at /local, and /var is bind-mounted from it.

The supported way in is a node debug pod:

kubectl debug node/<node-name> -it \
  --profile=sysadmin \
  --image=docker.io/dheeth/ubuntu-k8s-utils:1.37

Two details matter.

Use --profile=sysadmin. The default profile runs without the privileges needed to read containerd and kubelet directories on Bottlerocket. You will not get an error. You will get quietly incomplete numbers, which is worse.

Use a full image. Busybox ships a stripped-down find and du that lack the options the investigation needs. Ubuntu or Debian works. The node's root filesystem appears inside the pod at /host.

The script: what it checks

I wrote a single bash script that you copy into the debug pod and run. It reads from the node and never deletes or changes anything.

# from your workstation
kubectl cp k8s-diskpressure-debug.sh <debug-pod>:/tmp/dp.sh

# inside the debug pod
bash /tmp/dp.sh              # full report plus a 60 second growth sample
bash /tmp/dp.sh -i 600       # watch growth for 10 minutes
bash /tmp/dp.sh -i 0         # skip the growth sample

The script is at dheeth/k8s-diskpressure-debug.sh.

It produces a report with these parts:

• Filesystem state against the real thresholds. It reads the kubelet config on the node, so you see your actual eviction settings and how close you are to them, not the defaults.
• A breakdown by category. Writable layers of running containers, image layers, unused snapshots, the image content store, pod volumes, container logs and the systemd journal.
• Top pods by ephemeral storage. Writable layer plus emptyDir plus logs, per pod, with the top offenders listed.
• Largest files inside the heaviest containers. The path is shown as it appears inside the container, which tells you what the application is writing.
• Orphaned data. Pod directories with no running container, such as leftovers from finished Jobs.
• Deleted-but-open files. Grouped by owning container, so you can see which process to restart.
• Growth over time. The script measures twice and shows what grew, how fast, and how many hours until the node hits its threshold at that pace.
• History. It searches the kubelet and containerd journal for past eviction messages and image GC failures, and counts which pods were evicted most often.
• A ranked summary. The likely causes in order, each with a suggested fix.

The growth sample and the history search are what make it useful on a healthy node. If a node is at 70% today but climbing 3 GB an hour, you want to know before it pages you.

How does the script map disk usage back to a pod?

This is the part that is missing from most guides. On a node, disk paths are IDs. A directory named snapshots/412/fs tells you nothing about which application owns it. To turn IDs into names, the script uses three sources that exist on every containerd node.

1. The host's mount table. Every running container has an overlay mount, and the options include the path of its writable layer. Reading /proc/1/mountinfo from the debug pod (it shares the host PID namespace) gives you the container ID and its upperdir.

# find each running container's writable layer
awk '{ split($0,h," - "); split(h[1],a," "); mp=a[5]
       if (mp ~ /io\.containerd\.runtime\.v2\.task\/k8s\.io\/[0-9a-f]+\/rootfs$/) print mp, h[2] }' \
  /proc/1/mountinfo

2. Container metadata. containerd keeps a config.json for each task. The CRI annotations in it carry the Kubernetes names.

grep -oE 'io.kubernetes.cri.(sandbox-namespace|sandbox-name|container-name)":"[^"]*' \
  /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/<container-id>/config.json

3. Pod log folders. The directory names under /var/log/pods follow the pattern <namespace>_<pod>_<uid>. That links each pod UID in /var/lib/kubelet/pods to a human readable name.

Join those three and every directory on the data disk gets an owner. Anything left over is an image layer nobody is using, or data from a pod that no longer exists. Both are useful findings.

Reading the output

Here is the kind of output you get. This example is illustrative, with names changed.

 6. TOP PODS BY EPHEMERAL STORAGE
       TOTAL   RW-LAYERS     VOLUMES        LOGS  NAMESPACE/POD
    41.8 GiB     0.4 GiB    41.2 GiB      0.2 GiB  data/zookeeper-1
     6.3 GiB     5.9 GiB     0.1 GiB      0.3 GiB  apps/report-worker-7d9f
     2.1 GiB     0.2 GiB     0.0 GiB      1.9 GiB  apps/api-gateway-5c4b

15. GROWTH OVER TIME
  Window: 300s. Filesystem used changed by 38.4 MiB (about 460 MiB/hour)
  [WARN] At this rate the node reaches DiskPressure in about 31.0 hours.
  Fastest-growing owners
  +  31.0 MiB     372 MiB/h   pod dir     data/zookeeper-1

17. SUMMARY
  1. Pod data/zookeeper-1 uses 41.8 GiB (31.4% of disk): mostly emptyDir/pod
     volume data. Set emptyDir.sizeLimit and limits.ephemeral-storage.
  2. Live growth: 38.4 MiB in 300s (~460 MiB/h), DiskPressure in about 31 h.

A quick guide to the three columns in the pod table:

• RW-LAYERS is data written inside the container filesystem. A large number here means the app writes somewhere that is not a volume, such as /tmp, a cache folder or a local database file.
• VOLUMES is emptyDir and other pod-local volume data stored on the node disk. This is the most common source of surprise growth.
• LOGS is stdout and stderr stored by the container runtime. If this is far above containerLogMaxSize times containerLogMaxFiles, rotation is not keeping up.

A real case: 42 GB in under four days

The node I debugged this on had been evicting the same pod again and again. The script pointed at one ZooKeeper pod, and the biggest directory inside it was the data directory, full of snapshot and transaction log files.

The cause was a default most people never look at. ZooKeeper does not clean up old snapshots and transaction logs unless you turn that on. The Bitnami image exposes this as ZOO_AUTOPURGE_INTERVAL, which defaults to 0, meaning disabled. Over three and a half days, the pod had written about 42 GB.

The fix was a config change, with no manual cleanup inside the pod:

- name: ZOO_AUTOPURGE_INTERVAL
  value: "1"          # purge every hour
- name: ZOO_AUTOPURGE_RETAIN_COUNT
  value: "3"          # keep the 3 newest snapshots and the logs they need

Rolled out one pod at a time, with a readiness check that waits for the member to rejoin the quorum, it caused no downtime. The first purge runs at startup, and the old data on ephemeral storage disappears with the old pod.

The lesson applies to more than ZooKeeper. Databases, message brokers and search engines all write to local disk by design, and many ship with cleanup turned off or set very loose.

How to fix each root cause

Two settings are worth adding to any workload that stores data on the node:

resources:
  requests:
    ephemeral-storage: 2Gi
  limits:
    ephemeral-storage: 5Gi

The request tells the scheduler and the kubelet how much you expect to use. Under node pressure, pods that exceed their requests are evicted before pods that stay within them. The limit stops a single runaway pod from filling the node, though a pod that crosses its limit is itself evicted.

Limits of this approach

A few honest caveats.

• The scan reads a lot of directories. It runs at low priority, but on a large node it can take a minute or two.
• Without crictl, the script cannot name the image behind an unused snapshot. You can pass --install-crictl if the pod has internet access, or check images from your workstation.
• Growth sampling catches steady writers. Bursty workloads like batch jobs may need a longer window, such as -i 900 during their busy hours.
• It sees the node, not the cluster. To compare with what the kubelet itself reports per pod, use kubectl get --raw "/api/v1/nodes/<node>/proxy/stats/summary".

Frequently asked questions

How do I check which pod is using the most disk space on a Kubernetes node?

The kubelet stats summary gives ephemeral storage per pod: kubectl get --raw "/api/v1/nodes/<node>/proxy/stats/summary". For file-level detail and attribution of image layers, logs and deleted files, you need to inspect the node itself, which is what the script automates.

What is the difference between nodefs and imagefs?

nodefs is the filesystem holding kubelet data, including emptyDir volumes and pod logs. imagefs holds container images and writable layers. On most cloud nodes they are the same disk, so both thresholds apply to the same free space.

Why was my pod evicted when it was under its ephemeral-storage limit?

Node-pressure eviction is separate from limits. When the node crosses a threshold, the kubelet ranks pods by whether they exceed their requests, then by priority, then by usage. A pod can be evicted for the node's sake even if it is within its own limit, especially if it has no request set.

Do PodDisruptionBudgets protect against DiskPressure eviction?

No. Node-pressure eviction does not respect PodDisruptionBudgets. Use priority classes, ephemeral-storage requests and moving data to persistent volumes instead.

How do I SSH into a Bottlerocket node?

You normally do not. Use kubectl debug node/<node> --profile=sysadmin for a privileged pod with the host filesystem at /host. Bottlerocket also has an admin container reachable through AWS Systems Manager if you have enabled it.

Why does du show less usage than df?

Usually because a process holds a deleted file open, because du could not read some directories, or because data sits under a mount point. Check for permission errors first, then list deleted-but-open files under /proc/*/fd.

Does this work on nodes that are not Bottlerocket?

Yes. It detects the OS and falls back to the standard paths used on Amazon Linux, Ubuntu and similar distributions. You still run it from a privileged debug pod with the host root mounted.

Wrapping up

DiskPressure is rarely a mystery once you can see which pod owns each gigabyte. The hard part is that nothing on the node tells you that directly. You have to join mount tables, container metadata and log folder names to get from a path to a workload.

Run the script once on a node you think is healthy. If it finds a pod that has been growing quietly for days, you just saved yourself a page.

The script is at dheeth/k8s-diskpressure-debug.sh. If you extend it or find a case it misses, send a pull request.