Fix: Container OOMKilled (Exit Code 137) After Setting K8s Limits

The Issue: The “It Works on My Machine” Limit Paradox

A common frustration for Cloud Architects is a container that performs perfectly in a local Docker Compose environment but enters a CrashLoopBackOff with an OOMKilled status the moment it is deployed to a managed Kubernetes cluster (EKS, GKE, or AKS) with defined memory limits.

In Docker Compose, if no mem_limit is specified, the process often consumes as much host memory as needed. Kubernetes, however, enforces strict isolation via Linux Cgroups. If the application runtime (like the JVM, Node.js, or Python) is not “container-aware,” it may attempt to allocate memory based on the total Node capacity rather than the limit defined in your Deployment YAML, leading to an immediate kernel kill.

The Exact Error Log

When inspecting the pod or describing the deployment, you will see the following status:

# Command: kubectl describe pod <pod-name>
State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 01 Jan 2024 10:00:00 +0000
  Finished:     Mon, 01 Jan 2024 10:00:05 +0000

And in the Node’s dmesg or system logs:

[ 1234.567890] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=docker-xyz.scope,mems_allowed=0,oom_memcg=/kubepods/burstable/pod-uid,task_memcg=/kubepods/burstable/pod-uid/container-id,task=java,pid=1234,uid=0
[ 1234.567900] Memory cgroup out of memory: Killed process 1234 (java) total-vm:4234567kB, anon-rss:1048576kB, file-rss:1234kB, shmem-rss:0kB

Diagnostic Checklist

1. Runtime Container Awareness: Is your application runtime (e.g., Java 8u131 or earlier) looking at /proc/meminfo (Host memory) instead of /sys/fs/cgroup/memory/memory.limit_in_bytes?
2. Cgroup Versioning: Is your Cloud Provider using Cgroup v2? (Run stat -fc %T /sys/fs/cgroup). Cgroup v2 handles memory pressure differently than v1, often killing processes faster when they hit the “High” threshold.
3. Transparency of Sidecars: Are you running Istio or a Cloud SQL Proxy? These sidecars share the Pod’s overhead but often aren’t accounted for in the primary container’s limit.
4. Buffer/Cache Pressure: Is the application performing heavy I/O? Linux treats cached files as part of the Cgroup memory usage. If the limit is too tight, the kernel cannot reclaim enough page cache, triggering the OOM killer.
5. Swap Configuration: Most managed Kubernetes nodes (EKS/GKE) disable swap by default. Local Docker environments often have swap enabled, masking memory leaks that become fatal in the cloud.

The Fix

1. Align Application Runtime with Cgroup Limits

If running Java, do not use fixed -Xmx values if you plan to scale. Use percentage-based RAM settings so the JVM respects the Kubernetes limit.

# Dockerfile adjustment for Java 11+
ENV JAVA_OPTS="-XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0"

# Execute using the environment variable
ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]

2. Configure Kubernetes Resource Alignment

Ensure your requests and limits are logically gapped. Setting them to the exact same value (Guaranteed QoS) is safer for production but requires the application to be strictly tuned.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloud-app
spec:
  template:
    spec:
      containers:
      - name: app-container
        image: my-registry/app:v1
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            # Ensure limit accounts for: Heap + Non-Heap + Stack + Sidecar
            memory: "1Gi"
            cpu: "1000m"

3. Adjust for Cloud Provider OS Specifics (Cgroup v2)

On modern OS images like Amazon Linux 2023 or Ubuntu 22.04, Cgroup v2 is the default. If your older application fails to read limits, you can force the node back to Cgroup v1 via UserData (not recommended long-term) or update your runtime base image to a version that supports Cgroup v2 (e.g., Node.js 16+, Python 3.11+, Java 17+).

To check Cgroup version on the node:

# SSH into the Node
mount | grep cgroup
# If it says 'cgroup2', your application MUST be on a modern runtime version.

4. Handling Memory Spikes with Transparent Huge Pages (THP)

In cloud environments, THP can sometimes double memory usage for certain databases or caches. If OOMKilled persists, disable THP in your Node Template or via a DaemonSet:

echo never > /sys/kernel/mm/transparent_hugepage/enabled