OOM Killer in Containers: Correctly Reading and Preventing Exit Code 137

1. How the OOM killer triggers inside containers

The out-of-memory killer is a kernel mechanism that steps in when memory is requested that is no longer available, and no process is willing to voluntarily free memory. Outside of a container context, the OOM killer only activates once the entire system runs out of memory, a rare and usually dramatic event. Inside a container with a memory limit set, that changes fundamentally: the OOM killer intervenes as soon as just that one container's cgroup hits its limit, regardless of how much free memory remains on the rest of the host.

Concretely, that means a host with many gigabytes of free RAM can still kill a single container the moment its individual memory.max is reached. The kernel selects a process within the affected cgroup based on a score, the so-called oom_score, and terminates exactly that process with a SIGKILL signal. Most Docker containers run only one main process, which means in practice the entire application inside the container gets terminated, not just a single thread or worker.

2. Correctly decoding exit code 137

At first glance exit code 137 looks like an arbitrary number, but it follows a fixed Unix convention: exit codes above 128 signal that a process was terminated by a signal, calculated as 128 plus the signal number. SIGKILL carries signal number 9, and 128 plus 9 equals exactly 137. A process that ends with exit code 137 was therefore not terminated normally and had no chance to shut down cleanly, because unlike SIGTERM, SIGKILL cannot be caught or delayed.

It is important to distinguish, though: not every exit code 137 necessarily comes from the OOM killer, theoretically a manual docker kill or an orchestrator can also produce this code. Docker itself provides a reliable way to tell them apart: docker inspect explicitly shows true in the State.OOMKilled field when the container was actually terminated by the OOM killer. That field should be the first thing checked for any unexpected container crash with code 137, before diving into application log analysis.

# Check exit code and OOM status of a terminated container
docker inspect --format \
  'ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}' \
  my-container

# Example output:
# ExitCode=137 OOMKilled=true

3. Reading kernel logs: dmesg and journalctl

Even after docker inspect confirms the OOM killer was active, the kernel log itself provides considerably more context: exactly which process was killed, how much memory was allocated at the moment of the kill, and which cgroup was affected. On the host this is accessible through dmesg or, on systemd systems, through journalctl -k. The relevant line typically contains the phrase Killed process, followed by the process ID, process name, and the amount of memory allocated in kilobytes.

A look at this log line often reveals more than expected: sometimes it is not the application's main process that gets killed but a child process or a temporarily spawned helper process, which can point to a different underlying issue, for example a memory leak in a subprocess rather than the main application. The exact timestamp of the kill, cross-checked against application metrics or load spikes, often helps narrow down the trigger too, for example a batch job that briefly consumed far more memory than usual.

# Search the kernel log for OOM events
dmesg -T | grep -i "killed process"

# On systemd systems, alternatively via journalctl
journalctl -k --since "1 hour ago" | grep -i oom

# Example output:
# Out of memory: Killed process 48213 (node) total-vm:1235412kB,
# anon-rss:524288kB, file-rss:0kB, shmem-rss:0kB

4. Measuring real memory usage instead of guessing

The most common reason for OOM kills is not a memory leak, it is a limit that was set too tight and was never validated against the application's actual usage. Instead of guessing a value like --memory=256m because it appeared in some tutorial, the real memory requirement should first be measured without a limit, or with a generous one, over a representative time period, ideally under realistic load rather than idling right after startup.

The command docker stats shows current usage live but is a poor fit for capturing short-lived load spikes, because it only delivers snapshots. More reliable is checking the cgroup file memory.peak, which records the highest memory value ever reached since the cgroup was created, regardless of exactly when that peak occurred. Only once that peak value is known across several days or several typical load cycles does it make sense to set a limit with a reasonable safety margin, typically 20 to 30 percent above it.

# Observe live usage (snapshot, not a peak value)
docker stats my-container --no-stream

# More reliable: read the actual peak since container start
CID=$(docker inspect --format '{{.Id}}' my-container)
cat /sys/fs/cgroup/system.slice/docker-${CID}.scope/memory.peak

# Log over a longer period (e.g. every minute)
while true; do
  echo "$(date -Iseconds) $(cat /sys/fs/cgroup/system.slice/docker-${CID}.scope/memory.current)"
  sleep 60
done >> memory-log.txt

5. Behavior with and without swap

Whether swap is enabled noticeably changes OOM behavior. Without any additional swap budget, meaning identical values for --memory and --memory-swap, the OOM killer fires very quickly and quite hard the moment the physical limit is reached. With extra swap headroom, the kernel can first page out memory to the swap partition before triggering the OOM killer, which delays the hard crash but simultaneously leads to drastically worse performance, because swap I/O is orders of magnitude slower than RAM access.

In most production container environments, swap for containers is deliberately disabled or heavily restricted, precisely because a container that starts swapping is usually already in a broken state, and a fast, clean restart through a restart policy is often the better solution than a slowly dying, swapping container. For latency-critical applications such as databases or caches this recommendation applies especially strongly, since unpredictable swap latencies are far more damaging there than a clean, immediate restart.

6. Language-specific traps: the JVM and Node.js

Certain runtimes do not automatically detect container memory limits, or only started doing so in relatively recent versions, leading to especially tricky OOM kills. JVM versions before Java 10 calculated the default heap size based on the entire host's memory, not the cgroup limit, so a JVM inside a container with a 512-megabyte limit could try to allocate a heap of several gigabytes. Modern JVMs from Java 10 onward are cgroup-aware, but should still be configured explicitly through flags such as -XX:MaxRAMPercentage to leave a safety margin for non-heap memory like thread stacks and metaspace.

Node.js has a similar but inverted problem: the V8 engine limits the heap by default independently of the container limit, often to a value too high for small containers, so the application only notices it is approaching a boundary very late, while the Node process, through additional memory outside the V8 heap such as buffer objects, has long since outgrown the cgroup limit. Here it helps to set --max-old-space-size explicitly, well below the container limit, so the application itself gets a chance at controlled error handling instead of a hard OOM kill.

# JVM: configure heap explicitly as a percentage of the container limit
docker run -d --memory=1g myapp:latest \
  java -XX:MaxRAMPercentage=75.0 -XX:+UseContainerSupport -jar app.jar

# Node.js: set the heap limit clearly below the container limit
docker run -d --memory=512m mynodeapp:latest \
  node --max-old-space-size=384 server.js

7. Continuous monitoring instead of a one-off measurement

A single measurement of memory usage is rarely enough, because an application's usage patterns change over weeks and months, for example through growing data volumes, new features, or changing user numbers. That is why ongoing monitoring through tools such as cAdvisor or the Node Exporter, combined with Prometheus, pays off, recording memory usage, OOM events, and restart counters over time and making them visualizable in Grafana.

Particularly valuable here is the metric container_oom_events_total, which cAdvisor derives directly from kernel cgroup statistics, along with alerts that already trigger at 80 or 90 percent of a limit's memory usage, long before the actual OOM kill occurs. That shifts the workflow from reactive debugging after a crash to proactively adjusting limits before users even notice an outage.

8. Preventive measures working together

The most effective prevention combines several of the measures discussed: realistic limits based on measured peak values rather than guessed ones, language-specific heap configuration that stays clearly below the container limit, continuous monitoring with early alerts, and a sensible restart policy such as on-failure or unless-stopped, so a single OOM kill does not turn into a permanent outage but instead the container restarts automatically while the actual cause is investigated in parallel.

It also helps to distinguish memory reservations from memory limits, provided the orchestration in use, such as Docker Swarm or Kubernetes, supports it: a reservation ensures a container always gets at least as much memory as needed for normal operation, while the limit defines the absolute ceiling for outliers. This combination prevents both resource starvation during normal operation and uncontrolled growth during malfunctions.

9. Troubleshooting checklist for suspected OOM

When a container crashes with exit code 137, a systematic approach beats random trial and error: first check the OOMKilled field via docker inspect, then search dmesg for the exact kill message, next compare the cgroup's memory.peak against the configured memory.max, and finally check the application logs for unusual activity shortly before the crash, for example a batch import or an unusually large request.

The table below summarizes the key diagnostic tools along with their respective purpose, as a quick reference for the next container that dies with exit code 137 for no obvious reason. Following this order usually finds the cause within a few minutes, instead of spending days searching the application code for a bug that does not exist.

10. Summary

11. FAQ: OOM Killer in Containers: The Essentials at a Glance