Docker Debugging for Running Containers: exec, inspect, stats, top, events

1. Systematic Docker debugging: the right approach

Docker debugging does not start by opening a shell inside the container, it starts by understanding the problem from a bird's eye view. Before you jump into a container interactively with docker exec, it pays to clarify the question first: is this a process failure, a resource bottleneck, a configuration problem or a network problem? Depending on the answer, a different tool is the right first choice. A systematic approach saves time and avoids the trap of poking around in an open shell for ages when docker inspect would have delivered the answer in seconds.

The typical workflow for Docker debugging in production: first docker ps -a to see the current status of all containers. Then docker stats --no-stream for a resource snapshot. Then docker logs --tail 100 for the most recent output. Only once these three steps fail to give a clear picture do you reach for docker inspect for a full configuration review, or docker exec for direct investigation inside the running container. This order avoids blind action and significantly reduces the mean time to resolution.

2. docker exec: running commands inside a live container

docker exec is the most important Docker debugging tool for direct investigation of a running container. It starts a new process inside the container's namespace without interrupting the existing main process. With -it you open an interactive shell, with -u root you can work with elevated privileges even if the container process normally runs as a regular user. The difference from docker run is fundamental: exec enters a running container, run starts a new one. For Docker debugging, exec is almost always the right choice.

A common pitfall: the container is based on a distroless image or on Alpine without a shell. In that case docker exec -it container sh simply fails. The fix for Docker debugging in such containers is nsenter at the host level (covered in section 8), or a temporary debug image started with docker run --pid=container:target that shares the same process namespace. Since Docker 24 there is also docker debug, which automatically attaches a debug toolbox image to the container's namespace.

# Open an interactive shell in a running container (bash or sh fallback)
docker exec -it my-app bash 2>/dev/null || docker exec -it my-app sh

# Run a single diagnostic command non-interactively
docker exec my-app cat /etc/resolv.conf
docker exec my-app env | sort
docker exec my-app ps aux

# Execute as root even if container runs as non-root user
docker exec -u root -it my-app bash

# Check open network connections inside the container
docker exec my-app ss -tlnp
docker exec my-app cat /proc/net/tcp

# Inspect the filesystem, find recently modified files
docker exec my-app find /app -newer /tmp -type f -ls 2>/dev/null | head -20

# Docker 24+: debug distroless containers with built-in toolbox
docker debug my-app

3. docker inspect: reading the full container configuration

docker inspect returns the complete JSON representation of a running or stopped container: network configuration, volume mounts, environment variables, resource limits, health check status, labels, the restart counter and the exact exit reason. For Docker debugging, the State block is especially relevant: it contains ExitCode, Error, OOMKilled (a boolean indicating whether the container was terminated by the OOM killer), StartedAt and FinishedAt. With --format you can extract exactly the piece you need.

The Go template syntax of --format allows precise queries without jq. Particularly useful for Docker debugging: docker inspect --format '{{json .NetworkSettings.Networks}}' shows all IP addresses and networks, {{range .Mounts}}{{.Source}} to {{.Destination}}{{println}}{{end}} lists all volume mounts, and {{.State.OOMKilled}} checks directly whether the last crash was an OOM kill. Combined with python3 -m json.tool, the full JSON can also be pretty printed for readability.

4. docker stats: resource usage in real time

docker stats shows continuously updated metrics for running containers: CPU percentage relative to all available cores, RAM usage against the configured limit, network I/O since the container started, and block I/O. For Docker debugging the --no-stream flag is important: it produces a one time snapshot, ideal for scripting and captures. With --format you can customize the columns. The CPU percentage is relative to all cores combined, so a value of 200% means the container is fully saturating two cores.

A common debugging scenario: a container consistently consumes 100% CPU with no obvious reason. docker stats confirms the CPU load, docker top shows which process is responsible, and docker exec container cat /proc/PID/cmdline returns the full command line. Combining these three steps usually leads straight to the cause in Docker debugging, whether that is an endless loop in application code, a stuck cron job, or a runaway import process.

# One-shot resource snapshot for all running containers
docker stats --no-stream --format \
  "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}\t{{.BlockIO}}"

# Inspect the full container state, check for OOM kill, exit code, restart count
docker inspect --format '
Name:         {{.Name}}
Exit Code:    {{.State.ExitCode}}
OOM Killed:   {{.State.OOMKilled}}
Restarts:     {{.RestartCount}}
Started:      {{.State.StartedAt}}
Finished:     {{.State.FinishedAt}}
Health:       {{.State.Health.Status}}
' my-app

# List all volume mounts
docker inspect --format \
  '{{range .Mounts}}{{.Type}} {{.Source}} → {{.Destination}} ({{.Mode}}){{println}}{{end}}' my-app

# Show all environment variables (useful for configuration debugging)
docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' my-app | sort

5. docker top: viewing processes inside the container

docker top shows all processes running inside a container as seen from the host, technically it reads the host's /proc and filters by the container's process group. That means it also works for containers with no ps installed at all. For Docker debugging this is essential when the container is based on a minimal image. By default docker top container shows the columns PID, UID, STATUS and CMD. With an optional ps format string (after the container name) you can request any columns you like.

docker top is especially valuable for Docker debugging of PHP-FPM containers: you can immediately see how many worker processes are active, whether they are in state S (sleeping, waiting for a request) or R (running, actively busy), and whether zombie processes (Z) have piled up. Zombie processes occur when the parent process fails to reap the exit codes of its child processes. In Docker this happens frequently when the container's process does not implement correct PID 1 behaviour.

6. docker events: the event stream of the Docker daemon

docker events shows a continuous stream of all events the Docker daemon generates for containers, images, networks and volumes. For Docker debugging in production this is especially useful for understanding why a container restarted unexpectedly, exactly when a health check failed, or when a network was reconnected. Without docker events, such events can often only be reconstructed from the daemon logs.

With --filter you can narrow the event stream to relevant types. docker events --filter type=container --filter event=die --filter event=oom shows only container deaths and OOM events. With --since and --until you can query past events, handy when a failure occurred before you started Docker debugging. docker events --format '{{json .}}' outputs machine readable JSON that can be integrated into scripts.

# Stream all container events in real time
docker events --filter type=container

# Filter only crash and OOM events since last hour
docker events \
  --filter type=container \
  --filter event=die \
  --filter event=oom \
  --since "1h"

# Show processes inside container, works even without ps installed
docker top my-app
# With custom ps format: show PID, PPID, state, CPU, and full command
docker top my-app -eo pid,ppid,stat,pcpu,args

# Detect zombie processes (state Z = defunct)
docker top my-app -eo pid,ppid,stat,args | grep " Z "

# Fetch logs with timestamps, limit to last 50 lines, follow
docker logs --timestamps --tail 50 --follow my-app

# Copy a file out of the container for analysis (no exec needed)
docker cp my-app:/var/log/app/error.log ./container-error.log

7. docker logs: reading output and timestamps

docker logs reads the buffered output of the container's main process from the Docker daemon. For Docker debugging the --timestamps flag is indispensable, since it shows the exact time of every output line. With --since and --until you can narrow the time range, --tail N limits output to the last N lines. Important to know: docker logs only works when the logging driver is json-file or journald. With the syslog or gelf driver, logs are not stored locally and docker logs returns an error.

A common problem in Docker debugging with logs: the container process does not write to stdout/stderr but to files instead. In that case docker logs provides no useful information. With docker exec container tail -f /var/log/app/error.log you can still access those logs, or you can copy them out with docker cp container:/var/log/app/error.log ./ onto the host. The docker cp command also works on stopped containers, which makes it particularly valuable for post mortem Docker debugging.

8. nsenter: entering container namespaces from the host

nsenter is the tool of last resort in Docker debugging: it enters the Linux namespaces of a process from the host, without needing Docker at all. This is especially useful when the Docker daemon itself is hanging, when the container has no shell interpreter, or when you want to access a container's network namespace with host tools such as tcpdump or strace. You can find the PID of the container's main process with docker inspect --format '{{.State.Pid}}'.

With nsenter --target PID --net ip addr you can query the container's network configuration using the host's ip command, ideal when the container has no ip or ifconfig installed. With --mount --pid --net --ipc --uts you enter all namespaces at once and effectively get a shell in the container's context, but with the host's toolset. This is the most elegant solution for Docker debugging on minimal production images.

# Get the host PID of the container's main process
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' my-app)
echo "Container PID on host: $CONTAINER_PID"

# Enter the container's network namespace with host tools
# Useful for tcpdump, ss, ip, even if container has no shell
nsenter --target "$CONTAINER_PID" --net -- ip addr
nsenter --target "$CONTAINER_PID" --net -- ss -tlnp
nsenter --target "$CONTAINER_PID" --net -- tcpdump -i eth0 -c 100 -w /tmp/capture.pcap

# Enter all namespaces, effectively a shell in container context with host tools
nsenter --target "$CONTAINER_PID" --mount --pid --net --ipc --uts -- bash

# Attach strace to a specific process inside the container
WORKER_PID=$(docker exec my-app cat /var/run/php-fpm.pid)
nsenter --target "$CONTAINER_PID" --pid -- strace -p "$WORKER_PID" -e trace=network,file

# Inspect /proc filesystem for memory maps and open files
ls -la /proc/"$CONTAINER_PID"/fd | wc -l        # number of open file descriptors
cat /proc/"$CONTAINER_PID"/status | grep VmRSS   # actual resident memory

9. Diagnostic tools compared

The various Docker debugging tools each have their own strengths and use cases. Choosing the right tool for the question at hand significantly reduces diagnosis time.

In practice the most efficient Docker debugging sequence is: docker inspect for a quick configuration check, then docker stats and docker top for resources and processes, then docker logs for output, and only as a last step docker exec for interactive investigation. Anyone who knows and follows this order wastes no time in an open shell when the answer was already visible in inspect or stats.

10. Summary

Systematic Docker debugging of running containers follows a clear tool selection principle: docker inspect for configuration and status, docker stats for resource usage, docker top for processes, docker logs for output, docker events for the timeline of daemon events, and docker exec for direct investigation of the container filesystem. Each of these tools answers a different question and is most valuable at a different stage of troubleshooting. The most efficient Docker debugging starts from the outside and works its way in.

For containers with no shell or on minimal images, nsenter is the way to apply host tools like tcpdump, strace or ip inside the container's namespace. Since Docker 24, docker debug simplifies this process with a built-in debug toolbox. Combining docker events with a time filter and docker inspect .State.OOMKilled enables post mortem analyses that still yield relevant information even after a container has been restarted.

11. FAQ: Docker Debugging for Running Containers