How to Debug Containerized Test Environments

Containerized test environments simplify testing by isolating applications and their dependencies, ensuring consistency across development, CI/CD, and production. However, debugging these environments can be challenging due to their isolation, minimal setups, and temporary nature. Here's a quick guide to tackling debugging issues effectively:

• Start with the Basics: Ensure Docker is installed and running. Check for errors like "Permission denied" and resolve them by adding your user to the Docker group.
• Key Debugging Tools: Use commands like docker exec for shell access, docker logs for output logs, and docker inspect for container metadata. For distroless images, Docker Debug tools or debug containers like nicolaka/netshoot can help.
• Analyze Crashes: Check exit codes (e.g., 137 for memory issues) and preserve logs or snapshots (docker commit) for further analysis.
• Network Diagnostics: Use debug containers or Kubernetes tools to troubleshoot connectivity and DNS resolution.
• IDE Integration: Configure remote debugging in IDEs like Visual Studio Code or JetBrains Rider by mapping local and container paths and exposing necessary ports.
• Best Practices: Implement structured logging, health checks, and multi-stage builds to create debug-ready images. Document common debugging scenarios to streamline troubleshooting.

Debugging containerized environments requires a mix of tools, techniques, and preparation. By following these steps, you can identify and resolve issues efficiently.

Debugging Innovation with Docker (DockerCon 2023)

sbb-itb-7ae2cb2

Prerequisites and Initial Setup

To get started, confirm that Docker is installed and running smoothly. Run the command docker version. If you only see Client details without Server information, it likely means the Docker daemon isn't running. On Linux, you can check its status with sudo systemctl is-active docker. This step ensures that your environment is ready for debugging. Once the environment is stable, you can begin mapping out detailed test scenarios to validate your application's behavior.

Encountering a "Permission denied" error for /var/run/docker.sock? That’s a sign you need to add your user to the Docker group. Use the command sudo usermod -aG docker $USER, then log out and back in to apply the changes. Afterward, test your setup by running docker run --rm hello-world. If this works, Docker is properly configured to pull images, create containers, and manage networking.

Familiarize yourself with key Docker commands for debugging:

• docker exec: Access a container's shell.
• docker logs: View output logs.
• docker cp: Transfer files between host and container.
• docker inspect: Retrieve container metadata.

For minimal or distroless images, Docker Debug (available on Docker Desktop 4.33+ for Pro, Team, and Business users) can be a lifesaver. It attaches a debugging container equipped with essential tools like vim, htop, and curl. Once you’ve verified your CLI tools, ensure your containers are configured for debugging.

When planning your test strategy and setting up containers, expose the necessary ports for debugging. For instance:

• Use -p 9229:9229 for Node.js.
• Use -p 5678:5678 for Python with debugpy.

Additionally, mount your source code as a volume using -v ./src:/app/src. If a container crashes during testing, you can preserve its state by running docker commit <container_id> to create a snapshot.

To make Docker output easier to read, use formatting tools like jq. For example, run docker container ls --format '{{ json . }}' | jq to avoid clipped or messy output. This can save you time and effort when managing multiple containers.

Inspecting and Interacting with Running Containers

Using Docker Exec for Shell Access

One of the quickest ways to inspect a running container is by using the docker exec command:

docker exec -it <container_name> <shell_path>

Here’s how it works: the -i flag keeps the standard input open, while the -t flag allocates a pseudo-terminal for interactive use. Importantly, opening a shell this way does not restart the container.

Start by attempting to use bash for shell access. If the container uses Alpine Linux or another minimal image, switch to sh instead. For cases where you encounter permission issues while inspecting files, you can elevate privileges by adding the -u root flag.

"The docker exec command enables you to run commands or access a shell inside an already running Docker container without restarting it." - Brian Boucheron, Senior Technical Writer, DigitalOcean

If the container is a distroless or slim image that doesn’t include a shell, you’ll need to prepare in advance by setting up Docker Debug tools. Additionally, you can simulate specific runtime conditions by injecting environment variables using the -e or --env-file flags during your debugging session.

Once inside the container, you can begin diagnosing issues by examining processes and resource usage. This data is essential for a QA Metrics Analyzer to track performance trends.

Checking Process States and Resource Usage

After accessing the container’s shell, the next step is to analyze running processes and resource usage to identify potential performance bottlenecks.

• Use ps aux to list all active processes and their resource consumption.
• Run top -bn1 to pinpoint processes consuming the most CPU.
• Check disk space with df -h to locate full or nearly full partitions.
• Use free -m to monitor memory usage and identify any shortages.

From the host machine, the docker stats command provides a real-time overview of CPU usage, memory consumption (relative to limits), and network I/O. If you prefer a single snapshot instead of continuous updates, run:

docker stats --no-stream

If the container exits with a code 137, it indicates a SIGKILL due to an out-of-memory event. Identifying these failures early is a key part of using a QA Risk Analyzer to prevent production downtime. To confirm, inspect the container's OOMKilled flag using docker inspect.

For CPU-related issues, check for throttling by inspecting /sys/fs/cgroup/cpu.stat inside the container. Compare the nr_throttled and nr_periods values to calculate the throttling percentage. If CPU usage registers at 0% but the container still performs poorly, use top to examine the %wa (I/O wait) metric. A high %wa suggests that disk access might be slowing down processes.

Analyzing Logs and Diagnosing Crash States

Retrieving Logs with Docker Logs

Once you've completed resource diagnostics, the next step is to review application logs for errors. The docker logs command is your go-to tool for accessing everything sent to STDOUT and STDERR.

Start simple: use docker logs <container_id> to fetch all output. If the container has already stopped or crashed, run docker ps -a to locate its ID. Remember, Docker retains logs even after a container exits - unless you've removed it with docker rm.

To make sense of large log files, filtering is key. For example:

• Use --tail 100 to display only the last 100 lines.
• Add --since 30m to view logs from the past 30 minutes.
• Include --timestamps to add RFC3339 timestamps, which can help you align container events with external failures.

For real-time monitoring, the -f (or --follow) flag streams new log entries as they appear.

"Docker logs are the fastest way to understand what is happening inside a running container when something breaks, slows down, or behaves unexpectedly." - Atmosly

If logs aren’t showing the expected output, there are a few things to check. Python applications, for instance, may buffer output. To fix this, set the PYTHONUNBUFFERED=1 environment variable to ensure logs are displayed immediately. Additionally, confirm the logging driver with docker inspect <container> | grep LogDriver. If it's set to syslog or none, docker logs won't retrieve anything. Some applications may also log to internal files (e.g., /var/log/nginx/access.log) instead of STDOUT. In such cases, use docker exec to explore these files directly. Once you've gathered the logs, secure any crash artifacts before they’re lost.

Handling Crash States and Preserving Artifacts

When a container crashes, preserving evidence is critical. You can redirect logs to a file using docker logs <container_id> > crash_report.log for later analysis. Always save logs before removing a container, as deletion erases them permanently.

"Once you remove a container with docker rm, all its logs are permanently deleted. If you need to preserve logs after container removal, you'll need to use external log drivers or copy logs to persistent storage first." - Preeti Dewani, Technical Product Manager, Last9

To understand why a container crashed, check its exit code with:

docker inspect <container> --format '{{.State.ExitCode}}'

Common exit codes include:

• 137: Indicates termination due to memory limits (verify this by checking the State.OOMKilled field).
• 139: Points to a segmentation fault.
• 1: Suggests a generic application error.

For deeper investigation, snapshot the container using docker commit <container_id> debug_image:latest. You can then extract critical files with docker cp <container_id>:/path/to/logs/ ./local_dir, allowing you to recreate the failure environment locally without affecting production.

To prevent future disk space problems while keeping enough log history, configure log rotation. Use options like --log-opt max-size=10m and --log-opt max-file=3 to balance storage use and log retention effectively.

Debugging Network and Process Issues

Using Debug Containers for Network Diagnostics

Troubleshooting network issues in containerized environments can be tricky, especially when production images are stripped down to the bare essentials. Instead of modifying these images to include diagnostic tools, you can use debug containers that share the same network namespace as your application.

A popular choice is the nicolaka/netshoot container, which has been pulled over 100 million times on Docker Hub. It comes preloaded with tools like tcpdump, iperf3, nmap, dig, and curl, giving you everything you need for network diagnostics without inflating your production images.

"kubectl debug revolutionizes network troubleshooting in Kubernetes by providing instant access to comprehensive network tools without modifying pod definitions or rebuilding images." - Nawaz Dhandala

For example, you can use dig <service-name>.<namespace>.svc.cluster.local inside the debug container to confirm if Kubernetes DNS is resolving correctly. To check connectivity, the command nc -zv <destination-ip> <port> can verify if a specific port is accessible, bypassing application-related issues. If packet capture is required, the command tcpdump -i any port <port> -w /tmp/capture.pcap allows you to save network traffic for later analysis in tools like Wireshark.

From Kubernetes 1.25 onward, you can attach ephemeral debug containers with this command:

kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container-name>

This approach shares both the network and process namespaces of the application container, making it particularly useful for diagnosing problems in "distroless" or minimal images that lack built-in troubleshooting tools.

Once network issues are under control, you may need elevated permissions to dig deeper into process-level problems.

Enabling Advanced Linux Capabilities for Inspection

Some problems require more than just network diagnostics. Advanced Linux capabilities can help you investigate process-level anomalies without granting full privileges to your containers, which is especially important in environments prioritizing security.

For example, enabling CAP_NET_ADMIN and CAP_NET_RAW in the debug container's security context allows tools like tcpdump to capture packets or adjust routing tables. If you need to attach debuggers or use tools like strace to inspect running processes, adding CAP_SYS_PTRACE is essential.

When dealing with containers that crash immediately, you can use kubectl debug --copy-to and set an overridden entrypoint like sleep infinity. This keeps the container running, allowing you to inspect its filesystem via /proc/1/root/ from the debug container - even if the original image lacks a shell.

For Docker environments, you can launch a debug container that shares namespaces with the target container:

docker run --pid=container:<id> --net=container:<id>

This mirrors the target container's view of processes and network interfaces. It can be particularly helpful for diagnosing issues like an application mistakenly bound to 127.0.0.1 instead of 0.0.0.0, which can block communication between containers.

IDE Integration and Remote Debugging

Configuring Debugging for Common Platforms

Modern IDEs like Visual Studio Code, Visual Studio, and JetBrains Rider can directly connect to applications running in containers, but this requires specific setup. To get started, you'll need three essential components: a Dockerfile, docker-build and docker-run tasks, and a launch configuration that ties everything together.

A crucial step is setting up path mapping between your local workspace and the container's filesystem. This varies depending on the platform you're using:

• For Node.js, use "localRoot" and "remoteRoot".
• For Python, configure "pathMappings".
• For .NET, rely on "sourceLink" or "pathMap".

Each platform also has its own debugging protocol and default port:

• Node.js: Uses the V8 Inspector on port 9229 with the --inspect=0.0.0.0:9229 flag.
• Python: The debugpy library listens on port 5678.
• .NET on Linux: Relies on vsdbg, which operates on port 4022.

Make sure these ports are exposed in your Dockerfile and mapped correctly when starting the container.

To keep production images lean while enabling debugging, consider using multi-stage builds. This method allows you to create distinct stages in your Dockerfile - one for development (with debugging tools like vsdbg or debugpy) and another for production. For development, you can include extras like SSH servers, debugger agents, or source maps, while keeping the final production image minimal.

If you're working with containers that lack shells or preinstalled debugging tools (e.g., distroless images), the Dev Containers extension in Visual Studio Code can be a game-changer. It runs a VS Code Server instance inside the container, offering features like IntelliSense, code navigation, and debugging. Additionally, the Docker extension can help you quickly set up your launch configuration and task files using the "Containers: Add Docker Files to Workspace" command.

Streamlining Debugging with Ranger

Integrating IDEs with Ranger's pre-configured environments takes container debugging to the next level. Managing containerized test environments manually can be time-consuming, especially when juggling multiple debugging sessions across platforms. Ranger simplifies this by providing hosted test environments that seamlessly integrate with your CI/CD pipeline through tools like Slack and GitHub.

Ranger’s real-time testing signals help pinpoint failing tests and their root causes, saving you from the tedious process of combing through container logs or manually attaching debuggers. When its AI-powered system identifies an issue, human reviewers verify the findings, ensuring you're focused on real bugs instead of quirks tied to the environment. This combination of automation and human oversight makes debugging faster and more efficient.

Best Practices for Debugging Containerized Test Environments

Implementing Structured Logging and Health Checks

Switching to JSON-formatted logs can make debugging much easier. For example, if a Node.js app logs messages using console.log("User logged in"), it becomes tough to filter and analyze logs at scale. Instead, structured JSON logs - with details like timestamps, log levels, service names, and correlation IDs - allow you to track requests across containers and pinpoint issues faster using tools like jq.

Stick to the twelve-factor app principle by directing all logs to stdout and stderr rather than internal files. This keeps containers stateless and simplifies log collection. Configure Docker's logging driver with settings like max-size (around 10 MB per container) and max-file to avoid running out of disk space. For production, Docker suggests using the local driver, which offers better performance and built-in compression compared to the default json-file driver.

Adding a HEALTHCHECK instruction to your Dockerfiles is another must. This lets you monitor application readiness through HTTP endpoints, catching issues like database connection failures or resource exhaustion. For Python apps, setting PYTHONUNBUFFERED=1 ensures logs are flushed immediately. Also, include startup validation logic to check for necessary environment variables and database connectivity.

Incorporating these logging and health check practices into your build process ensures a fully debug-ready container image.

Building Debug-Ready Images with Multi-Stage Builds

Multi-stage builds are a great way to create separate development and production stages in a single Dockerfile. In the development stage, you can include debugging tools like vim, curl, htop, and compilers. The production stage, on the other hand, focuses on copying only the final compiled artifacts. Use the --target flag (e.g., docker build --target builder) to stop the build at an intermediate stage. This lets you inspect files, environment variables, and dependencies before finalizing the image.

Organize Dockerfile instructions based on how often they change. For instance, copying package.json and running npm install before adding the full source code can prevent unnecessary dependency reinstalls during development. If a build fails unexpectedly, use flags like --progress=plain for detailed output or --no-cache to force a rebuild. Don’t forget a .dockerignore file to exclude unnecessary files like local logs, node_modules, and git history. This reduces the build context size and ensures cleaner images.

Once your build process is solid, document common debugging workflows to make troubleshooting smoother.

Documenting Common Debugging Scenarios

Technical tools are important, but good documentation is just as critical for quick troubleshooting in containerized environments.

"Developers can spend as much as 60% of their time debugging their applications, with much of that time taken up by sorting and configuring tools and setup instead of debugging." - Tyler Charboneau, Docker

Create runbooks that map common exit codes to their causes. For example, exit code 137 typically means an out-of-memory termination, while exit code 139 points to a segmentation fault. Referencing sections like "Handling Crash States and Preserving Artifacts" can provide deeper insights into these codes. Establish a standard triage workflow that begins with non-invasive steps - like reviewing container details using docker inspect or checking logs with docker logs - before escalating to more involved methods like accessing the container’s shell. Including snapshots from docker inspect in incident reports can help capture the state of mounts, capabilities, and environment variables at the time of failure.

Document environment-specific quirks to address "it works on my machine" issues. Common culprits include missing environment variables, CPU architecture mismatches (ARM64 vs. AMD64), or missing native dependencies. Clearly outline which debug sidecar images (like netshoot or custom Ubuntu-based images) your team uses so engineers don’t waste time installing tools into stripped-down production images. Finally, ensure each fix includes a verification step to confirm the solution works.

Wrapping Up

Debugging containerized test environments doesn't have to swallow up 60% of your development time. By following a structured process - like analyzing logs, inspecting containers, using interactive shells, and running network diagnostics - you can tackle issues efficiently without resorting to disruptive methods.

The strategies outlined here - effective logging, proactive health checks, streamlined multi-stage builds, and interpreting exit codes - work together to improve container transparency. For example, exit codes like 137 and 139 point directly to problems such as memory limits being exceeded or segmentation faults.

In cases where containers lack shells or crash too quickly, tools like ephemeral debug containers and commands like docker commit allow you to capture the exact failure state for deeper analysis. Remote debugging setups can also bridge your IDE directly to running containers, offering a hands-on way to solve complex issues. These techniques collectively simplify and improve your debugging process.

For teams aiming to optimize their testing workflows, Ranger offers AI-driven QA testing with human oversight. It automates test creation and maintenance, helping you release features faster without compromising quality.

With the right combination of tools, techniques, and clear documentation, debugging becomes a far more manageable and efficient task.

FAQs

What should I do if my container crashes before I can exec into it?

If your container crashes before you have a chance to exec into it, the first step is to check its logs and exit code. Run docker ps -a to see the container's exit status, and then use docker logs <container_id> to review the logs for any errors or issues.

For deeper troubleshooting, you can start a new container equipped with debugging tools or an interactive shell. If you're working with Kubernetes, the kubectl debug command is a helpful way to investigate crashed pods or containers.

How can I debug a distroless image that has no shell or tools?

Debugging a distroless image can be tricky since it lacks a shell or standard tools, making traditional methods like docker exec ineffective. Instead, you’ll need to rely on specialized techniques.

One option is to attach an ephemeral debug container that includes the necessary tools. For Docker, you can use docker debug to connect a temporary container with debugging utilities. If you’re working in Kubernetes, kubectl debug allows you to troubleshoot the container without modifying the original image.

These methods give you access to essential tools, making it possible to diagnose and resolve issues in minimal images.

Why does my container have exit code 137 or 139?

Exit code 137 occurs when the system's Out-Of-Memory (OOM) killer terminates a container because it ran out of memory. This typically happens when the container exceeds its allocated memory limit.

Exit code 139, on the other hand, signals a segmentation fault (SIGSEGV). This usually results from invalid memory access or using binaries and libraries that aren't properly aligned with the container's environment.

To address these problems, start by reviewing the container’s memory allocation and adjust it if necessary. Additionally, ensure that all binaries and libraries within the container are compatible and correctly configured.

Related Blog Posts

• How CI/CD Testing Impacts Engineering Velocity
• Parallel Testing in CI/CD Pipelines
• Scaling Cloud Test Infrastructure for CI/CD Pipelines
• Common CI/CD and Cloud Testing Integration Mistakes