Introduction
Docker is a tool that revolutionized modern software development, solving numerous issues such as “dependency hell,” portability, and more. You just need to put everything into a container, and it magically works!
Now, what exactly is a container, and how does it work? Many would characterize a container as a “lightweight virtual machine,” which could be a good short answer to create a brief understanding of the term. Actually, it has many significant differences compared to a VM. In the following sections, we will cover the key components of Docker and how it works “under the hood.”
What is a container?
The key difference between a Virtual Machine and a container is that a container uses the same kernel as the host Operating System, while a VM uses a hypervisor to create multiple OS instances and a virtualized kernel as well. VMs are more isolated but also heavier than containers.
Now, the main question is, how is containerization implemented? The answer lies in the following key Linux technologies that are used: namespaces and control groups (cgroups).
Interestingly, these two features are exclusive to Linux, meaning containerization and Docker are not runnable on MacOSX or Windows.Now you could ask – “How am I then running Docker without any issues?”. The thing is that Docker runs a Linux VM on top of the host OS behind the scenes, which makes it possible to run Docker on top of these systems as well.
Control groups
Control groups (cgroups) are a Linux feature that allows you to allocate and manage system resources – like CPU, memory, disk I/O, and network bandwidth – among processes.
The Linux kernel has components called subsystems. A subsystem in Linux is a component of the kernel or user space responsible for managing specific system functions, such as process scheduling, memory management, file systems, networking, or device control.
Each subsystem has a tree. Each running process is in a node in that tree. Each tree starts with the root node. Each node equals a group of processes sharing the same resources. On the following image we can see a tree view of the subsystem for the CPU. The PID for each running process must be somewhere in that tree. For example, “nginx” is a cgroup of 3 processes that share the same memory constraints and limits:
PID 1 (which is always “systemd”) is placed at the root of each tree. New processes start in the same group as their parents, but they can be moved.
The previous structure was changed in a newer version of this feature, called cgroup v2. Cgroup v2 uses a single unified hierarchy. That means that all subsystems are part of one tree. Cgroup v2 is a newer, unified version, and it’s generally recommended for modern systems. It simplifies cgroup management. It was released in the Linux kernel 4.5 in 2016., and became the default version in Linux kernel 5.15, which was released in 2021.
Now we can play with cgroups. Let’s say we want to limit a few processes to 1GB of memory. First of all, we need to create a cgroup. We do that by creating a directory inside /sys/fs/cgroup (cgroup v2 is used):
mkdir /sys/fs/cgroup/my_cgroup
The /sys folder is a virtual file system. Content inside that file system is not stored on the disk, but it contains information about the system and hardware configuration. Here, we also have the information about cgroups stored.
The next step is to set the memory limit for the cgroup. To do so in v2, we write the corresponding limit to the following path:
echo 1073741824 > /sys/fs/cgroup/my_cgroup/memory.max
To now add a process with the PID 1234 for example, we write the PID to a file called “cgroup.procs” that is inside our “cgroup” folder:
echo 1234 > /sys/fs/cgroup/my_cgroup/cgroup.procs
Monitor the processes: To ensure the processes are within the memory limit, you can monitor the memory usage using tools like top, htop, or by checking the memory usage in the “memory.usage_in_bytes” file:
cat /sys/fs/cgroup/my_cgroup/memory.usage_in_bytes
Namespaces
Namespaces are the second fundamental feature of the kernel mentioned here that allow the isolation of system resources. They provide a way to group resources together, so that processes running in different namespaces can have their own separate views of these resources, preventing them from interfering with each other. There are multiple types of namespaces:
- PID namespace – Processes within a PID namespace only see processes in the same PID namespace. Each PID namespace has its own numbering.
- Mount Namespace – Ensures that each container has its own separate filesystem and can mount file systems independently.
- UTS Namespace – Allows containers to have their own hostname and domain name.
- Network Namespace – Provides network isolation, allowing each container to have its own network stack (interfaces, IP addresses, routes, etc.).
- IPC Namespace – Isolates inter-process communication mechanisms like shared memory.
- User Namespace – Enables containers to map user IDs inside the container to different user IDs on the host, improving security.
- Time Namespace – This namespace was released quite recently in Linux (2020), it allows having different system times within our system by specifying different time namespaces.
- Cgroup Namespace – Introduced in 2016 as part of Linux release 4.6, limits the resource usage in our system (cpu, memory, disk, etc) for a particular group of processes (under this namespace).
For example, to create a new PID namespace, the following command should be run:
sudo unshare --pid --fork --mount-proc /bin/bash
Container runtime
A container runtime is the software responsible for running containers on a system. It manages the lifecycle of containers, including pulling images, starting, stopping, and managing container processes. The container runtime interacts with the underlying operating system to ensure containers are running efficiently and securely. Docker uses two key components for container runtime functionality: containerd and runc.
Containerd is an industry-standard core container runtime used by Docker (and other container systems) to manage the lifecycle of containers. It handles the high-level container management tasks like pulling container images, managing container execution, and managing container storage. Containerd provides a higher-level API that Docker uses to interact with containers, allowing it to abstract away much of the lower-level details of container operations.
Runc is the low-level container runtime that actually creates and runs the container. It’s responsible for creating and managing containers based on the specifications defined in the container image (e.g., the filesystem, environment variables, and settings). Runc interfaces directly with the Linux kernel’s container features (like namespaces, cgroups, etc.) to create an isolated environment for each container. It is typically invoked by containerd to launch containers.
How Docker uses containerd and runc:
- Docker interacts with containerd as the container management service. Docker sends commands to containerd (such as “run this container”) through its API.
- Containerd, in turn, delegates the actual task of running the container to runc, which directly communicates with the kernel to create and run the container.
- Containerd manages the lifecycle of containers (e.g., stop, restart, and delete containers), while runc focuses on the execution details of each container.
Container creation under the hood
Now, since we introduced all of these components, let’s take a look at what happens when the user runs a command like “docker run”
- User runs the “docker run” command. Docker CLI then sends the request to the Docker Daemon (dockerd). Docker Daemon is the core service that manages Docker containers, images, volumes, and networks. It listens for client requests (CLI, API) and handles them.
- dockerd checks if the image is available locally. Once it assures the image is present, container creation preparation starts.
- The Docker Daemon passes the container creation request to containerd, which handles the container lifecycle. containerd acts as an intermediary between the Docker daemon and the lower-level container runtime.
- containerd hands off the task of starting the container to runc. Runc does all the low-level steps:
- namespace creation – runc uses the clone() system call with flags like CLONE_NEWNS, CLONE_NEWPID, CLONE_NEWNET, etc., to create new namespaces for the container. It forks a child process where these namespaces are applied, so the container gets its own isolated view of mounts, process IDs, network interfaces, and more. For example, the container might have its own loopback interface and see PID 1 as its own process.
- cgroups creation – runc writes to files under /sys/fs/cgroup (v1 or v2 depending on system), e.g., it might echo 500000 to cpu.max or memory.max to limit resources. It creates a new cgroup directory like /sys/fs/cgroup/memory/container_id/, sets limits based on the OCI runtime spec (config.json), and moves the container process into this cgroup by writing its PID to cgroup.procs.
- setting up the file system – runc mounts an overlay filesystem that merges image layers (via overlayfs with lowerdir, upperdir, and workdir), binds necessary host paths (like volumes or /etc/resolv.conf) into the container’s rootfs, mounts pseudo-filesystems like /proc and /sys, and finally performs pivot_root() to make the container’s rootfs the actual /. After that, the original root is unmounted and detached.
As the last step, it starts the container’s entry point, which is the command defined by the container’s image.
5. At this stage, the container is running in its isolated environment, thanks to namespaces, and under resource constraints, as enforced by cgroups. The container has its own independent network stack, process tree, and filesystem, and it is limited to a specific amount of resources, such as memory and CPU. This isolation ensures that the container operates independently of other containers or the host system.
Conclusion
Docker simplifies software development using containers that share the host’s kernel, unlike virtual machines that run their own OS. Containers rely on key Linux features—namespaces and control groups (cgroups)—to ensure isolation and efficient resource management. Namespaces isolate system resources, while cgroups limit resource usage to prevent overloading.
The container runtime, including containerd and runc, manages the lifecycle of containers. Docker commands like “docker run” interact with these components to create and run containers in isolated environments with resource limits. This architecture enables Docker to provide portable, scalable, and efficient containerized applications. Understanding these technologies helps optimize container performance and security.
“How Docker Containers Work Under the Hood: Namespaces and Cgroups” Tech Bite was brought to you by Ahmed Pašić, DevOps Engineer at Atlantbh.
Tech Bites are tips, tricks, snippets or explanations about various programming technologies and paradigms, which can help engineers with their everyday job.
