Infrastructure

Kubernetes and cgroup v1 vs. v2: Adoption Requirements and Considerations

Kubernetes traditionally uses cgroup v1, but cgroup v2 offers a unified approach, requiring kernel 5.8+, supported runtimes, and specific Kubernetes versions for stable adoption.

AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Understanding cgroup v1 and v2 in Kubernetes

In Linux, cgroups (control groups) are a kernel feature used for managing system resources. Kubernetes has traditionally relied on cgroup v1 for resource isolation and enforcement. However, cgroup v1 has several limitations, including a fragmented hierarchy and inconsistent behavior across different subsystems. The Linux kernel documentation describes both interfaces, with cgroup v2 offering a more unified and simplified approach. One known issue with cgroup v1 is that it remains alpha in v1.36, but now separates memory throttling from memory reservation and adds tiered memory protection. Memory QoS is available only on Linux nodes that use cgroup v2.

Requirements for Adopting cgroup v2 in Kubernetes

Adopting cgroup v2 in Kubernetes requires several key prerequisites. First, you must be using a Kubernetes version that supports v2 cgroup management, which has been stable since Kubernetes v1.25 and is included in all supported releases. Additionally, you need at least one Linux node, as cgroup is a Linux-only concept. The operating system must be configured to run with cgroup v2 enabled. The kernel version must be 5.8 or later, with 5.9 or later recommended for memory QoS features. Finally, the container runtime must support cgroup v2. These requirements ensure compatibility and proper functionality when transitioning to cgroup v2 in Kubernetes environments.

Kernel and Runtime Support for cgroup v2

The shift to cgroup v2 in Kubernetes requires careful consideration of kernel and runtime support. When Kubernetes was first announced in 2014, only cgroup v1 was available. However, as of Kubernetes documentation, cgroup v2 support requires a minimum kernel version of 5.8, which introduced the system-level cpu.stat file in the root cgroup. This update enables more accurate resource accounting and control. In addition to kernel requirements, the runtime must implement the RuntimeConfig CRI RPC, which is supported by containerd v2.0+ or CRI-O v1.28+. These requirements ensure compatibility and stability when transitioning to cgroup v2, but they also highlight the need for updated infrastructure components to fully leverage the benefits of this newer cgroup version.

Configuration and Feature Gates for cgroup v2

Enabling cgroup v2 in Kubernetes requires careful configuration through feature gates and kubelet settings. For instance, to activate tiered memory protection, administrators must set MemoryQoS: true and configure memoryReservationPolicy: TieredReservation within the KubeletConfiguration API. According to Kubernetes documentation, the memoryThrottlingFactor can be adjusted to 0.9 for fine-grained control. However, the Kubernetes project advises against enabling Alpha features in production environments without thorough testing. Additionally, the singleProcessOOMKill flag should only be set to true if cgroup v1-compatible behavior is required, as this may impact system stability. As noted in v1.36, accurate enforcement of certain features depends on cgroup v2, highlighting the importance of ensuring compatibility during upgrades.

Troubleshooting and Verification Tools for cgroup v2

When transitioning to cgroup v2 in Kubernetes, having the right tools for troubleshooting and verification is essential. One key command is stat -fc %T /sys/fs/cgroup/, which checks if cgroup v2 is enabled by returning cgroup2fs. Additionally, verifying resource allocation discrepancies between pod specifications and actual enforcement is crucial. Using kubectl commands, administrators can retrieve the node name and compare container resource specifications with enacted values in the pod status. This helps identify potential misconfigurations or out-of-sync states, especially after in-place resizes. These commands should be executed on the target node as root, assuming the systemd cgroup driver and cgroup v2 are in use. For more detailed analysis, tools like jq may be required. Proper verification ensures that resource constraints are accurately enforced, minimizing performance issues and ensuring stable cluster operations.