GPU Cluster Management and Scheduler Optimization at Ai2 and Hugging Face

Ai2 and Hugging Face manage large GPU clusters with diverse workloads, implementing scheduler optimizations that significantly reduced queue wait times and improved resource allocation efficiency.

AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Cluster Infrastructure and Workload Diversity

Cluster Infrastructure and Workload Diversity is a critical consideration in managing GPU resources effectively. At Ai2, thousands of NVIDIA H100, B200, and B300 GPUs are organized into clusters ranging from 88 to 1024 GPUs, designed to support large-scale distributed training of AI models. These clusters serve approximately 150 internal researchers working across a wide range of AI domains, including large language model (LLM) and vision-language model (VLM) training, robotics reinforcement learning (RL) simulation, and post-training for scientific agentic use cases. This diversity in workload demands flexible and efficient scheduling strategies to ensure optimal utilization of resources. As one anecdote illustrates, a search company once required users to guarantee high utilization before allocating dedicated machines, highlighting the challenges of managing heterogeneous and unpredictable workloads in GPU clusters.

Scheduler Optimization and Performance Metrics

Scheduler optimization in GPU clusters has shown measurable improvements in performance metrics. According to Hugging Face, prior to implementing the budget-based system, the team aimed to identify potential bottlenecks and test configuration parameters such as lookback window length and minimum runtime thresholds. Hugging Face highlights that the new system significantly reduced p90 debug workload wait times from approximately 6 hours to just 5 minutes. Furthermore, Hugging Face confirms that the actual performance aligned closely with simulated predictions, as the p90 queue wait time for debug workloads dropped from 2 hours to 30 seconds, matching the expected outcome of 6 hours to 5 minutes from hand-crafted test scenarios. These results demonstrate the effectiveness of the scheduler optimization in reducing wait times and improving resource allocation efficiency.

Occupancy and Allocation Efficiency

Occupancy and allocation efficiency are critical factors in managing GPU clusters effectively. According to Hugging Face, occupancy on the cluster remained steady at 98% before and after the change, with demand consistently exceeding capacity by 2-3x in both periods. This highlights the persistent challenge of meeting high demand with limited resources. Hugging Face notes that 18% of delivered GPU time was unallocated, which helped maintain high occupancy during periods when funded use cases were not ready to run. Additionally, Hugging Face reports that over a 30-day test period, teams were delivered 98% of the GPU hours they were owed, with 13 of 15 team allocations receiving 95% or more, and the worst case receiving 90%. These figures underscore the importance of balancing resource allocation with actual usage patterns to ensure fair and efficient distribution of GPU time.

Queue Latency and Time-Slicing Benefits

Impactful scheduling for GPU clusters has shown measurable improvements in queue latency through the implementation of time-slicing techniques. According to Hugging Face, on the largest H100 cluster, the median queue wait time decreased from 5 minutes to 24 seconds, and the p90 wait time dropped by approximately one-third, from 2.8 hours to 1.8 hours. These improvements were observed as a side effect of time-slicing, which allows for more efficient utilization of GPU resources. Additionally, Hugging Face notes that time-slicing introduces a constraint on protected runtime, limiting sessions to an 8-hour cap after which they become preemptible if they exceed their allocated time. This approach helps manage resource contention while ensuring fair access to GPU resources. Research into debug workloads, as described in Hugging Face, highlights the importance of short, low-GPU-demand jobs that require at least 15 minutes of runtime to determine success or failure due to bugs or misconfigurations.

Future Infrastructure and User Experience Improvements

Future Infrastructure and User Experience Improvements will focus on enhancing scheduling mechanisms for GPU clusters. While current efforts include investing in a CPU-only cluster adjacent to on-prem storage for development sessions centered on data preparation tasks, the emphasis remains on refining workload management. Priority inflation is a notable feature, as workloads can still declare priority, though this only influences sorting within a team. Additionally, restorable sessions are planned for CPU-only workloads, aiming to improve user experience and workflow continuity. These steps reflect a strategic approach to infrastructure development, ensuring that both GPU and CPU resources are utilized effectively while maintaining flexibility in task prioritization and session management.