Presenters

Source

Scaling the AI Frontier: Productionizing and Operating NVIDIA’s GB200 Clusters 🚀

The world of AI is evolving at lightning speed, and at the heart of this revolution are powerful computing clusters. NVIDIA’s latest GB200 and GB300 systems represent a monumental leap forward, but bringing these behemoths to life and keeping them running at peak performance is a complex undertaking. Join us as we dive into the incredible journey of productionizing and operating these cutting-edge clusters at scale, straight from the experts at NVIDIA!

Our Mission: Powering AI Innovation 💡

Sachin Lakharia, Douglas Wightman, and Ankur Srivastava from NVIDIA shared their invaluable experience in a recent talk, shedding light on how their team plays a crucial strategic role in the NVIDIA ecosystem. Their responsibilities span across:

  • Deploy and Scale: Bringing next-generation compute, storage, and networking online and transforming them into high-performance computing (HPC) clusters.
  • Operational Expertise: Optimizing these HPC clusters to deliver peak performance for demanding AI research workloads.
  • Build and Improve: Developing advanced monitoring and researcher productivity tools.
  • Ecosystem Innovation: Advancing the hardware and software stack not just for NVIDIA, but for the entire industry.

The Scale Journey: Exponential Growth Driven by LLMs 📈

The demand for AI computing has exploded, with NVIDIA’s fleet growing a staggering 2 to 2.5x year over year. This phenomenal growth is primarily fueled by the insatiable appetite for large language model (LLM) research. Their fleet boasts a diverse range of NVIDIA architectures, from the A100 and H100 to the groundbreaking GB200 and GB300 clusters.

The GB200: A Supercomputer in a Rack 🌌

Douglas Wightman introduced the revolutionary NVL GB200 system, highlighting its key advancements:

  • Multi-Node NVLink: Unlike previous generations, the GB200 connects GPUs across multiple nodes using direct links via NVSwitches. This transforms 18 nodes into a single, massive supercomputer with 72 fully connected GPUs communicating at full speed.
  • Unprecedented Performance: This architecture is a direct response to the industry’s trend of rapidly increasing model complexity. The Blackwell architecture delivers a 30x inference speedup and a 25x improvement in energy efficiency.
  • NV Switch System: The core enabler is the NV Switch system, which links these 72 GPUs into a single NVLink domain boasting an aggregate bandwidth of 130 terabytes per second.

Revolutionizing Workload Scheduling: Slurm’s Topology Block Plugin 🛠️

To harness the power of the GB200, a significant change was required in their workload scheduling engine, Slurm.

  • The Challenge: Traditional Slurm scheduling was “best effort” and insufficient for the GB200’s architecture. These clusters demand that the job scheduler treat NVLink domains as strict boundaries.
  • The Solution: NVIDIA collaborated with SchedMD to develop the topology block plugin. Each multi-node NVLink domain is now modeled as a rigid scheduling unit or “block.”
  • Granular Control: Administrators can define these topologies in Slurm’s configuration, allowing for fine-tuned bandwidth allocation for large jobs.
  • Efficient Job Distribution: A naive scheduler might scatter a 32-node job across any free nodes. However, Slurm’s topology block algorithm can efficiently distribute the job across multiple blocks, splitting them equally. This is crucial for utilizing the high bandwidth needed by mixture of experts (MOE) workloads.
  • Application Tuning: Developers are encouraged to experiment with segment sizes to maximize performance. Larger jobs with high bandwidth needs benefit from larger segment sizes, while smaller jobs with lower I/O can use smaller segments to optimize cluster utilization.
  • Tradeoffs: While strict topology scheduling guarantees performance, it can potentially impact overall data center utilization. However, these tradeoffs can be mitigated.

Tackling Production Readiness: Simulation and Workload Diversity 🧪

Sachin Lakharia elaborated on the challenges posed by this new scheduling approach, amplified by the sheer scale and diversity of their workloads:

  • Workload Spectrum: They support a wide range of workloads, from LLMs and generative AI to digital humans, with job requirements varying from a single GPU to tens of thousands of GPUs.
  • High Expectations: Users expect high reliability and demand 95% or more cluster occupancy.
  • Occupancy Defined: Occupancy is the total number of GPUs running any workload divided by the total number of GPUs available. The goal is to keep no more than 5% of GPUs idle.
  • The Block Scheduling Dilemma: The hard constraint of block scheduling guarantees job performance but limits scheduler flexibility. Without optimization, this can lead to a single-digit drop in occupancy.
  • Bridging the Gap with Simulation: Since physical GB200 clusters were scarce, NVIDIA built a first-class Slurm simulator. This simulator runs the exact Slurm codebase, supports different versions and configurations, and can replay production data with accelerated runtime (7-day simulation in 7 hours).
  • Simulation Scale: For GB200, they simulated a cluster of 5,000 nodes (equivalent to 20,000 GPUs) with 15,000 jobs over 7 days, modeling reliability by assuming some nodes were in triage.

Key Simulation Findings: Optimizing for Occupancy and Performance 🎯

The simulations yielded crucial insights:

  • Minimal Self-Fragmentation: Contrary to intuition, the block topology plugin was effective at fitting smaller jobs into available slots without significantly fragmenting blocks for larger jobs. Very minimal to no self-fragmentation was observed.
  • Occupancy Recovery: While a “no topology” approach offers theoretical maximum occupancy, simulations showed that with enough small jobs, occupancy loss could be recovered. They achieved occupancy within 1% of the “no topology” benchmark.
  • Workload and Site Considerations:
    • Allowing all jobs to utilize the NVL-72 domain without topology constraints leads to a drop in overall utilization due to stranded capacity.
    • Allowing small jobs to spread helps recover occupancy.
    • Segment size is critical. Disproportionate allocation leads to performance drops.
  • Recommendations:
    • Larger jobs should use a segment size of 16 for maximum performance.
    • Smaller jobs should use a segment size of 2 or 1.
    • These defaults were made user-agnostic and implemented transparently.

From Simulation to Global Rollout: Testing and Validation ✈️

Ankur Srivastava detailed the rigorous testing and validation process that enables the global rollout of GB200:

  • Comprehensive Testing: This includes microbenchmarks for network, compute, and storage, as well as workload tests at various scales (single node, rack, multi-rack).
  • CPU Importance: For MOE workloads, CPU performance is critical, and microbenchmarks are developed to identify CPU-related issues.
  • Straggler Detection: Identifying a single slow GPU or rack can significantly impact training across thousands of GPUs. They employ methods like nickel all-to-all benchmarks and workload-based testing, along with ad hoc analysis to pinpoint stragglers.
  • Adaptive Testing Strategy: The testing strategy must be adaptive, even on a day-to-day basis. If training runs smoothly, testing can be minimal. If issues arise, extensive parallel testing or capacity blocking for cross-rack issues may be necessary.
  • Automation is Key: Developing automation tools for testing is paramount. These tools support various GPUs, offer a library of microbenchmarks and workload tests, and allow submission via a single CLI command. This enables rapid iteration and informed decision-making.
  • Real-World Success: NeMo Tron 3: NVIDIA worked closely with the NeMo Tron team to train a 120 billion parameter model across tens of thousands of GPUs, processing hundreds of billions of tokens per day. This was achieved through software improvements like async checkpointing and garbage collection, along with straggler detection and automated testing.

The Future of AI Infrastructure: Rack-Scale and Continuous Innovation 🌐

Sachin concluded by looking ahead:

  • The Rise of Rack-Scale Computing: Model complexity is driving a shift towards rack-scale computing like GB200 and GB300. Frontier labs are managing clusters with over 50,000 nodes.
  • Addressing Failures: As job sizes increase and failures become inevitable, NVIDIA is focusing on:
    • Continuous Testing and Validation: Moving beyond one-time tests to multi-node and multi-rack continuous testing.
    • Adaptive Efficiency: Leveraging simulation technologies to dynamically fine-tune clusters based on workload and configurations for maximum GPU utilization.
  • Applying Lessons Learned: These hard-won lessons are being applied to upcoming rack-scale systems and stabilizing existing ones like GB300.

NVIDIA is not just enabling itself but the entire AI industry by pushing the boundaries of AI infrastructure and open models. The future of intelligence is being built, and it’s an exciting journey to be a part of!

Appendix