Presenters
Source
Reimagining AI Storage: From Stalled GPUs to Blazing-Fast Innovation 🚀
Hey AI enthusiasts and tech pioneers! We’re Sidharth and Venkat, and today we’re pulling back the curtain on how we’ve completely re-architected Meta’s storage infrastructure to keep pace with the insatiable demands of AI workloads. Get ready to dive deep into the challenges, the innovations, and the mind-blowing results! ✨
The Evolving Data Landscape 🌐
The world of data is in constant flux, and for AI, these changes are amplified. We’ve witnessed three seismic shifts:
- Data Deluge: The sheer volume of data used to train frontier models has seen a staggering 10 order magnitude increase. That’s a number so big, it’s hard to comprehend!
- Multimodal Data: We’ve moved beyond just text. Today’s AI models feast on a diet of text, audio, long-form video, and short-form video, further ballooning data requirements.
- AI-Generated Content: While human-generated content still dominates, AI-generated content is rapidly catching up, adding another layer of complexity and volume.
Why Storage is the Unsung Hero (or Villain) of AI 💡
You might think of GPUs as the stars of the AI show, but storage is the critical supporting act. Here’s why it matters so much:
- The GPU Stall Crisis: In a 120-second snapshot of GPU performance, we observed around 5 stalls, meaning GPUs were idle for about 20% of the time. This isn’t just a minor inconvenience; for data centers the size of Manhattan, that 20% stall translates to tens of millions of dollars lost every hour. 💸
- The Opportunity Cost: Beyond the financial hit, stalled GPUs mean slower research. The pace of innovation is paramount. In 2020-2022, major model upgrades took around 104 weeks. By 2026, we anticipate new frontier models emerging every four weeks. Research velocity is as crucial as cost. 🏎️
Meta’s Storage Evolution: A Journey Through Time 🕰️
Our storage journey at Meta has been one of continuous innovation:
- 2024: Llama Workloads: We focused on improving our file API to specifically support Llama workloads.
- 2025: QLC Deployment: We achieved one of the largest production deployments of QLC technology globally.
- Today: Object API Optimization: We’ve fine-tuned our object API path on top of our block layer to empower upcoming Meta models.
Our overarching theme today is twofold: maximizing GPU utilization for cost-efficiency and accelerating research velocity.
Maximizing GPU Utilization: Tackling the Data Bottleneck 💾
Let’s dive into how we’re ensuring our GPUs are always crunching data, not waiting for it.
The Unique Challenges of AI Workloads 👾
AI model training presents a unique storage challenge:
- Massive Parallelism: Hundreds and thousands of GPUs operate simultaneously.
- Synchronized Operations: GPUs often synchronize before data loading or checkpointing, creating intense bursts of activity.
- Bursty Reads/Writes: This synchronization leads to high IOPS demand.
- Varied IO Patterns: Different workloads exhibit vastly different IO patterns, making a one-size-fits-all approach difficult.
The Data Loading Dance 💃
Imagine GPUs processing data in batches while CPUs prefetch the next batch. This works seamlessly as long as prefetching is faster than GPU processing. However, when storage latency spikes, GPUs stall. And because they synchronize, even a fast GPU waits for its slower counterpart, wasting precious compute cycles.
Key Takeaway: Predictable latencies, all the way up to Pmax, are critical for AI workloads.
Rethinking Legacy Block Storage 🔄
Our traditional block storage architecture, designed for Meta’s family of apps (Facebook, Instagram), faced significant hurdles with AI:
- Metadata Lookups: Mapping a blob name to a storage address involved a series of metadata lookups, some spanning regions. This order of N operation was a non-starter for AI’s millisecond access needs.
- Global Replication: To ensure high availability, data and metadata were replicated globally. While great for user-facing apps, this hurt AI performance.
- HDD-Centricity: Optimized for dollar-per-byte with HDDs, it couldn’t meet the high IOPS demand of AI, which requires flash.
- Power Inefficiency: Every kilowatt spent on storage was one less for GPUs.
- Consistency Semantics: Traditional workloads often require strong consistency, whereas AI’s “write once, read many” datasets can benefit from relaxed semantics.
The New Foundation: A Paradigm Shift 🛠️
We’ve rebuilt our storage foundation with three key changes:
- Unified Schema: Distributed metadata is now consolidated into a single, flat schema, enabling an order of 1 lookup for blob-to-address mapping.
- Client-Side SDK: We’ve eliminated the data plane proxy, pushing logic into the client-side SDK. This allows clients to stream bytes directly from storage nodes, boosting throughput, lowering latency, and improving power efficiency.
- Regional Deployment: The architecture is now deployed regionally, co-locating data and metadata with GPUs.
Taming the Bursts: Read Plan and Hot Object Caches 🔥
To handle bursty reads and writes, we’ve introduced two powerful components:
- Read Plan Cache: A distributed, in-memory store holding blob-name-to-storage-address mappings. It serves high QPS with low latency, absorbing spikes effectively.
- Hot Object Cache: We repurpose a small portion of GPU host memory as a peer-to-peer cache. In production, this achieves an impressive 80% cache hit rate across various workloads.
These mechanisms, coupled with a host of further optimizations, have significantly improved GPU utilization.
Maximizing Research Velocity: Accelerating Innovation 💡
Beyond raw performance, we’re empowering researchers to iterate faster and innovate more rapidly.
The Researcher’s Workflow Bottleneck 🐢
A typical researcher’s workflow involves:
- Curating datasets.
- Selecting a training region.
- Copying datasets to that region (this can take hours to days!).
- Running the training job.
- Analyzing outputs and iterating.
The data copying step is a major bottleneck, especially since GPUs are distributed across many clusters globally. For performance, data needs to be co-located with GPUs.
The “Ingest Once, Consume Anywhere” Dream 🌟
For the vast majority of jobs that complete within hours, iteration speed is paramount. We’ve adopted a “write once, read many” philosophy, inspired by operating system concepts:
- Global Blob Storage as the “Disk”: Researchers ingest data once into our global blob storage, serving as the single source of truth.
- Tiered Caching: We’ve built a tiered caching solution leveraging GPU host memory (L1), NVMe (L2), and regional flash-backed blob storage (L3).
- On-Demand Hydration: When a job needs data in a specific region, the client-side SDK transparently hydrates it from global storage through the tiers, all the way to GPU memory.
Hiding Latency with Smart Prefetching 💨
To mitigate potential latency introduced by on-demand hydration, we employ two prefetching techniques:
- Data Loader Prefetch: The data loader fetches the next batch of data while the GPU processes the current one.
- Background Prefetch: The data loader proactively identifies and prefetches data needed in the coming minutes.
These mechanisms effectively hide latency, keeping GPUs utilized.
Tangible Results: A 90% Reduction in Ingestion Time! 📊
Our on-demand hydration technique has led to a remarkable 90% decrease in ingestion times across the board. This approach is rapidly becoming the default.
Key Takeaways and the Road Ahead 🛣️
- Data-Hungry AI: Modern AI workloads are incredibly data-intensive, with data volumes growing exponentially.
- Storage is Critical: High storage performance is non-negotiable for maximizing GPU utilization and research velocity.
- Rethinking the Stack: The unique demands of AI necessitated a complete re-architecture of our storage stack.
- Global Data, Local Access: Tiered caching, pre-fetching, and on-demand hydration are essential for accessing globally distributed data efficiently.
The journey is far from over! We’re pushing storage to network limits, optimizing checkpointing, and now focusing on the next frontier: inference workloads. The pace of innovation in AI storage is relentless, and we’re excited to continue leading the charge.
Thank you for joining us on this deep dive!