Presenters

Source

Unlocking Exabytes: How Meta Reimagined Training Data Storage for Next-Gen Recommender Systems 🚀

The world of recommender systems is hungry. Not just for better algorithms, but for data. Massive amounts of it. We’re talking exabytes – that’s a 1 followed by 18 zeros! At Meta, this insatiable appetite for training data has driven a monumental effort to scale their storage infrastructure. But as new data paradigms emerge, like the rich and complex “user sequences,” the old solutions simply buckle under the pressure.

This is the story of how Meta tackled this colossal challenge, transforming their training data storage from the ground up. Join us as we dive into the innovative strategies and critical optimizations that power the next generation of AI models.

The Exabyte Challenge: Why Data Storage is King 👑

In the realm of recommender systems, more data equals better predictions. Models need to ingest exabytes of information to truly understand user behavior and deliver personalized experiences. This reality has made training data storage a top priority, a sentiment echoed in numerous research papers.

Meta has invested heavily in this area, developing critical optimizations to keep pace with rapid data growth. Their initial successes were built on four pillars:

  • Columnar Compression: Squeezing more data into less space by compressing data column by column.
  • Row Reordering: Arranging data rows for more efficient access.
  • Column Reordering: Optimizing the order of columns within data for faster retrieval.
  • Model-Aware Placement: Strategically placing data based on how models will access it.

These techniques worked wonders for traditional features like dense and sparse data, exploiting data similarities and common access patterns to minimize physical data copies. But then, a new data paradigm arrived, changing everything.

The User Sequence Revolution: A Data Paradigm Shift 🔄

Enter the user sequence. This isn’t just a snapshot; it’s a comprehensive history of user interactions spanning months, even years. User sequences encode incredibly rich insights into user behaviors, content understanding, and contextual information.

The impact? High data density. User sequences are 100x or more larger than traditional features.

On the modeling front, Meta embraced this new paradigm, pivoting to learn from user sequences for both advertisement and organic content recommendations. This led to market-leading papers showcasing significant improvements across 12 AI models. The longer the user sequence, the more substantial and steady the model quality gains.

However, on the data side, this explosion in size meant user sequences became the single dominant driving factor of training data storage from 2023 onwards. Even with existing optimizations, the growth further accelerated.

The result? Training data now spans tens of exabytes, spread across more than 10 data centers. This massive undertaking consumes 10% of Meta’s power budget.

The Costly Consequences: More Than Just Money 💸

Beyond the staggering cost, this exabyte scale data introduced a cascade of challenges:

  • Data Downsampling: Due to data center capacity constraints, valuable training samples are constantly being downsampled and discarded upfront.
  • Loss of Long-Term Knowledge: For similar capacity reasons, training samples are often kept for only 1 month, leading to the loss of crucial long-term knowledge like seasonality.
  • Experimentation Hurdles: Experimenting with new data formats requires processing and generating huge volumes of data, a process that is both time-consuming and costly.

Meta’s previous solutions, while strong for traditional features, were no longer sufficient. The landscape had fundamentally shifted. A complete rethink of how training data is generated was imperative.

Normalization: The Key to Unlocking Efficiency 🔑

The core of the problem lay in data duplication. The previous data model relied on fully self-contained rows, where each training row held a complete snapshot of features used at inference. This made reconstruction simple, but it led to massive redundancy.

Why? ML features for training rows close in time often change very little, and in many cases, are exactly the same. This duplication was an acceptable trade-off when training data was smaller and ML features were condensed summaries. But with user sequences, storing significantly more information over longer periods made this model untenable.

The solution? Normalizing training data. This well-established pattern from relational databases proved highly relevant. The core idea is to split data into multiple tables.

In this new model:

  • User sequences, previously duplicated within each training row, are moved to a dedicated user sequence table. Each sequence is identified by a unique user sequence ID.
  • The original training table now contains only a lightweight pointer to the corresponding rows in the user sequence table.

This approach dramatically reduces data duplication.

However, there’s a trade-off: more complex training sample reconstruction. The training preprocessor now needs to perform a join operation, reading from both the training table and the user sequence table to reconstruct samples on the fly. The success of this entire solution hinges on optimizing these joins.

Sequence Storage: The System Built for User Sequences 💾

To handle user sequences efficiently, Meta built Sequence Storage. Here’s how it works:

  • The training table (with lightweight references) resides on warm storage (HDDs).
  • The user sequence table, optimized for point lookups, is moved to an efficient, indexable storage. Meta leveraged ZippyDB, a distributed key-value store, for this.
  • A specialized connector on the training preprocessor side performs the join between the training table and the user sequence table.

The initial implementation of Sequence Storage proved more inefficient than expected, even costing as much or more than the original single-table approach. This was partly because ZippyDB, optimized for OLTP workloads, wasn’t ideal for training. The complex reconstruction also added overhead.

This necessitated a deep dive into optimizations for both the training preprocessor and the storage server. The focus here is on the groundbreaking storage server optimizations.

Storage Server Optimizations: Turbocharging Throughput ⚡

The primary goal was to reduce the cost per gigabyte per second of training throughput. Three major inefficiencies were identified and addressed:

  1. High Read Amplification:

    • Problem: Frequent background compactions in RocksDB, triggered by row-by-row insertions, consumed significant server resources (flash bandwidth, CPU). This led to high-depth log-structured merge trees, requiring reads to touch multiple SSTable files, amplifying I/O.
    • Solution: Introduced an offloaded SSTable compaction and optimization pass. A new API allows loading pre-generated SSTables directly, bypassing expensive memtables and write-ahead logs. Data pipelines now periodically compact, optimize, and index data outside of the ZippyDB servers. The result: a single-level log-structured merge tree, achieving one I/O per read.

  2. High Read Latency due to Index Join:

    • Problem: RocksDB and ZippyDB performed scans serially. For a batch of user sequences, total latency was the sum of individual scans, leading to GPU QPS regression.
    • Solution: Implemented native multi-scan implementation in RocksDB. The new API identifies IO coalescing opportunities (e.g., adjacent disk blocks) and issues IOs in parallel. This reduced storage media IOPS and ensured latency was determined by the single slowest scan, significantly boosting training GPU QPS.

  3. Data Over-fetching:

    • Problem: User sequences, a time series of events, contain multiple attributes. Models often only need specific events within certain time ranges and a subset of attributes. Fetching the entire sequence and filtering on the GPU side led to increased network throughput and significant CPU/latency overhead.
    • Solution: Made the ZippyDB query engine aware of schematized data. This allows on-demand deserialization of user sequence key-values. Additionally, feature projection filters out uninterested event attributes, and length truncation keeps only relevant events within specific time ranges. These optimizations reduced data fetched by approximately 1/3.

The Results: A 5x Efficiency Leap and Beyond ✨

These comprehensive optimizations yielded remarkable results:

  • 5x More Efficient Storage Servers: Each server now supports over 1 GB/s read throughput, up from 200 MB/s. This drastically reduced the overall setup cost.
  • Massive Sequence Length Scaling: User sequence length increased from 2K events to 80K events—an almost 40x improvement—without a significant cost increase.
  • Production Ready: Sequence Storage is currently serving 30+ models across various product groups.

Key Takeaways for the Future 💡

The journey of Sequence Storage offers critical insights:

  1. Training Data Scale is Paramount: The problem of training data scale is immense and growing. Emerging workloads like user sequences are pushing limits by 5-10x, demanding cost-effective solutions.
  2. Normalization is the Path Forward: Data duplication, once manageable, becomes cost-prohibitive with the increasing data density of user sequences. Normalization is the only viable path.
  3. Full-Stack Innovation is Crucial: Handling emerging AI/ML workloads requires innovation across the entire stack, from client-side optimizations to storage server enhancements, to ensure cost-effectiveness and practicality.

What’s Next for Sequence Storage? 🔮

Meta’s investment in Sequence Storage continues, with two major areas of focus:

  • Scaling Sequence Length: The ultimate goal is to increase sequence length by 10x to capture lifetime user history. This will necessitate exploring techniques like smarter encodings and tearing.
  • Generalization Beyond User Sequences: The ambition is to leverage the Sequence Storage framework for other feature types, such as item features, content features, and embeddings. The vision is a canonical storage for all normalized training data at Meta.

This ongoing exploration promises to further revolutionize how Meta handles its vast training data, ensuring its AI models remain at the cutting edge.

Appendix