Presenters
Source
Meta’s Bold Leap to the Cloud: From On-Prem Masters to Hybrid Powerhouses ☁️🚀
Meta, the tech giant behind Facebook, Instagram, and WhatsApp, has long been a pioneer in building its own incredibly robust and efficient on-premises infrastructure. For years, they meticulously crafted every layer of their tech stack, from custom hardware in their data centers built through Open Compute to specialized software like HHVM for their social graph. This “code design” approach, where hardware and software are optimized together, fueled their rapid growth and reliability.
However, the explosion of Large Language Models (LLMs) in late 2024 presented a new challenge. The insatiable demand for more storage, compute, and especially GPUs outpaced even Meta’s impressive on-prem capabilities. Building data centers from scratch takes time, and in the fast-paced world of AI, every day counts. This realization sparked a strategic pivot: Meta decided to embrace public cloud to acquire the much-needed GPU capacity.
The Race to Cloud: A 3-Month Sprint to Capacity ⏱️💨
The journey to integrate public cloud wasn’t a leisurely stroll; it was a high-stakes race. Meta made the crucial decision to acquire cloud capacity in mid-December, and remarkably, handed off that capacity to their partners by mid-February. This entire onboarding and transfer process took less than three months!
Why Did Cloud Finally Work? 💡
The key enabler for this rapid cloud adoption was the evolution of GPU compute. Nvidia’s DJX Superpower reference platform transformed GPU compute into a commodity, with vendors building to a common hardware and topology standard. Since Meta’s own GPU infrastructure was already similar, this standardization made the leap to the cloud feasible.
Navigating the Clouds: Security and Reliability Hurdles 🛡️🔒
While the promise of readily available GPU capacity was enticing, the transition to public cloud wasn’t without its significant challenges. Meta’s deeply ingrained security and reliability standards, honed over years of owning their infrastructure, needed to be translated to the cloud environment.
Fortifying the Cloud: Meta’s Security Blueprint 🏗️
Meta’s on-prem security is a multi-layered masterpiece:
- Physical Security: Armed guards protect their data centers.
- Network Security: Strict access controls (ACLs) limit physical network access.
- Transport Security: End-to-end cryptographic authenticity and authentication secure data in transit.
- Root of Trust: A hardware-tied cryptographic root of trust underpins everything.
- Hardened OS: Meta’s own operating system, Metal OS, is hardened against zero-day threats.
The cloud, however, lacked these foundational elements.
The Off-Net Services Stack: A Trusted Framework in Untrusted Environments 🌐
Meta’s secret weapon in this security challenge was their off-net services stack. This ingenious system allows Meta to run its technologies on untrusted infrastructure, even in hostile environments where they don’t control the physical data center.
Here’s how it works:
- Immutable OS: Starts with a live CD-like model using a mutable operating system.
- Hardware Attestation: Verifies that the hardware hasn’t been tampered with.
- Local Installation: Meta’s components and software are installed locally.
- Meta OS & Daemons: The Meta operating system and necessary daemons are deployed.
- Off-Net Services Proxy: Instead of direct connections to Meta’s data centers, all traffic goes through a single proxy layer. This layer allows for auditing connections and enforcing authentication.
By adapting this stack and collaborating with the off-net team, Meta successfully enabled its first cloud capacity in a mere 60 days!
The Reliability Riddle: Bridging the Gap in Fleet Health 🩺
Once security was addressed, reliability emerged as the next major concern. Meta’s on-prem data centers utilize a sophisticated fleet health system. Millions of signals generated by their massive fleet are analyzed to detect patterns, identify failing systems, and proactively evacuate workloads.
Cloud vendors, due to their arms-length relationship with customers, lack this granular visibility into individual workloads. They operate at a higher system level. To bridge this gap, Meta deployed its own fleet health and network PE teams to implement similar tooling in the cloud.
Challenges Faced:
- Limited Visibility: Lack of access to crucial metrics like CPU temperature or backend network statistics due to shared resources.
- Cloud-Specific Workarounds: The need to develop creative solutions to compensate for missing data.
Despite these hurdles, Meta’s teams successfully developed workarounds, enabling them to detect system failures and ensure their critical ML training jobs remained uninterrupted.
Unifying the Cloud: From Bespoke to Universal Workloads 🎯
As Meta’s cloud adoption matured, the goal shifted from supporting specific workloads to enabling any workload in the cloud. This presented a new challenge: onboarding new ML workloads often required deep dives into team-specific requirements and dependencies on Meta’s internal services like caches, data stores, and security/privacy capabilities.
The traditional approach of bespoke configurations for each workload was proving inefficient and hindering cloud adoption. Engineers were accustomed to a common tooling layer, TWW shared (Twine Shared), a unified capacity pool for all Meta applications, and expected similar ubiquity in the cloud.
The Two Pivotal Decisions:
- Migrate to TWW Shared: The decision was made to migrate from specialized stacks to the unified TWW shared environment. The vision was to make cloud capacity identical to TWW shared on-prem. This would allow any Meta engineer familiar with TWW shared to deploy workloads in the cloud with zero additional work.
- Prove Cloud Viability with Web: To demonstrate this capability and reduce the friction caused by missing dependencies, Meta chose to migrate web, one of its oldest and most complex workloads, to the cloud.
Why Web? A Strategic Choice for Maximum Impact 🌐
Choosing web wasn’t about picking the easiest workload; it was about proving Meta’s ability to handle complexity:
- Foundational Role: Web is central to most of Meta’s product and business logic. Collocating it in the cloud reduces latency and cross-region traffic for all dependent services.
- Sprawling Dependency Graph: Web interacts with dozens of services, caches, and privacy systems. If web’s dependencies can work from the cloud, it validates the solution for virtually any other workload.
- Credibility Builder: A successful migration of web would significantly erode trust issues caused by past ML workload failures due to missing dependencies.
The Transparent TLS Revolution: Unlocking Seamless Connectivity 🔓
A major roadblock in migrating web was the off-net security stack’s reliance on individual firewall rules for each dependency. Web’s hundreds of dependencies, many unknown or legacy, made this approach unscalable.
Enter Transparent TLS (TTLS), Meta’s internal service that transparently encrypts and authenticates traffic.
How TTLS Works:
- eBPF Hooks: Leverages eBPF hooks to intercept outgoing connections on cloud hosts at the transport layer.
- Application Agnostic: Requires no application changes or manual service reconfiguration.
- TLS Tunneling: Encrypts traffic and wraps it in a TLS tunnel.
- Host Attestation: Proves the connection originates from a Meta-provisioned host with proper credentials.
- Transparent Operation: On the on-prem side, a reverse proxy unwraps the tunnel, forwarding plain text to the destination service. Services and developers remain unaware of the cloud deployment.
With TTLS in place, Meta unlocked seamless connectivity for all its services, making cloud capacity effectively indistinguishable from on-prem for developers. This led to about 1% of Meta’s employee traffic being served from cloud capacity.
The Next Frontier: Mastering Cloud Maintenance and Hardware Integration ⚙️🔧
As AI workloads continued to demand more and more capacity, Meta faced new challenges, particularly with the unique nature of GPUs and the integration of new hardware.
Maintenance Trains Meet Cloud Chaos: The Rise of PCM 🚂💥
Unlike CPUs, GPUs don’t behave as independent computers. A single GPU failure can bring down an entire job or cluster, making maintenance incredibly complex. Meta’s on-prem maintenance trains automate server decommissioning, OS reprovisioning, and physical maintenance every 45 days, seamlessly integrated with Twine.
In the cloud, this process devolved into manual coordination with vendors through Slack, Jira, and ticketing systems. This was unsustainable, especially with multiple cloud providers.
The Solution: Public Cloud Manager (PCM) 🎛️
To bring order to this chaos, Meta developed the Public Cloud Manager (PCM). PCM acts as a layer of abstraction over public cloud APIs and workflows, including maintenance.
- Automated Maintenance: PCM integrates cloud vendor maintenance into Meta’s internal maintenance trains, eliminating manual coordination.
- Fleet Health Integration: When fleet health detects a failure, PCM communicates this to the cloud vendor for instance repair.
- Abstraction Layer: PCM abstracts away the complexities of different cloud provider APIs, allowing a single engineer to manage a given cloud.
- Unified Fleet View: PCM presents cloud capacity as just another type of capacity, feeding into Meta’s existing infrastructure and providing operators and users with a single, unified fleet view.
Code Design for the Cloud: Shaping Future Hardware 💎
Meta’s commitment to “code design” extends to its cloud strategy. They’ve begun engaging with cloud vendors during the ideation and early design phases of new hardware. This proactive approach, utilizing Meta’s New Product Introduction (NPI) checklist, ensures that new cloud SKUs come with the necessary debugability, operability, and introspection capabilities from day one.
This level of collaboration unlocks a rare form of code design in the industry, accelerating Meta’s ability to onboard and ensure the reliability of new hardware in the cloud.
The Cloud is Here to Stay: A Future of Hyper-Liquid Capacity 🌊
Today, the public cloud is a critical and growing part of Meta’s infrastructure. The company has achieved record onboarding times:
- New Cloud Vendor Onboarding: Less than a month 🗓️
- New Cloud Region Onboarding: Approximately one week 🗓️
- New Cloud SKU Onboarding (with code design): About two weeks 🗓️
Meta’s journey to the cloud is a testament to their adaptability and engineering prowess. By embracing cloud capacity, integrating it seamlessly with their existing robust systems, and even influencing the future of hardware design, Meta is unlocking a future of hyper-liquid capacity, where any resource can be tapped, anywhere, anytime. This transformative approach is not just changing Meta’s infrastructure; it’s poised to change the industry once again. ✨