Presenters
Source
When Meta Almost Had a Really Bad Day: A Deep Dive into a Near Catastrophe 🤯
Hey tech enthusiasts! Gather ‘round, because we’ve got a story that will make your palms sweat and your admiration for robust infrastructure soar. Phil Lopreiato and Rahul Iyengar from Meta’s Web Foundation team are here to pull back the curtain on a nail-biting incident that could have spiraled into a major outage, reminding us that even the giants of the tech world face hair-raising challenges.
The Alert That Froze Everything 🚨
Imagine this: the workday is winding down, the team is mentally checking out for happy hour, and then… BAM! Alerts start screaming. Dashboards flash red. On-call phones buzz incessantly. A critical alert fires, screaming “Something is broken!” The team is pulled back in, a conference room is commandeered, and the investigation begins.
What they saw was chilling: a Gantt chart, a familiar sight for anyone who’s navigated Meta’s services, showing a critical main web job in a region turning from healthy green to alarming pink. It had just stopped. This immediately conjured memories of the infamous October 4th, 2021 outage. The fear was palpable: are we about to relive that nightmare?
Meet the Web Foundation Team 👨💻
Phil and Rahul are part of Meta’s Web Foundation team, the guardians of Meta’s main web tier and the primary responders for site-wide incidents. If the site falters, chances are they’re on the front lines. They’re about to take us on a journey through a specific incident that unfolded last spring, detailing the “how,” the “why,” and the “how we survived.”
The Design Philosophy: Expect the Unexpected 💡
During complex incidents, the core principle is to expect your tools and assumptions to be broken. Systems can have hidden dependencies, and normal access paths might be unusable. The 2021 outage even saw out-of-band access routes fail. The key is to embrace the unexpected and apply creative problem-solving.
Practice Makes Perfect: Storm Drills ⛈️
Meta’s Web Foundation team isn’t new to stress. Their recurring storm drills, where they safely take all traffic out of a region and then bring it back, build crucial confidence and competence. This well-trodden path minimizes risk and arms them with a wider array of tested operations for future incidents.
The Wilderness of Debugging 🌲
Sometimes, though, you find yourself off the beaten path. As James Mickens famously put it, debugging distributed systems is “Texas style.” You need stoic individuals, primitive tools, and the courage to venture into the unknown. While primary and backup plans might fail, the right people, tools, skills, and preparation can overcome almost anything.
A Toolkit for Disaster 🛠️
Meta has invested heavily in tools to combat such crises. The Bell Jar toolkit, born from the ashes of the 2021 outage, allows for low-level infrastructure operations and running services with minimal dependencies. Their disaster recovery program, dating back to Hurricane Sandy in 2012, ensures site resilience even in the absence of an entire region, reinforced by frequent drills.
Twine: The Orchestration Engine Under Fire ⚙️
Rahul dives into the technical heart of the matter, introducing Twine, Meta’s container orchestration framework. This complex system involves schedulers (managing up to a million tasks), allocators (deciding where containers run), resource brokers, and agents (managing on-host container lifecycles).
The Fan-Out Problem and the Agents Manager 📈
A significant challenge for Twine was the massive fan-out traffic between schedulers and agents. The explosion of AI workloads led to more machines, more tasks per host, and a desperate need for more scheduler shards. To tackle this, Meta introduced the agents manager service. This service provides schedulers with a consistent view of agents and transforms an M x N mesh into a more manageable M + N mesh.
Zelos and Delos: The Pillars of Consistency 🪞
Both the scheduler and the agents manager rely on Zelos for data persistence. Zelos implements the Zookeeper API on top of Delos, Meta’s in-house low-dependency table store. The critical point? Both systems demand strong consistency and will cease to function if Zelos falters.
A Cascading Failure in the Scheduler Tree 🌳
A crucial detail: Twine schedulers are self-hosted in a tree-like structure. Regional schedulers are managed by a root meta scheduler. However, Zelos itself was run as a regular job under a regional scheduler (shard zero). This seemingly minor detail would prove catastrophic.
The Bug That Broke the World 💥
The incident kicked off with a rollout of the agents manager service. Upon restarting, and without reading data from Zelos, it incorrectly reported that no containers were running. The scheduler, acting on this faulty information, initiated a full restart of all containers – including a cleanup phase that stopped and destroyed existing ones.
The bug hit shard zero of the region, which, in turn, halted the Zelos ensemble for the meta region scheduler. Because the meta region scheduler relies on Zelos for its strong consistency guarantees, it froze. And with the meta region scheduler frozen, the scheduler responsible for managing shard zero also became inaccessible, creating a deadly dependency cycle. Standard update paths were dead. To make matters worse, the agents manager’s Zelos ensemble also stopped, paralyzing its ability to restart containers.
The result? All containers managed by shard zero were stopped, and there was no way to revert the change through normal channels.
The Recovery: Dripping with Ingenuity and Preparedness 💧
Phil takes us back to the intense investigation. The initial relief came from realizing the issue was region-specific, allowing for containment.
Draining the Region: A Well-Oiled Machine 🌊
Thanks to their frequent drills, the team confidently executed a region drain. This operation, where traffic is safely rerouted, resolved user-facing impact within 15 to 20 minutes. This is a testament to the power of regular practice.
Untangling the Knot: Feature Flags Save the Day 🎉
With the site stabilized, the focus shifted to the root cause. Logs revealed the scheduler had stopped tasks. The team untangled the dependency cycle and correlated the issue to a release that enabled the new dependency behind a feature flag. This highlights the critical importance of gating changes and the benefit of staged rollouts, limiting the blast radius to a single, assumed fault domain.
The Bell Jar Toolkit to the Rescue 🧰
But the scheduler itself was broken, preventing a standard fix. This is where the Bell Jar toolkit came into play. The team needed to find the hosts that used to run the scheduler to potentially salvage state. This required using internal tools to sift through logs, as their usual low-dependency methods were compromised.
The next steps involved:
- Finding the hosts: Manually reconstructing where the tasks used to run.
- Ensuring the binary: Discovering the necessary binary was still present on the hosts – a stroke of luck!
- Starting a recovery container: Constructing a command to run the binary without the problematic flag.
While the Bell Jar toolkit had append-only flag support, it proved complex. Ultimately, they edited the job spec directly to set the flags correctly. This allowed the schedulers to restart, and subsequently, all downstream tasks.
Restoring Full Operation: A Three-Step Tango 💃
With the core infrastructure recovered, the final phase was to restore full site operation:
- Reverting the flag: Performing a normal flag revert via a TTY update on the meta meta scheduler.
- Starting “real” containers: A second set of containers, fully managed, started up.
- Handing over control: The recovery containers were stopped, allowing the newly started, properly configured containers to take over.
- Undraining the region: The well-trodden runbook was executed to bring traffic back into the region.
Lessons Learned and Future-Proofing 🚀
After the dust settled and a well-deserved deep breath was taken, the team reflected on the incident and the necessary follow-ups.
Architectural Fixes: Elevating Zelos ⬆️
Recognizing the relentless march of AI and increasing complexity, Meta made significant architectural changes. Zelos was promoted from a regular service to a critical part of the Twine ecosystem. Each Zelos ensemble is now scheduled as a peer to the scheduler it serves, a trade-off with operational implications but crucial for infrastructure resilience.
Improving Tooling: Less Luck, More Reliability 🎲➡️💡
The incident exposed gaps in tooling, particularly in identifying which hosts ran specific tasks. Historical logging was significantly improved, reducing dependence on luck.
Enhanced Drills: Practice Makes Resilient 🛡️
Drills were intensified:
- Bell Jar testing: More rigorous testing of the recovery tooling, with automatic runs.
- Human drills: Quarterly or monthly drills ensured engineers were intimately familiar with operating the recovery tools, not encountering them for the first time in an emergency.
The Inevitability of Incidents 🔄
The core message: incidents like this will happen again. Since this event, Meta has already had to use this recovery tooling two more times.
The AI Deluge and Human Context 🌊🚶♂️
The pace of change is accelerating, driven by AI. This means more complexity and a decreasing human understanding of these intricate systems. Sometimes, you still need to know how to operate with a “shovel and a lamp.”
What Went Right? ✨
Looking back, several factors contributed to their success:
- Good Engineering: Feature flags, staged rollouts to a single shard, and a defined process made identifying the culprit straightforward.
- Investments in DR and Tooling: The disaster recovery program and the developed tooling (region draining, recovery tools) proved invaluable.
- A Touch of Luck: The incident required recovery of only a handful of core jobs, and monitoring tools remained functional. As they say, luck is where preparation meets opportunity.
The question for all of us in the tech industry, especially with the AI revolution, is: are you dependent on luck, or do you have the tooling you need to make it work when things inevitably break? This story is a powerful reminder that resilience isn’t just about building; it’s about preparing, practicing, and having the courage to face the unknown.