https://development.tldrecap.tech/presenters/phillip-liu/ Recent content in Phillip Liu on TLDRecap ⏮️ https://development.tldrecap.tech/images/tldrecap_logo.jpg https://development.tldrecap.tech/images/tldrecap_logo.jpg Hugo en Fri, 26 Jun 2026 16:18:15 -0700 https://development.tldrecap.tech/posts/2026/scale-systems-reliability/meta-tackles-nccl-watchdog-timeout-ai-reliability/ Fri, 26 Jun 2026 16:18:15 -0700 https://development.tldrecap.tech/posts/2026/scale-systems-reliability/meta-tackles-nccl-watchdog-timeout-ai-reliability/ <p><strong>Presenters</strong></p> <ul> <li> <a href="https://development.tldrecap.tech/presenters/phillip-liu">Phillip Liu</a> </li></ul> <p><strong>Source</strong></p> <ul> <li> <a href="https://development.tldrecap.tech/sources/scale-systems-and-reliability-2026">Scale Systems and Reliability 2026</a> </li></ul> <h1 id="taming-the-nccl-watchdog-timeout-metas-journey-to-ai-reliability-">Taming the NCCL Watchdog Timeout: Meta&rsquo;s Journey to AI Reliability 🚀</h1> <p>Ever hit a wall with your AI training jobs? Those dreaded &ldquo;NCCL watchdog timeout&rdquo; errors can feel like a black box, leaving you scratching your head and wasting precious compute time. Well, buckle up, because we&rsquo;re diving deep into how Meta tackles this beast, making complex debugging accessible and even exciting! ✨</p>