Presenters

Source

Rethinking Production Debugging at Hyperscale: Safer, Faster, and Smarter with Cosmic 🚀

In the world of massive cloud infrastructure, debugging live systems presents a colossal challenge. Imagine managing tens of millions of pods across thousands of clusters, spread across the globe, running diverse workloads. That’s the reality for Microsoft’s Cosmic platform, and it demanded a complete overhaul of how production debugging is handled. Shridivya Sharma and Luke Kelly from Microsoft Substrate recently shared their journey in transforming this critical process, moving from high-privilege access to an AI-driven, secure, and scalable solution.

The Foundation: Substrate and the Birth of Cosmic 💡

Before diving into the debugging revolution, it’s essential to understand the bedrock. Substrate is the foundational platform layer powering M365 services like Exchange, OneDrive, and SharePoint. It provides the shared infrastructure, deployment models, and operational controls necessary for global-scale reliability.

In the past, individual services managed their own infrastructure. While this offered flexibility, it led to fragmented environments with inconsistent security and compliance. Substrate aimed to standardize these aspects. However, capacity management remained a significant hurdle, largely tied to individual machines, making dynamic adjustments difficult.

This is where Cosmic emerged. Cosmic, short for Container on Substrate Managed Intelligent Clusters, extends Substrate by providing a container-based, multi-tenant platform built on Azure Kubernetes Services (AKS). It centralizes deployment, scaling, routing, and compliance, managed by the Cosmic team. This standardization significantly improved isolation, enforced compliance by design, and enabled much more efficient infrastructure utilization.

Today, Cosmic is a behemoth, running across 12,000 clusters, hosting services like Microsoft Teams and other M365 applications. It manages tens of millions of pods and hundreds of thousands of nodes, backed by nearly 6 million cores of compute, and spans all Azure regions, including public and sovereign clouds. It uniquely supports both Windows and Linux workloads within a single, unified environment, making it the second-largest customer of AKS at Microsoft.

The Debugging Dilemma: From Standing Access to Fragmented Tools 😟

Historically, production debugging relied on high-privilege access. Engineers could directly access production clusters to troubleshoot issues for partners. However, as Cosmic scaled, this model became impractical. It created standing accesses that were difficult to audit and posed significant security risks. Furthermore, it fostered dependency, with partners relying on the Cosmic team for even basic troubleshooting.

To address this, various tools were developed. Some worked for Linux, others for Windows. While they tackled parts of the problem, this led to a fragmented tooling landscape and inconsistent operational experiences.

The Cluster Debugging Tool (CDT): Just-in-Time, Secure, and OS-Agnostic 🛠️

Through iterative development, the Cluster Debugging Tool (CDT) emerged as a core, unified solution. CDT operates on a just-in-time (JIT) model. Access is granted only for a specific debugging task and is automatically revoked upon completion. Crucially, the task executes in an isolated environment, and the tool is completely OS-agnostic.

Luke Kelly elaborated on how CDT works in practice:

  • Engineer Initiates: An engineer facing an incident or investigating a failing pod initiates a single CLI command, specifying the target (pod or node) and the desired debug command. The CLI abstracts away the underlying complexity, making it feel like a local execution.
  • Authorization Service 🛡️: The first backend hop is to an authorization service. Every debug command undergoes individual authorization, checking the user’s identity, eligibilities, and command permissibility. Commands are managed, with varying approval requirements – read-only commands might be self-approved, while riskier ones require explicit sign-off from a team lead or incident manager. Authorization is a critical path; no authorization, no execution.
  • Short-Lived SSH Certificates 🔑: Upon approval, the system issues a short-lived SSH certificate. This certificate embeds the authorized command directly into SSH’s native “force command” options, along with metadata like the target, user, and namespace. These certificates are valid for minutes, not hours, and are scoped to a specific execution target and user. This eliminates the need for downstream components to call back to the auth service, as they can validate the certificate signature and embedded constraints.
  • Encrypted SSH Connection and JIT Server 🔒: With the certificate, the CLI establishes an encrypted SSH connection to the target. This involves creating a Just-In-Time SSH server tailored to specific constraints. The connection is secured at multiple layers, including network-level access restricted to corporate networks.
  • Ephemeral Environments and Command Execution 🏃: Inside the cluster, a custom controller watches for debug requests and schedules ephemeral environments. These environments accept the certificate, validate its signature, extract constraints, and execute the command. User input beyond the certificate is ignored. When the command finishes, the SSH connection terminates. For pod targets, the container interacts directly with the container runtime on the node, enabling debugging of distroless containers lacking their own shell or tooling. Standard output, standard error, and exit codes are preserved.
  • Comprehensive Auditing 📝: Every component in the pipeline emits audit trails. This includes who ran the command, against what target, for how long, and the output. Auditing is not an afterthought; it’s baked into every component from start to finish, ensuring that “if it happens, we know about it.”

The AI Orchestration Layer: Automating Investigations with Agents 🤖

While CDT provides powerful interactive debugging, it still relies heavily on the engineer’s domain knowledge – knowing which command to run, when, and how to interpret the output. To simplify this, an AI-driven orchestration layer has been introduced.

This layer utilizes a central AI orchestrator within Cosmic’s unified operational portal (COP). COP provides partners with a consolidated view of their service health. The AI orchestrator understands user requests and coordinates with specialized agents to resolve them.

  • Agent-to-Agent Architecture 🤝: The orchestrator delegates tasks to specialized agents, including a diagnostic agent that leverages CDT, a knowledge base of historical incidents, and platform actions.
  • Hypothesis, Validation, and Refinement ✨: Each agent forms a hypothesis, uses tools like CDT to validate it, and refines its approach based on findings. For instance, an agent can analyze logs, check system health, and retrieve relevant guidance.
  • Parallel Processing and Unified Response 🌐: Agents operate in parallel, each focusing on a specific capability. They retrieve information and consolidate it into a single, comprehensive response for the user. This shifts the model from manual, user-driven workflows to AI-orchestrated ones, where the system composes and executes the entire workflow.

Security, Evaluation, and Guardrails for AI Agents 🔒

The introduction of AI agents necessitates robust security measures, rigorous evaluation, and strict guardrails:

  • Security by Design 🛡️:
    • Tool Layer: Exposed Microsoft Cloud Platform (MCP) tools are read-only, supporting only get, list, and describe operations.
    • Auth Layer: On-behalf-of tokens exchange is used, meaning agents inherit the user’s role-based access control (RBAC) and are never elevated beyond user permissions.
    • Prompt Layer: System instructions explicitly enforce read-only behavior.
    • Runtime Layer: Agent containers run in sandboxed environments with limited node access, preventing unauthorized actions even if requested.
  • Evaluation and Performance Metrics 📊:
    • The system was evaluated across 19 scenario types, covering diagnostics (pending pod state, performance, networking, etc.) and remediation (currently limited to command generation).
    • Root cause accuracy is around 85-90%.
    • Factual grounding is around 70-80%.
    • The average agent response time is approximately 37 seconds.
  • Guardrails for Trustworthy AI ⚖️:
    • Every agent claim must cite a specific tool and query to prevent unverified assertions.
    • Each investigation is limited to 10 tool cycles to prevent runaway loops.
    • Every investigation must consult the knowledge base at least once, ensuring agents are grounded.
    • Output is always in a standardized RCA format with a summary, root cause, and finding tables with source attributions.
    • A hallucination guard ensures that if an agent finds no evidence, it states so explicitly rather than fabricating information.

The Future of CDT and Agent-Driven Debugging 🔮

Luke highlighted the future vision for CDT: integrating the robust audit trail and tooling into the “agent age.” The goal is to pre-package all tools, wikis, and documentation into a shell, with agents inside that understand how to use them. This aims to create a one-stop shop for on-call engineers, providing a compliant and audited environment where agents assist in all necessary debugging tasks.

In conclusion, Microsoft’s journey with Cosmic demonstrates a profound shift in production debugging. By moving away from high-privilege standing access and embracing a JIT model with CDT, and then layering on AI-driven orchestration with specialized agents, they have successfully made production debugging safer, faster, and smarter at hyperscale, all while maintaining stringent security and robust guardrails.

Appendix