Presenters

Source

🚀 Bringing Instagram to Life in VR: A Deep Dive into Spatial Media at Scale 🌐

Instagram’s journey into the metaverse is gaining serious traction! Jaime Del Palasio, a software engineer on the Instagram VR team, recently shared a fascinating look at how they’re building a native Instagram app for Quest headsets. It’s not just about slapping a 3D interface onto the existing app; it’s a complete rethinking of the Instagram experience to be truly immersive. Let’s dive into the technical challenges and innovative solutions behind bringing spatial media to millions.

🗓️ Timeline & Early Launch 🎯

The project kicked off in early 2024, with the initial version launching in October 2024. This early release focused on core functionality: the home feed, Reels, and profiles. This allowed for early user feedback and iterative development.

✨ The Vision: 3D Content for Everyone 💡

The goal isn’s just 3D; it’s stereoscopic 3D. Instagram aims to convert all content into a VR-only experience. This means presenting slightly different versions of images and videos to each eye, creating the illusion of depth – just like in 3D movies. It’s a bold move, but one that promises a uniquely immersive experience.

🛠️ Technical Hurdles & Solutions: The Race Against Latency 🦾

Building a VR experience at scale isn’t easy. Here are some of the biggest challenges Instagram faced, and how they tackled them:

  1. When to Convert? Converting every post as it’s created would be incredibly resource-intensive, and most content goes unviewed. Converting only when a user views it guarantees no wasted resources, but demands near-instantaneous processing. Instagram chose consumption-time conversion – converting content only when it’s about to be viewed.

  2. Where to Convert? Running the conversion on the headset itself (client-side) would eliminate network overhead, but the Quest’s hardware has limitations. Server-side conversion allows for more powerful processing and caching, benefiting multiple users. Instagram opted for server-side conversion for better quality and scalability.

  3. Speeding Things Up: Initially, converting a 20-second Reel took a staggering 200 seconds! This was unacceptable. To overcome this, Instagram implemented a multi-pronged approach:

    • Parallelization: Splitting video frames across multiple GPUs to process them simultaneously.
    • Code Optimization: Refining the AI model and tuning encoder parameters. A key optimization was quantizing the AI model to a smaller floating-point size.
    • Proactive Conversion: Starting the conversion process 2 seconds before content delivery to users, creating a buffer for processing.

    The result? A dramatic reduction in conversion time – from 200 seconds to under 10 seconds! This real-time conversion is crucial for a smooth VR experience.

🤖 AI & Model Development: Introducing MetaDepth 🧠

Existing AI models simply weren’t cutting it for real-time depth estimation. So, Instagram developed a custom AI model: MetaDepth. This model analyzes 2D images to estimate the depth of each pixel, a critical step in creating the stereoscopic 3D effect.

⚖️ Quality Considerations & Trade-offs: Comfort is Key 👨‍💻

It’s not just about making things look cool; it’s about making them comfortable. Instagram prioritized user comfort above all else. They discovered that negative parallax (objects appearing in front of the screen) was fatiguing. So, they opted for positive parallax to create a more natural and comfortable viewing experience.

Another challenge arose with text within posts. AI models often misinterpreted text as depth, leading to distortion. To address this, Instagram implemented a dedicated model to specifically detect and place text in front of other elements.

💾 Technology Stack: Powering the Experience 📡

Here’s a quick rundown of the core technologies Instagram is using:

  • Quest Headsets: The target VR platform.
  • MetaDepth: The custom AI/ML model for monocular depth estimation.
  • GPU Acceleration: Leveraging multiple GPUs for parallel processing.
  • Server-side Infrastructure: Crucial for efficient conversion and caching.

🔭 Future Directions: What’s Next? 🌐

Instagram isn’t stopping here! Future plans include:

  • Deeper Immersion: Allowing users to virtually “walk into” Instagram posts, truly stepping inside the content.
  • Continued Optimization: Further refining performance to handle the increasing demands of spatial media.

The journey of bringing Instagram to life in VR is a testament to the power of innovation and a commitment to pushing the boundaries of what’s possible. It’s an exciting glimpse into the future of social media, and we can’t wait to see what Instagram creates next!

Appendix