Presenters
Source
Breaking the Chains: Mastering Progressive Delivery for Stateful Applications 🚀
In the fast-paced world of software engineering, we often treat stateless applications like royalty. They scale easily, they roll back without a fuss, and they love Canary releases. But the moment a database enters the room, everything gets complicated.
Anastasiia Gubska, a Lead Software Engineer at Chase UK and the first deaf CNCF ambassador, recently took the stage to solve one of the most persistent headaches in DevOps: how to perform progressive delivery when your application is tied to a database. Using British Sign Language (BSL) and a live demo, she mapped out a journey to decouple database schemas from application deployments.
🌐 The Progressive Delivery Landscape
Before diving into the deep end, we must understand the tools at our disposal. Progressive delivery means releasing software changes incrementally rather than all at once.
- Blue/Green Deployments: You maintain two identical environments. You switch 100% of traffic from Blue to Green. It is an all-or-nothing strategy. If things break, you switch back instantly.
- Canary Releases: These are much smarter. You route a small percentage—perhaps 10%—of traffic to a new version, monitor the metrics, and then gradually promote it to 90% and finally 100%.
While Canary works perfectly for stateless apps, it creates a massive tradeoff for stateful ones. During a rollout, version 1 and version 2 of an application run simultaneously, reading and writing to the same database schema. If version 2 deletes a column that version 1 still needs, your system crashes. 💥
⚠️ The “Deploy Together” Trap
A common instinct is to deploy the application and the database schema changes at the exact same time. Anastasiia warns that this is a recipe for data loss.
Imagine you want to rename a full name column into first name and last name columns:
- The Promotion Problem: If version 1 writes to full name while version 2 writes to the new columns, promoting to version 2 means you lose all the recent data written to the full name column.
- The Rollback Nightmare: If you roll back both the app and the DB, any data written by version 2 to the new columns vanishes.
To survive this, you must decouple the database schema from the application deployment. 🔓
🛠️ The Database Refactoring Matrix
The solution lies in database refactoring. This involves designing changes to be both backwards and forwards compatible. This allows both application versions to coexist safely.
There are four primary patterns for this transition:
- Expand ➕
- Sync 🔄
- Migrate 🚚
- Delete 🗑️
Anastasiia focuses on the Rename pattern, which is the most complex. The secret is to avoid “renaming” entirely. Instead, you add the new columns and eventually drop the old one.
🏗️ The Three-Phase Rollout Strategy
To achieve a seamless transition using tools like Argo Rollouts, you must follow a multi-stage approach.
Phase 1: The Expansion 🐣
You expand the schema by adding the new columns (first name, last name). These must be nullable. Application version 1 continues to read and write only to the full name column, completely ignoring the new additions.
Phase 2: Dual Writing ✍️
You deploy application version 2. This version is smarter: it reads from the new columns but writes to all three. This ensures that if you need to roll back to version 1, the data it needs (full name) is still being updated.
Phase 3: The Cleanup 🧹
Once version 2 is stable at 100% traffic, you deploy version 3, which only interacts with the new columns. After confirming stability, you finally drop the old full name column. At this point, rollback is no longer possible, but your code and database are now clean and modern.
💻 Live Demo: Argo Rollouts in Action
Anastasiia demonstrated this using two CLI terminals. On the left, the database; on the right, the Argo Rollouts CLI.
- The Setup: The database started with one column (full name) featuring famous British names.
- The Migration: A script triggered the schema change, adding first name and last name. Application version 1 continued running without a single error.
- The Canary: The terminal showed yellow pods (Canary) alongside green pods (Stable). Version 2 began dual-writing. A new entry, John Vin, appeared, successfully populated across the new columns.
- The Result: The full name column was dropped, leaving a clean, refactored database with zero downtime. 🎯
🛑 Hard Rules for Stateful Success
To implement this successfully, follow these strict guidelines:
- Never deploy application and database schema changes together.
- Never allow the application to make schema changes (e.g.,
ALTER TABLE). The application should only read and write data. - The database team (or specialized migration scripts) must handle the schema evolution independently.
❓ Question & Answer
Q: In Phase 1, how do the new columns get populated with existing data? A: The database team handles this through background migration scripts or triggers. The application should stay focused on data logic, not schema migration logic.
Q: What happens if there is a desync between version 1 and version 2? A: Because the application and schema are decoupled, the app only knows how to run on the schema it expects. Any synchronization concerns are handled at the database level, ensuring the app remains stable regardless of background changes.
Q: How does a change in version 1 transition to the new columns in Phase 2? A: During the migration phase, you must use a background sync or database trigger to ensure that data modified in the old format is reflected in the new columns.
🤝 Building an Inclusive Community
Beyond the technical deep dive, Anastasiia highlighted the importance of accessibility in the Cloud Native world. As part of the Deaf and Hard of Hearing Working Group, she encourages engineers to become allies.
- Learn the signs: There are specific BSL signs for industry terms like Kubernetes, Argo, and Linkerd. 🦾
- Support Mentorship: Many deaf and hard-of-hearing engineers need allies to help them navigate the open-source ecosystem.
- Accessibility First: Making your company accessible—through sign language interpreters and inclusive onboarding—is not just a nice-to-have; it is essential for a thriving tech community. 🌐
By mastering progressive delivery and fostering inclusion, we don’t just build better software; we build a better industry. ✨
For those looking to experiment, you can access the demo code and accessibility resources through the CNCF community channels. 💾📡