Presenters
Source
The Art of the Unsung Hero: Navigating the Complex World of Systems Engineering 🛠️✨
In the fast-paced world of technology, we often celebrate the flashy new apps and groundbreaking software. But what about the bedrock upon which all of this is built? The complex, often invisible, systems that keep everything running smoothly? This is the domain of the systems engineer, a role that Matthew Liss, with over 30 years of experience at giants like American Express, JP Morgan Chase, and Goldman Sachs, deeply understands. In a recent conversation on the Architects podcast, Matthew shared his unique perspective on what it means to be a systems engineer, the challenges they face, and the profound satisfaction derived from building resilient and scalable platforms.
From Tinkering to Tremendous Impact: A Journey of Building 👨💻
Matthew’s path into systems engineering wasn’t a direct one, but rather a natural evolution from a childhood fascination with how things work. Growing up in an era before ubiquitous computers, a pivotal moment came at age eight or nine when he played chess against a mainframe at the University of Oslo. This sparked a lifelong passion for making ideas manifest in the real world.
“I was always a tinkerer,” Matthew recalls. “Putting together computers, writing code, even soldering electronics… I fell in love with this whole concept of being able to think of something and then go make it happen.” This hands-on approach, reminiscent of his carpenter father, laid the foundation for a career dedicated to building robust systems.
The Crucial Difference: Tinkering vs. Solid Engineering 🧱
While the urge to tinker is a powerful motivator, Matthew highlights a critical distinction between casual experimentation and solid engineering practice. Unlike a wobbly table leg that might be forgivable, a single weak point in a software platform can lead to significant trouble.
“System engineering, or what I think is in an apprenticeship, is no different than any other craft,” Matthew explains. “You get good at a craft by learning from others, from making mistakes, and gradually understanding what great looks like. But it takes experience. It takes apprenticing. It takes being willing to take risk and learn from the mistakes and work through it.”
He shares a poignant early career anecdote from his time soldering cables at Schlumberger. After painstakingly building his first cable, his boss, rather than testing it, cut it in half after a visual inspection, declaring the soldering “shocking” and instructing him to “Do it again.” This harsh but effective lesson underscored the importance of attention to detail and doing things properly. This iterative process of making small mistakes, learning from them, and apprenticing with others is how Matthew built his expertise.
The AI Conundrum: Where Do New Engineers Learn? 🤖❓
The rise of AI brings a new layer of complexity to this discussion. As AI takes on more of the “easy coding jobs,” a crucial question emerges: where will the next generation of engineers gain their foundational apprenticeship?
“I think that’s the most profound, I mean, at least to me right now, it’s the most profound,” Matthew muses. “I was lucky I could apprentice, I could learn to do stupid little things to begin with that gradually became more and more complex things over time.” He worries that if AI handles the routine tasks, engineers might miss out on the essential learning experiences that build deep understanding.
This raises concerns about a potential “pipeline problem,” where the path to becoming a senior engineer is disrupted. While acknowledging that AI can be a powerful tool, Matthew emphasizes that human oversight and understanding remain critical. The ability to read code, even if generated by AI, and to troubleshoot complex systems is paramount.
The Three S’s: Stability, Security, and Scalability 🚀🔒📈
At the core of Matthew’s work, particularly in financial services, are the non-negotiable principles of stability, security, and scalability – the “three S’s.” These form the bedrock of platforms that developers build upon.
“You could compromise on it pre-production, but once you have and you’re trying to support a trading app, a banking app, an ATM app, and so on, the expectation is that these three S’s are always true,” he states. This often leads to a conservative approach, carefully balancing risk and change to maintain the integrity of critical systems.
He likens a complex system to an organism: “If one part no different than a human body, you know, if your lungs don’t work well, even if everything else is perfectly fine, your body doesn’t work.” Understanding one’s place within this larger ecosystem and managing upstream and downstream dependencies is key.
Risk Management: The Art of the Trade-off ⚖️
System engineering, Matthew argues, is fundamentally about risk management. It’s about understanding how much risk you’re willing to take when evolving a system and finding that delicate balance. He touches on the concept of error budgets and measuring customer journeys to inform these decisions.
“If I’m failing too little, I’m clearly not taking enough risk, and I fail too often, taking too much risk,” he explains. The goal is to optimize for customer outcomes, understanding that a failure that impacts customers is far more critical than one that goes unnoticed.
The Perennial Question: The Feedback Loop 🔄
A significant challenge in system engineering is establishing a perfect feedback loop from customer experience back into architecture. Matthew acknowledges that perfection is elusive but highlights the power of focusing on customer outcomes.
“If you think about ultimately, why are we building software? We’re building software to support certain business outcomes which support our customers,” he says. By constantly asking “how does this impact the customer?” engineers can better direct their efforts and prioritize what truly matters.
Navigating the Edge of Chaos: When Systems Teeter on the Brink 🎢
In complex environments, systems often operate “on the edge of chaos.” Matthew uses the example of a credit card transaction, a seemingly simple customer journey that involves a vast and intricate ecosystem of systems and parties.
“There’s a huge amount of complexity that the customer never sees and never should see,” he notes. To manage this, rigorous testing, chaos testing, and scenario planning become essential. He emphasizes that anticipating scale is often where complex systems break, as increased volume pushes them closer to resource contention.
The Unsung Hero: The Frustration and Fulfillment of Invisibility 🎭
One of the most challenging aspects of systems engineering, Matthew reveals, is that the job is often about invisibility. When systems work perfectly, no one notices the engineers who made it happen. This can lead to a lack of appreciation, especially when funding is scarce or complexity is underestimated.
“Your job is to be invisible. And if you do a really good job, no one ever knows you exist,” he states. This can be frustrating when clients trivialize the complexity and expect perfection without understanding the effort involved.
However, the flip side is the immense satisfaction of seeing complex systems successfully deliver value. The “magic moments” when an idea becomes a tangible, functioning reality are what keep him motivated.
The Evolving Landscape: Agentic AI and the Future of Platforms 🤖🌐
The advent of agentic AI presents a new frontier for systems engineering and platform development. Matthew views agentic AI, at this stage, through a similar lens as human developers – the fundamental accountability for system functionality remains.
“I’m still accountable to make sure this stuff works, and if it’s built by humans or built by agents, it still needs to function,” he asserts. The key difference lies in the speed at which these systems operate and the need to feed them vast amounts of data.
The challenge then becomes scaling observability platforms to handle the speed and volume required by AI. “We have to scale up all that,” he explains. “The same way for agentic reading is because now reading this stuff 100, a thousand, 10,000 times faster, I now need to build the underlying platforms for that that I have to scale way more than I ever anticipated.”
The Heart of the Matter: Culture, Collaboration, and Continuous Evolution ❤️🤝
Ultimately, Matthew stresses that successful platform engineering hinges on culture. Great culture builds great teams, and great teams build great products. Empowering teams to make autonomous decisions within defined guardrails, fostering a culture where mistakes are learning opportunities, and ensuring seamless collaboration are paramount.
“If you build platforms, they need to hang together,” he concludes. “It’s like a meal. You go to a restaurant to eat a meal, not to eat a carrot and a potato and a steak.” The cohesive, coherent experience is what consumers – whether human developers or AI agents – expect.
The world of systems engineering is one of constant evolution, a never-ending challenge of balancing speed, security, and scalability. It’s a testament to the unsung heroes who dedicate themselves to building the invisible foundations of our digital world.