Software Testing Unleashed - QA, DevEx & Quality Engineering

Testing AI models means testing a magic box: nobody has the source code, and the behavior is neither controlled nor understood. Michael Bolton calls that strongly magical, and with him I talk about what follows from it. His principles are simple: if something matters very much, don't depend on magic, and if you use it anyway, responsibility demands that you test it and maximize testability. Simple output checks don't stand a chance against a language model, so testing becomes experiencing, exploring and experimenting, like peeling an onion. And Penn and Teller's clear plastic cups show why we fall for these models in the first place: the insight we see in their text comes from us.

Testing is supposed to prevent failures in production, yet sometimes the test itself is what breaks things. Bart Knaack calls this the dark side of testing, and we talk about test data leaking out of poorly protected environments, test systems that give attackers a look at your code, and why testing in production needs a risk assessment of its own. Bart would rather ask questions than hand out checklists, because a ticked box stops people thinking. He also tells the story of a team with 10,000 end-to-end test cases that ended up needing only 50.

Quality hunting grew out of Rik Marselis's frustration with bug hunting, and I wanted to know how it works when the goal is information about quality. Rik walks me through a session: teams of about four people, three hours of exploratory testing and a goal agreed up front by asking stakeholders what would help them sleep well at night. He explains why the report structure comes first, why the prize should stay small and why the point just before several sprints go to production suits a hunt well.

What does it take to turn a risky one-time event into a conference that testers travel across borders for? With Okan Çakmak I talk about how the Istanbul Software Testing Conference grew out of something missing: after the pandemic, the big Turkish testing events stayed online, and people lost the chance to meet, network and hug each other in person. He tells me how the team wondered whether 50 or 100 people would come, then welcomed 200, and how the "light in their eyes" of those first participants convinced them to keep going.

What happens when AI ships poor quality faster, and nobody notices because the pipeline stays green? With Michaël Pilaeten I talk about why test automation without a clear strategy is a problem that AI only accelerates, not solves. We get into the question of what separates a tester from a test automation engineer, why those two roles need each other but are not interchangeable, and how chasing 100% coverage on millions of lines of legacy code can be a waste everyone agrees to quietly.

What separates a handful of agents from a real agentic system? With Nishan Portoyan I talk about exactly that line, and why most teams are still on the wrong side of it. We walk through a full testing pipeline where eight to twelve agents handle everything from requirement review to performance test conversion, and I keep coming back to the question of where the human has to step in and why the AI simply cannot replace that judgment. Nishan is direct about what these systems cannot do: they do not think, they gather information and hand it back, and anyone expecting otherwise will run into trouble fast.

What happens to quality when anyone can generate code with a prompt but nobody checks whether it actually works? With Olivier Denoo I talk about why that question puts expert testers in a stronger position than many managers currently assume. We get into the real risk of AI that always sounds right, the edge cases a lost suitcase in Brussels reveals, and why the hardest unsolved problem in testing is still the same one Olivier ran into three decades ago: describing precisely what you want. The skills I keep coming back to in this conversation are business understanding, communication, and the ability to spot what the AI confidently got wrong.

Why do we rush to fix problems before we actually understand them? With Nirmala Saneechur I talk about Lean management, what it means to deliver quality at the right time rather than just on time, and why spending a full day understanding a problem is not wasted time. We get into what waste really means in a software project, including the story of 15 features a client never used, and how working in silos creates the kind of back-and-forth that slows everything down.

Slow systems don't just annoy users, they cost companies real money, and most teams only find out when the phone starts ringing. With Rao Dhaligadoo I talk about what it actually takes to catch performance problems before customers do. We get into how a 70 percent CPU threshold can trigger an automated chain of monitoring, ticket creation, and root cause analysis, and why that matters more than it sounds. Rao also shares what it feels like to spend 45 hours in a war room at a bank, watching a CIO walk in at 1 a.m. while the whole team sits in silence, and how that experience shaped the way he thinks about proactive testing.

Forty percent of automation time lost to flakiness and maintenance: that is the reality Lilia Gargouri describes. I talked with Lilia about end-to-end test automation at scale, and what struck me most was how far back the real solutions reach: a single HTML attribute added in 2017 still pays dividends today, simply because the team decided to use it consistently and never change it. We get into the full picture, from how to accurately identify every UI element without XPath gymnastics, to why fifteen well-chosen test cases can outperform five hundred noisy ones, to naming conventions and layered library architecture that keep maintenance from spiraling. And for anyone sitting on an old, flaky test suite right now, Lilia has a concrete answer for where to start.

With Sebastian Malyska I talk about what sits behind the APIs testers often take for granted. We get into why REST is still on top but not the whole story, how GraphQL and webhooks fit into the picture, and what actually happens when an AI agent talks to a real system through MCP. There is a practical argument running through all of it: if you understand the protocol layer, you can make smarter decisions about where to test, how to structure automation, and how to avoid burning unnecessary tokens and budget. Sebastian also makes the case that front-end testers are already closer to this world than they think, and that the next step in the career is simply to look one layer down.

With Klaudia Dussa Zieger I talk about where language models actually deliver in testing and where they fall flat. We get into the concrete details: why a small function like generating keyword documentation works so well with a standard chat model, why reviews only started working once they switched to a reasoning model, and why one engineer working on embedded legacy code said no to AI help without hesitation. Klaudia also shares the translation use case I found most vivid, turning a cryptic technical defect message into something a domain tester can actually read.

With Jean-Francois Riverin I get into a question that bothered him for over 20 years: why are there still bugs in production even when you tested as well as you possibly could? His answer is that software stopped being a complicated problem, like building a bridge, a long time ago. It is now complex, more like medicine or meteorology, and that means the whole idea of finding the right answer through testing is the wrong goal. We talk about what it looks like to switch that mindset, what "safe to fail" means in practice, and how to make that argument to a manager who still expects predictable budgets and zero surprises on Friday afternoons.

Why do so many software teams end up with the exact architecture they never intended to build? With Vanessa Formicola I talk about the hidden forces that shape technical decisions, and why those forces have far less to do with technology than most engineers want to admit. We get into patterns she calls the kitchen sink, Cirque de Soleil coding, and the monolith constellation, each one a recognizable trap that grows from career incentives, delivery pressure, and organizational culture rather than bad technical judgment. I keep coming back to her concept of holistic engineering, which asks us to treat an organization as a living system and factor in product direction, people's actual lives, and structural dynamics when making architectural choices.

With Kari Kakkonen I talk about what happens when you explain software testing through a fantasy story for children, where dragons are defects and knights are testers. We get into how the analogy grew into a full world covering functional, performance, usability, and security defects, each mapped to a chapter with its own kind of dragon. Kari describes how he tested the book iteratively with children, teachers, testing professionals, and non-IT readers across four rounds, and how a twelve-year-old at a book fair said she wanted testing as a profession after flipping through it. We also touch on the donation campaign that brought the book into schools across Finland and beyond, and what a possible second volume might cover, from testing techniques to automation and AI. One detail I keep coming back to: not every analogy in the book is spelled out, and the black dragon is left without explanation on purpose.

What does it actually take to move from finding defects to preventing them? With Zsolt Hargitai I talk about TMMI, a framework for improving test processes, and why that shift in mindset marks the difference between a team that tests and a team that has genuinely matured. We get into what the maturity levels mean in practice, how a free self-assessment tool on the TMMI website gives you a first impression of where your organization stands, and why the framework holds up across agile, V-model, and DevOps environments alike. I was also curious about the human side: Zsolt mentions that companies implementing TMMI tend to see lower fluctuation rates because people feel more comfortable and satisfied, which is not the angle most managers expect when they start talking about process improvement.

What does it actually mean to unlearn something, and why does that question matter as much as learning new skills right now? With Graziela Tonin I talk about how universities, companies, and individuals are struggling to find their footing in a world where AI can write code, run tests, and sometimes outperform human decision-making. We get into the research showing that students who offload thinking to AI tools lose cognitive capacity rather than gain it, and what that means for how knowledge should be taught and shared. I keep coming back to her point that the rarest and most durable skills are not the technical ones: critical thinking, ethical behavior, knowing how to work in a team, and knowing how to learn at all. She also reminds me that the environment does not need us, we need it, and that is probably the sharpest piece of unlearning any of us could do right now.

What happens to the teams and organizations left behind after an agile transformation that didn't deliver what it promised? With Michael Mahlberg I talk about why the word "agile" has lost its meaning, how big consultancies pushed companies deeper into rigid structures under an agile label, and what it actually takes to help an organization find its footing again. We get into the question of what genuinely worked from the last 25 years, from fast iterations and real customer feedback to decision-making practices borrowed from outside the agile world entirely. What stays with me is Michael's point that arguing for change rarely works, but pulling people into the experience of what isn't working just might.

What does a tester actually need right now, when AI tools are everywhere and QA teams keep getting cut? With Tara Walton I talk about why communication and the fundamentals of testing matter more than ever, and why being the best bug-finder in the room means nothing if you can't explain what you found and why it counts. We get into how AI's "toxic positivity" makes critical thinking a skill you have to practice deliberately, and what it means that when you remove QA, your users become your first line of defense. I also ask Tara about career direction, and her answer comes back to something simple: follow what makes the light bulb go on, then go deeper into that.

How do you bring AI into a medical lab when the regulatory landscape is still taking shape? With Alexis Savkin I talk about exactly that tension, and the answer turns out to be less about technology than about focus and strategy. We get into why starting with a very specific, measurable problem makes the regulatory side manageable, and how a single-page risk diagram convinced compliance stakeholders faster than any technical pitch. I keep coming back to his point that regulators are not enemies but partners trying to make things safer, and that the real mistake is trying to solve "AI" as one giant problem instead of one concrete thing at a time.