Software testing is a hot topic in the tech world. The increasing importance of software across multiple industries, as well as a growing number of areas which need testing, means it is a fast-growing – as well as evolving – space. Comparison Technologies CTO Julian Browne discusses the developments in this field. 

 

What are your thoughts on independent testers versus integrated testing houses?

Like many agile practitioners, I view testing as an intrinsic part of the delivery process, so the idea of separating it out as a distinct exercise feels uncomfortable. Certainly, it used to be that way: something that happened after development as a way of confirming the development team were 'doing the right thing'. The former being the internal/requirement view and the latter being the external/quality view.

But if you think about it further, it’s clear there’s a third area, which is the ability to test new features with real customers to support learning, failing and evolving. To generate real value in software you need to be able to make things customers love or fail forward (meaning you’ve picked up learning experience) quickly – that’s competitive advantage. It’s a critical skill and a capability that product teams need to learn if they don’t have it.

Agile talks about the definition of ‘done’ which varies from project to project but basically means ‘we’ve demonstrated it and confirmed it does what it should’. But I think ‘done’ is changing. We build and test in ever faster cycles – Amazon for example famously updates its offering every 11.7 seconds – but without customer validation that’s cost, not value. Amazon knows some features will fail, but they opt to test and learn rather than launch and wait.

My own sentiment on testing is shifting. Whilst I’d not outsource the ownership of quality, or the need to think in terms of outcomes over requirements, I’d like the ability to quickly and cheaply validate which aspects of business demand are commercially sound, deliver them, and move on to the next thing. If someone could coherently pitch that as a service offering, I’d use it. 

What about the commoditisation of testing and the resultant price pressure?

Fifteen years ago, before testing was fashionable, organisations typically had a test team that tested during an independent late phase in the delivery cycle. Agile changed that: we learned that testing could be done at the same time as software development and then, with test-driven development (TDD), that it could be done before it. TDD taught us that software designed to be testable is cleaner and easier to maintain than software that’s tested after the fact. Suddenly becoming “test infected” was a thing and that gave rise to a revolution in aspects like pricing. Testing tools are so commoditised I struggle to think of one anyone pays for anymore, unless they prefer the convenience of running it ‘as a service’ in a cloud platform. Security is possibly the main exception to this rule. Enterprise tools like Fortify can charge for their service because they act as objective and independent assessors of secure coding standards. But even in security there are plenty of good open source tools if you have the expertise to use them well.

I see cloud testing tools going the way of anti-virus software – we used to pay for it and have to update our virus definitions ourselves (which we didn’t always remember to do) – now we just want to pay our subscription and have someone else keep it up to date.

Who offers better value, big players of scale or small boutiques?

I don’t think big firms or boutiques can provide the answer. Big firms may provide scale but software developers with deep expertise are not drawn to work for big companies unless they’re Google or Facebook, which they’ll happily join for less money. Ambitious brains will join or even set up boutique outfits, but then they face the real problem of engaging with boards that are more used to dealing with large consultancies like Accenture.

The truth is we need both: board engagement and the ability to clearly communicate why all this stuff is so important to commercial competitiveness followed by actual delivery of what is quite a complex set of highly technical skills. I’m not seeing many out there able to pull this off.

 

I think there’s a gap in the market for a ‘boutique-plus’ firm that can provide the expertise and also communicate it professionally at board level.

 

How is the role of QA evolving?

I think it’s becoming invisible. Fifteen years ago, we had test team who scripted ways to exorcise the bugs from our software. That was quality assurance. Then more and more it became the responsibility of software developers, called QAs or Software Development Engineers in Test (SDET), who sat in the development team and wrote tests with other developers. The team became one and quality assurance became a transparent service provided by it. Over the last couple of years however, my peers have noticed they’re hiring fewer and fewer QAs. Like architects and DBAs, the specialism of the role is disappearing due to commoditisation. The activity is still there, and smart companies are exploiting its democratisation for competitive advantage. 

 

Ultimately, it’s about what you do with it now, not what you do.

 

What role can crowd testing play nowadays?

Despite all the excitement of automation and advanced tooling, I am a massive fan of crowd testing. I’ve worked with highly skilled teams who still love a good “bug bash” session on a Friday afternoon, manually trying to break your own software. It’s effective I think because it’s random and replicates the vagaries of real-world use. It’s an idea rooted in the open source community that “given enough eyeballs, all bugs are shallow.” The more people you can get to participate, the more likely you are to spot something. The size of the crowd, combined with them coming at it context-free, means they’ll try all sorts of things. You can have all the agile tools in the world, but it takes a human to break software. I think there’s a big future for more structured and tool-supported crowd testing.