Automation Upended – How QA Consultants Sees Automation

Over our 30-year history, we’ve written blog posts on the topic of software test automation. Our most recent being, Uncommon (but essential) Metrics for Measuring Software Test Automation ROI, supported by a downloadable guide. While technical material like this is important to the success of test automation projects, we thought it was important to highlight how our approach and philosophy have matured over the years, and recently, our need to reinvent how test automation was delivered. And since then, it has been the largest part of our rapidly growing business.

In short, we took a step back to review the market about 3 years ago. This was after we had, at that time, seen more than a dozen client engagements, involving several different vendors, where we were brought in after a low-cost offshore vendor had already implemented a test automation framework. Through that, a pattern emerged of “tool engineers” being asked to use the tools and frameworks they knew to automate tasks in testing without a True North goal. It was not value-based.

Further, as with most offshore delivery models, the incentive on the vendor was larger teams, and therefore progress and value are measured in the number of tests, lines of code, and number of bodies. It continued to baffle us that the value of automation was being measured by the volume of tests vs. the value of tests. And as a consequence to the volume of code, scripts, and people, that it become spaghetti code! In some cases, more code to manage and maintain than the applications to be tested.

At the same time, two other things were happening:

1. The rise of the SDET(Software development engineer in test) as a defined role, and
2. Our continued analysis of the psychology of team-based software development

(Note, automated testing IS software development and especially in this case).

With regards to SDETs, unfortunately, that term has now been overused and diluted by the large QA subcontracting community. Currently, anyone who used to be a test automation engineer or architect and may know some tools and frameworks, is calling themselves an SDET. This is making our recruiting teams work overtime to sift through the talent. That is NOT our definition of an SDET. For us, we ask a simple question: “Can this trained and experienced developer who has chosen a career in QA, be trusted to push code to production?”. Through this definition, we can quickly identify candidates who have never done it before (they’re not for us), who have not chosen this career (they are not going to stay with us for long), and who have not been part of complete code deployment teams (this is required for our test automation models). And we always prefer to grow our own SDETs from the smart and talented developers who have hit the market recently as part of our talent accelerator program. It is these resources, and a team-based approach, that provide the greatest value for our customers with very rapid results.

Regarding the psychology of development, we have found that individual contributors are a weak link, and a 2-person team usually ends up with hierarchy and heartache. A 3-person team has become our magic number. We have modified some language from the “Spotify model” and have retooled our entire Quality Engineering practice to deliver based on QE Squads. Each squad is a dedicated team of 3 SDETs with a squad leader who is part of every project. And these squads are built to provide the greatest value to a customer project based on their combined skills and expertise. These squads take on tasks in 2 weeks sprints that can be shifted on demand and wherever the highest burning fire is.

As a team of SDETs, they are expected to assess and analyze with an “Automate First” approach. This is usually where they are asked to start. Whether to build a framework and approach from scratch including scope and tool analysis, creation and deployment (Usually 4-8 sprints for the first smoke test), or to assess and improve an existing automation approach that had already been attempted. However, many times, the squad needs to start with a focus on manual testing, test asset creation and documentation, user story development, etc. That is fine too. Not everything can be automated and not everything should be automated, but establishing the guardrails for an organization is critical to making sure that automation becomes part of the culture. The critical value of the squads is that they are fixed price per week and as the needs of the project change, that change is absorbed, teams are modified, or new squads are added. If for a few weeks, additional data entry support is needed (test data management) or some more expertise in automaton framework architecture, there are no additional charges. As the backlog of automation development work grows, the equation becomes linear as to when a second or third squad will be needed (with volume and duration discounts applied, of course.)

So, what does this all mean in practice and how does it all come together?
We now have purpose-built teams, staffed with our version of ideal automation talent, with proven expertise, and a way of working that is value-based and sprint driven to meet our customer’s goals, not increasing the size of the team.

Our squads and leagues (groups of squads) work as teams, solving complex QA problems. This has grown from traditional automation frameworks to DevOps configuration and integration.

We fully believe all automation should be automated and if there is a need for someone to “kickoff” automation by pressing “play”, then it is not automation. The DevOps pipeline component has grown significantly as our teams have developed that expertise and we are now regularly seeing requests to configure pipelines, write the YAML, and get everything automated (with both functional and performance tests automated as part of build). And yes, this has increased in scope to include how we approach performance engineering. In fact, our performance engineering squads have been able to surpass our functional squads in terms of “time to value” because of the unique nature of that work as a managed service.

We currently have several active squad projects in Healthcare, Property & Casualty and Life Insurance, Retail, and Aerospace, among others. In all cases, this approach to not just “test automation”, but value-based quality engineering that has allowed our customers to manage us to their expected outcomes, increase quality coverage, reduce costs, effort and time as needed, and of course recognize the results in either customer retention, revenue, or experience. Reach out if you would like to learn more or even speak with our practice, league, or squad leaders.