Test Automation Pyramid and Its Multiple Misinterpretations

 

I Decided to Write this Post

I thought over a few months, should I write this post. I said, yes to myself for writing this post.  Before, I proceed, I want to express this:

Hello Mike Cohn, thanks for giving Test Automation Pyramid visualization to the software engineering community.  I see your thought process, experience, and work, in this model. This model's assistance to understand software systems is evident today.  

Again, thanks for giving this model as a metaphor for us.

What I See?

I see a risk for the Software Testing community especially when we use the word The Test Pyramid without knowing what it is and does it make sense. 

To keep you till the end of this post so that you get what is my point, I tell you this:

One of the common entities among Testing and Automation is the tests.  How the test's intent is expressed and executed in the testing and automation, is not similar; they are different and have to be different.  If similar, then it leads to misunderstanding -- automation as the only way of testing; or, the execution of tests by a human as the only way of testing.

Why did I Say "Yes" to Myself?

• The metaphor which Mike Cohn came up with is helpful
• I see, Mike Cohn knew what he is trying to say and it makes very much sense in today's Software Engineering
• His book Succeeding With Agile got published in 2010
• It has a chapter with title Quality
• I insist, that anyone who is involved in software development should read this chapter if cannot make the time to read this book
• This chapter talks about the pyramid
• The Test Automation Pyramid
• Mike Cohn came up with this concept
• This acts as a helpful metaphor and a heuristic
• In 2010, I don't know how much did it help then; but this model is a beautiful and simplistic explanation of how to categorize the software system's layers for test code
• In 2023, it helps everyone who is involved in the development of software system
• I have witnessed how software engineering has grown in the last 23 years
• Reading what Mike Cohn has to say about it, helps to clear the misunderstanding of The Test Automation Pyramid to The Test Pyramid, for first

Today the word "The Test Pyramid" is very common among people who are into software development. For this reason, is it misread and communicated?  Maybe!  The software engineering community refers to this model as The Test Pyramid and not as The Test Automation Pyramid.

I want to share my thoughts on the risks of calling it The Test Pyramid.

• I strongly see the software engineering community and importantly the software testing community need to know it

So, I have decided to write this post.

Quality Is a Team Effort

Mike Cohn, said this in 2010 in his book Succeeding With Agile. 

Today, who does not claim "We are Agile"?  Are your team and organization, Agile?

Yet, we see discussions on who should be owning and responsible for the quality while we are in 2023.  Especially the software test engineers ask this as they are said responsible for the quality of the software systems they are testing.

The chapter Quality has a section "Quality Is a Team Effort"; this section has the below lines:

Acquiring new testing skills, learning how to apply them within the strict timeboxes of Scrum, and paying off technical debts are the responsibility of the whole team.  These are not challenges to be sloughed off onto the testers.

Quality is everyone's effort and responsibility.

Test Automation Pyramid

Automation is one of the ways we do the testing.

Here are some of the tests (actions) that can be done through automation and asserted:

• Which are repeatable and can be asserted
• Which are run to setup, tear down, configure, create, edit, read, and delete the resource 
• Which are monitoring to inform the change in state/event/data
• Whose execution is in a defined order and trying to assert the expected outcome

Some tests are better evaluated via automation and with the help of automation.  Some tests are better evaluated by the execution of a human.

• Both are important and necessary in today's Software Engineering and Software Development

Back in 2010, the boxes were monolithic; the microservices were a theory.  Today, we are thinking beyond microservices, containers, serverless and more.

Given the complexity of  Software Systems, today, we need a model of how we can isolate it into categories and layers.  So that it is simple to learn, understand, to categorize the development of code & test, and execute the testing of same.

This is where I see the usefulness of a model given by Mike Cohn.  It shows the layered architecture pattern in a way, for a [part of] software system:

• That we are about to understand, and 
• Figure out the strategy for testing using automation in respective layers

Now, do you see the importance and help of the Test Automation Pyramid model?

Mike Cohn expressed the three layers in Test Automation Pyramid. It is as below.  I have copy pasted this below image from his book Succeeding With Agile. That way, I keep the original thought and picture of the model as given by the author.

Pic: The Test Automation Pyramid from Mike Cohn

Pic credits: Mike Cohn

Benefits of Test Automation Pyramid

• It helps to relook into the automation strategy
• It says, we need a better strategy to automate by showing us to categorize the tests in a layered pattern
• It helps to learn how the two different layers can communicate and can be used together in automation 
• For example, the service and UI
• So that tests are isolated and maintenance is manageable, while the deterministic feedback obtained from the tests automated is value-adding.

I have heard Dorothy Graham say she did automation when she started testing.  I see she has 40+ years of experience in Software Engineering & Testing.  So, automation is not new.  It is evolving.

This Test Automation Pyramid describes the automation which we talk about in day-to-day engineering to be more understandable and communicative.

Is This a Test Pyramid?

I understand, Test Automation Pyramid is not The Test Pyramid.

It is a pyramid for the tests carried out through automation.  

Test Automation Pyramid helps to identify, design, categorize, relate, map, and use the outcome of tests from another layer.  This benefit can be sought in both, that is, testing through automation and testing by a human.

This is a risk I see

• The Software Engineering and Software Testing communities are learning that automation is the only testing and the only way to test.
• No, it is not the only way to test and it is not what is testing all about
• Automation is one of the ways we do our testing and has its limitations
• Automation is a subset and a part of Software Testing
• Automation exists to assist Software Testing and to do better testing as a whole; not to replace the testing; not to claim it is the only testing and way to test in every dimension and context
• This is a problem for the upcoming generation of software engineers, and those engineers who take Software Testing as their full-time practice and job
• This sends a message that automation is testing and testing is automation
• This will be the problem to the practices of automation and testing in Software Testing
• For example, when I talk to test engineers and say, Test Automation Pyramid, I hear they are not aware of it; all they are aware of is The Test Pyramid of three layers used for automation
• As a result, there will be shallow results and benefits from both automation and testing with costs and short of value
• Here, automation and testing cannot complement each other effectively when used together

Calling it a Test Pyramid and referring it to just to talk about automation, is not right technically and logically.

I see Mike Cohn knows the differences between the value of Testing and the value of automation in testing, as different activities from his work and experiences.  I believe, for this reason, he labeled this model explicitly as The Test Automation Pyramid.

I did not find any references for Mike Cohn renaming this model as The Test Pyramid.

Also, calling it a Test Pyramid and the top of a triangle having a scoop or cloud with the name "Manual Testing" is misleading.  I see this is one more cause of confusion.  Does this triangle say to automate at every layer and just involve a human at the GUI interface for testing?

• No! We, humans, test at the service layer as well and also automate here
• At the unit layer,
• a human manages to setup/configure at the seam or/and use test doubles, stubs, mocks, and what is needed to execute the tests via code
• If you see, one pyramid, misinterpreted and misrepresented in multiple ways.

I learn, as a community, we are misinterpreting the Test Automation Pyramid model [work] and calling it by another name.  Is this right?

If one sees no difference between testing and automation, fine!  Then everything is applicable and looks alike.  Then calling Test Pyramid and Test Automation Pyramid does not make any difference.

But, actually, it is not the same [and similar].

The common thing between automation and testing is the tests.  The purpose, nature, procedure, design, strategy, and intent of the tests differ in automation and testing.

It is Test Automation Pyramid

We Software Test Engineers, think [of] [the] tests as automation most times because we believe and are said the programming and code are everything in the software system.

• The code of a software system is one of the byproducts of developing a Software System
• So are the tests.
• The test code is one of the byproducts of automation and not the whole software system

A part of the test is automatable. Which part? 

• It depends on how I express my test with its intent
• So that I will know which part and aspect of a test can be automated

Automation in testing is one of the subsets of Testing, but NOT similar to Testing in every dynamics, dimension, and aspect.

Automation has its own subsets of dynamics, dimensions, and aspects of testing.  If we practice by learning Testing and Automation are the same and if we remain in this understanding,

• We cannot identify
• The differences between Testing and Automation
• The differences between the tests identified for Testing through Automation and Testing by human
• The values each offer along with its limitations.

Referring it to as a Test Pyramid and just talking about automation out of it, is not Test Pyramid.

I call and refer it to as Test Automation Pyramid and not The Test Pyramid.  Again, I thank Mike Cohn for giving this clarity in his model and its name.

Let us refer to Mike Cohn's model by the name and understanding to which he refers it with.

If you ask/say, "it is a model; can't we improvise it to one's context?" 

• Yes, we can and a model is not fit for all contexts
• Showing the same model picture along with the same name for its attributes and expressing a different thought is a problem
• It misleads one if she/he takes this model as a reference
• This will have chains and it continues

To end,

• Embrace programming
• Practice programming
• Embrace automation
• Practice automation
• Implement automation
• Call and refer to automation as automation
• Know what is testing in automation
• Know what is testing
• Call and refer to testing as testing
• Practice testing
• Know your testing in Software Test Engineering
• Call and refer to the model with its name given by its creator/author
• That name has a purpose and intent
• Know it; understand it; learn it

References

• The book "Succeeding With Agile: Software Development Using Scrum" by Mike Cohn
• https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
• https://martinfowler.com/articles/practical-test-pyramid.html
• https://medium.com/tide-engineering-team/the-practical-test-pyramid-c4fcdbc8b497