There Is No Flaky Test! Then, What Is Flaky?

In general, I understand, the flake means brittle and inconsistent or uneven.  Like the flakes one will have as food.

I'm not a native English.  It took me years to understand and correlate between the words 'flake' and 'flakiness'.

Test and Flakiness

There are no Flaky Tests.  A test is not flaky anytime.

The responses of a system to a test can be flaky.  That is, the response is not deterministic and inconsistent to one's expectation and understanding.  

In other words, the responses of the system is unreliable to the same action, data, state and other parameters.

This is not a philosophy of testing or test.  In my opinion, it is a misunderstanding that is accepted and passed on in the software engineering.

A test is not flaky; the response of a system is flaky.

Are there no flakiness when testing for services?  It exists!  That is, the inconsistent behavior of a service for the same set of data, state and other parameters. What is inconsistent in context of a service's response?

Flakiness is not bounded to just GUI  It can be with any UI [User Interface].  And, an API is an another user interface which opens up to the bunch of services.

To end, 

• It is not Flaky Test.
• It is a flaky response of a system which a test is letting you and me to experience.
• What is a flakiness to one who is testing for security and performance?
• Flakiness is not confined to the UI response or a functional response of a system.

You decide how you should be seeing a test in your suite and label the response of the system you are testing.

Give a Name for the Base64 Image in an Extent Report

 

I read the below question in Testing Mini Bytes community.

Hi All, I am using extents reports in my c# automation framework. I am trying to attach the screenshot but it is always showing with name as base64img. Any idea how to overwrite that base64img to customised text?

Capturing a Screenshot in the Test Run

With the Extent report, one can capture the image of screen during the test run.  This capture can be on certain condition or state being met.  That is, it can be -- on fail of a particular action; in a transition between the states; etc.

In an Extent report, when the image is captured as Base64 image and the thumbnail is set to false, the images captured will have name "base64img".  Clicking on this, I see the respective image.  

No Customized Text as Name for Base64 Image

Say, a Extent report has 10 images captured as Base64 image with thumbnail false.  All these 10 different images will have the same name -- base64img.

• This can be more chaotic to interpret when a test run has captured more than one Base64 image, and, all have the same name -- base64img.
• Do you see that? 
• How do you know what is that image without opening it just by reading base64img?
• That's hard and you have to make additional clicks to know what is in each image!
• Is this a better use of time of you who is reading the Extent report?

Naming the Base64 Image in an Extent Report

I see, the semantics in C# should also be similar to what I'm sharing below.  

The above gist should help to have a customized text given for each captured Base64 image.

In this case, the captured Base64 image name will be the customized text "Resend_OTP_Button_Not_Available" and not base64img.  This makes it more contextual and relatable for the reader of an Extent report.

My First Hand Analysis of CrowdStrike Falcon Update Incident

I attempted to analyze the process dump of CrowdStrike shared by my friend.  He said, there could be an attack which is leading to crash of Windows OS globally.  This made me curious to look into the dump and learn.

I had no much context around it, but, a test engineer in me did not sit quite.  I started to analyze the dump information.  Here is my first hand analysis that I made on 19th July 2024 post 10:30 AM IST.

What I Saw?

• It is a Windows OS's process dump.
• Looks like something with C or C++ application reading how the memory offsets were in the dump.
• It started to read a memory offset.
• Then the process witnessed an exception.
• Here the program could not read further
• Why it could not read further from this offset?
• My little experience of testing drivers on Windows OS for a card printer machine, refreshed and recalled what I had witnessed when testing.

Scratching and Striking My Mind

I started to ask these questions myself while I asked what could have gone wrong.  I could not stop here as I was curious what led Windows machine crash.  I referred to web and learn there was an update by CrowdStrike, and then this incident.

The bugs do exist in every software no matter the level and depth of testing, automation and engineering's excellence.  All software do crash and OS is not an exception to it.  But, what made the update to crash the Windows OS?  Pointing and blaming CrowdStrike or Microsoft is not a way for the practicing test engineer.  If these two organizations are serving its huge customer base, they have something working and reliable.  Engineering does not eliminate problems.

By now, I had a thought that it is not an attack.  It is a software bug!  Where is the bug?  What is the bug?  Was it not experienced in pipeline?

The Open Ended Questions

I had these questions as I analyzed and spoke to my friend.

• What is Falcon?
• What was this update to Falcon?
• How frequently the updates are rolled out?
• How the updates are rolled out globally?
• What pipeline do they have in testing?
• Who is impacted the most in business? Is it Microsoft or CrowdStrike?  Impacted in what way?
• What is CrowdStrike?  What they do?  Who are the customers?
• Where do the CrowdStrike's Falcon sit in the OS and what it does?
• How CrowdStrike works in the machines and what it offers?
• What do the dump say? Relook into it with different perspectives.
• How this could have been prevented?
• How will I prevent this if I join this team knowing this incident?

With these questions, I started to analyze the process dump which was shared.

I had more such questions, but these were the first few that I crossed as I started.

Analysis of Process Dump

My interpretation, tells me the below for today

1. Accept that it is an incident as any other incident which I witness in production environment.
2. Do not fall to the speculation happening around.  Remain calm and focus to interpret and understand your exploration.
3. I see, if it can start to read from an offset and then ending to experience a non-existent or invalid offset, is it a NULL Pointer?
• What is NULL Pointer?
• A NULL Pointer is a pointer that does NOT point to any memory location and hence does not hold the address of any variables.
• If I do not initialize and assign, the pointer will have NULL as its value.
• For example, int *test;
• When I want to access the pointer test (a location in memory) pointing to, I will not be sure what is in the pointer when I read it.
• I may not set it later or set it.
• In this case, the code can tell if the pointer is valid or pointing to a garbage memory
• But, if I declare it like int *test = NULL;
• I can check if was set and initialized
• It is a better practice to assign a NULL value to a pointer during initialization so that we can check if it is NULL or as any address assigned to it.
• This understanding of Pointer makes me think, is it due not initializing a pointer and so the error code c0000005 on reading a memory that is not valid.
• When we assign a NULL value to pointer, it is a null pointer in C++
• We assign null value for testing and asserting
• If the memory is allocated to a pointer or not
• If it has a return address and is a valid one or not
• If a pointer is not initialized, assigning null it prevents problems to certain extent
• With this understanding, I also read, it started to read from an offset 0x9c, and then failing.
• What is 0x9c?
• In Octal it is 234. In Decimal it is 156.
• Can there be such address in a computer's memory? I don't know.
• If it is a access violation, then is it a memory which is in preemption of the OS?
• If so the OS can terminate the program or process which is trying to access it.
• Is this killing the process and aborting the operation of Falcon's IPC and eventually Windows coming to BSOD?
• This tells me it is not a NULL Pointer in first case but not initializing a pointer to NULL.
• I infer, if the pointer was assigned to NULL, that is initialized, there could have been some hint in the state and event when accessing the memory.
• This is my analysis; but, I have not seen the test code nor aware of the product.  All this inference is based on the process dump and my experience of testing drivers.
• It got something in between from update (a config or pattern?) for which it cannot find and read in the memory?  Why?
• This indicates me, it could be a bug, that is, a logical problem.  This is my hunch for today!
4. Data in the dump
• Exception Address
• Read from Address 0x9c
• Exception Code: c0000005 (Access violation)

Testing my Interpretations

CrowdStrike as an org when it caters its SAAS to such a customer base, won't it have a testing pipeline

• It will have, I have no doubt in it.  They test and roll out the updates, I believe in it.

Did they witness any such incidents earlier?

• I searched on web for it and I did not find something similar on the Windows, earlier.

Is this a NULL Pointer?  Are you sure?

• No, I'm not sure.  But, there is something that is leading it to address which does not exist or which is invalid?  I will have to wait for their RCA to know technically what caused this.  But this is my understanding reading the dump.

How do you think it is a memory access problem?

• The error code 0xc0000005 says that.
• I referred to driver easy website for the information because my experience of testing the drivers for Windows OS and experiencing such incidents led me there.  This is what I learn:
• https://www.drivereasy.com/knowledge/solved-how-to-fix-0xc0000005-error/

Do you think the programmer would not have handled the obvious Pointer and NULL initialization?

• I believe there will be a check for Pointer and what it is pointing to.  But is it due to no initialization?  Technically this has to be analyzed which I cannot do.  I will have to wait for CrowdStrike team to share the tech details.

Is this a driver problem that killed the Windows kernel?

• I don't know.  But, the .sys file will not have driver as per my learning.  It will have information about the drivers and any configurations.
• This incident is a problem, which impacted both CrowdStrike and Microsoft.  Maybe, both will have their areas to look and fix it they see so.  But, in this context, CrowdStrike can fix it quicker and that is much better -- is what I understand.
• I'm a Windows user for long time.  I see, Windows has worked well to all my contexts so far.  The Engineers of Windows OS knows better than me here.  I'm not well aware and informed as they are.
• CrowdStrike's engineering team are skilled and they are rolling out updates often in a day.  They have a better pipeline when this is being done.
• But, the question I have is, how did this happen?
• No one lets such problem into production when they are aware of it.  Do you?
• There is something that has not come to their observation and experience.  What is that?
• Knowing this will help to prevent this and similar incidents happening in future.
• I'm waiting to know what did not come to their experience and led to this incident.

What could be in the .sys file of CrowdStrike?

• I don't know!  I want to learn that.
• But, from my testing of .sys file and drivers on Windows OS, I learn there could be a configuration details with certain pattern or information to capture at run time, and help the installed software to run.  This is my learning and awareness from my testing.
• That said, testing at OS level and Anti Virus engines are not obvious.  Testing of drivers is like the risky mines.  What is sufficient and good enough in test coverage?  It needs an expertise at OS internals level.
• Windows OS having such a fragmentation in its versions, updates and patches, it is a battle field and mines for engineers building such solutions for sure!
• I learn, the Windows OS stopped when an application tried to access the invalid region or non-existent memory.
• The update which was rolled out, did it have a configuration or a pattern that showed a logical problem when processing it?
• I have such questions and thoughts that are striking my mind as I think and build a problem model for the same.

Is this a race condition incident?

• I see, it is not a race condition incident as users across globe experienced it.

Is this specific to a Falcon version, OS version and hardware?

• Not all host machines would be on latest version of Falcon, is my presumption.
• At least, n-1 and n-2 versions should be on host machine which experienced this behavior.
• So it is not a Falcon version specific, I see.
• It looks to me as it is not specific to the Windows OS version and hardware configuration.
• It is an application software problem which occurred at driver level is what I see.
• This is an IPC communication and process is my understanding.
• The driver can receive the IPC communication in continuous mode.
• At times, this can get queued based on the application and what it does.

Where is the Problem?

Well, I'm looking and pulling from my visualization by relating with my experience of testing the driver on Windows OS.  I don't know the exact reason or close enough to tell what could have gone wrong.

Reading the process dump, it says accessing a memory that does not exist or corrupted.  One of the high possibility is, the starting offset is seen but it is not helping when reading.

• For example, Ravi has the address of India's Prime Minister house.
• But, he does not know from where to start despite having the address.
• He is void and null in knowing where to start and what to do when he is not initialized with the start location to begin the travel to the Prime Minister's house.
• In short, he do not know where the address is pointing to and what it has, though he is given a address to start.
• Can he access the Prime Minister's house premise without any access granted and authorized to do so?
• If not, won't he be arrested by police or other security forces and stop him?

Do I Know the Precise Problem?

I don't know!  I do not know the CrowdStrike product and platform.  I'm waiting to read the technical details from Crowd Strike.

I see, it comes to the data, state and event.  I would focus on how to prevent it learning which data, state and event led to this behavior.  I think of figuring out the Test Design and Strategy that can help me to identify such use cases.  I focus here and see can it brought into the automation so that it gets exercised and regressed consistently.

If it is due to the memory access that had a problem, I did such tests when testing driver for a hardware machine on Windows OS.  I will share the tests that I did in upcoming blog.

I wrote the technical analysis from process dump to CrowdStrike and Microsoft.  I did not get a response.  Anyways, I'm sharing the overall information in a non-technical way so that it is consumable to most readers here.

Note: Here are another threads of me sharing my thoughts on same:

1. https://x.com/testingGarage/status/1814215089525821763?t=XSFdx69ElL0ZmBOcEFrTjg&s=19

2. https://www.linkedin.com/posts/ravisuriya_%3F%3F%3F%3F%3F%3F%3F-%3F%3F%3F%3F%3F%3F%3F%3F%3F%3F%3F-%3F-activity-7221156949445206017-oeRa

Deep Link and its Testing via Automation

 

I get these question consistently from my fellow testers and community.

1. How to automate the mobile apps and web applications using Deep Links?
2. How to automate the business flows using Deep Links?
3. How to achieve end-to-end business flows testing on using Deep Links?
4. How to automate scenarios in mobile apps using Deep Links?
5. What is the best approach to automate the mobile apps using Deep Links?
6. What is the best practice to automate using the Deep Links?

And, more questions on same pitch.

No Deep Dive into - What is Deep Link?

A hyperlink in HTML is a kind of deep link within a website or to another website.

Deep Link is known with different names for web, Android app and iOS app.  All these names have the same understanding and intent at some point.

The Deep Links are URIs that takes me directly to a specific part (activity or fragment) of the app that I'm using or testing.  The Deep Link will have an intent which tells where I will be taken on using it.

When we converse on diving deep technically into testing and automation of Deep Link, will share more insights into its internals.

Deep Link and Challenges

This question is discussed with me often:

How to do end-to-end testing using the Deep Link?

Automation of a mobile app using Deep Link poses a challenge which is not experienced in web application.  

One such challenge is, say you have not installed the mobile app.  [This is solvable!]

• On using a Deep Link, I should be taken to Apple Store or Play Store based on the app.
• I have to install the app.
• Post this, in the traditional automation, I should start traversing the business work flows via GUI.
• Is this adding to the flakiness aspect of automation via GUI?

When we talk so much about flakiness and how to avoid (not prevent), should we exercise business workflows when automating using Deep Link?  What you are thinking?  Let me know!

Scoping of Automation Using Deep Link

Back to the fundamentals.

• We have to automate, no escape from it.  Let us automate what must be automated!
• Let us not fall into trap of "Automate everything!"
• For today, I'm in this mindset and attitude,
• What we automate depends on the objective or goal that we want to accomplish.
• Each test should have precise and deterministic goal.
• A test via automation is not an exemption to it.
• A test defined in automation should be precise, deterministic and have a single objective - Single Responsibility Principle.

What is the objective of my testing via automation for the Deep Link?  This define the scope and extent of my automation.  This will minimize the number of checks that I do using Deep Link.

The purpose of Deep Link is to take me to specific part of the mobile app.

• Should I start the end-to-end or exercising the workflow to be included in the Deep Link tests?
• If included, am I not complicating the testing via automation?

Automation using Deep Link

I ask this question to myself and to my team.

What is the goal of testing via automation using Deep Link?

This question helps me to pick minimal and necessity flow actions.   It has lead and leads me to define minimal tests for Deep Link based on what we want to learn from automation of same.

To me, the purpose of Deep Link is not end-to-end testing.  It's purpose is,

Am I taken to the intended state and data when used the Deep Link?

I have kept the test intent to this.

With this, I have come with tests that has minimal must evaluation and assertion to learn if the app is responding or not to the Deep Link.  This is what the business wants when the Deep Links are created.

The app usage and workflow function is not a problem statement of Deep Link in a general context.

Deep Link is not for end-to-end.  It is to take to you from a point to another point, that's it.

Are you automating using Deep Link?

When I Questioned My Learning -- What is Automation?

I'm questioning, revising, refining, and versioning to update my learning and practice with Automation in Software Testing & Engineering.  This blog post is about how do I define and understand the Automation and Software Testing.

Automatic and Ation

I started by breaking it into classes (words).  I see the better picture of what it does and what it means.

• Automatic
• which runs on its own with little human intervention where it is needed.  
• Ation 
• means an action

When put together, Automatic + Ation = Automation.

I do not want get into from which language did it come from.  I got what I need here.

Automation and Software Engineering

Today, the context, area, scope and subject of Software Engineering has grown when compared to last two decade.  This is a immense change, progress, and growth.  I can think of automation in each area of Software Engineering, today.

I will stick on to context of programming the code, testing of this code, and what to automate by learning how and why.  My work and practice is here for last one decade.

I unlearn and learn by redefining to myself:

Automation is the use of technology to execute a task with reduced human efforts.

I said, reduced human efforts and not elimination of human, and human efforts. 

What is Software Testing?

It is a parallel engineering that is engineered in Software Engineering.  When said engineering, it is about civilization, learning the [actual] problem and solving the [priority] problem.

Then, Testing is about learning how well it is being engineered.  It is an exploration, study, execution, evaluation and connecting all these to know the differences if any with its value and cost.

Testing is part of Engineering without which  Engineering cannot exist.  Testing goes in parallel as an engineering to the engineering.

That said, Software Testing is an engineering that runs in parallel to Software Engineering.  To me, the definition of Dr. Cem Kaner for Software Testing looks to be closer as a engineering practice.  It says,

• Software Testing is an
• empirical
• technical
• investigation activity
• carried out to provide quality related information
• to stakeholders
• so that they make informed decision

With the help of Software Testing, to an extent I will learn:

• In what all ways the Software System appears to work - how, why & when
• In what all ways the Software System fails - how, why & when

I will have a technical data that is rational and empirical to present my learning from the testing I do.

Automation in Software Testing

I'm learning consistently for over a decade now that automation in software testing is one of the ways to test and this has its own constraints and limitations.   It exists to assist the Software Testing & Engineering, and not to replace the Software Testing & Engineering.

When said automation, I use to think of Programming Language and executing a test via the code where the intent of the test is expressed.

But, today I learn, I can automate in multiple ways and in few context even without using any programming language.  Yet it is still an Automation in Software Testing.

I used a toy train to automate a use case of payment via card and NFC.  This worked very well in fact with actual card and POS equipment.

Most times, one equates Automation to the code.  It is not wrong!  But it does not have to be code of a programming language always, either. 

As I said, if Software Testing is an engineering, then, Automation in Software Testing is also an engineering.

What About You?

Did you redefine your understanding by unlearning?  Be it yes or no, I'm curious to listen you.  Let us catch up!

How To Start Learning The Automation?

 

I had this question.  I figured it out how to help myself.  I figure out better ways every day.  I read this question by other Test Engineers on forum and communities.

Hi everyone

I want to start learning automation testing.  Where to start and how start?  Can anyone guide please? 

You might have across the similar question.  And, you might as well have this question. Do you?

I share here in the way I say it to myself!

How Do I Start Learning Automation?

• Find all the ways you can for how to fall in love with Programming
• Fall in love with Programming
• Choose a language to express your love
• You have to express and practice better ways to express each day
• You will get better in proposing as you express it each day
• Is it one way love?
• Don't worry! It will be two way soon
• Let Programming understand you, and you understand the programming
• Love at first sight is not for all
• Some will take enough sights to fall in love
• And after that, you know; if not, you will know
• Do not give up! One small step at a time
• See the Testing in your work
• Improvise your Testing each day while you are in love with Programming
• Testing will help in getting you and programming to love each other
• Take help of this friend very well by understanding who is this friend
• Learn consistently how to express the intent of your test via the code you program
• Know the limit of the test intent you have expressed via the code that you have programmed

If you noticed there is no automation libraries, tools and Test Runners mentioned here.  All that is mentioned to get started is:

• Programming and love for it
• Understand and seek the help from friend -- Testing
• Know the test intent expressed in your programming and its limitations

I want to do better and effective automation in testing to help myself and my testing.

Black Box in Every Other Box of Software Testing

 

Modeling Software Testing With Boxes

The fact is something that is not put to scrutiny or questioned much and often.

As a fact, the Software Testing is explained to us using boxes.  That is,

• Black Box
• White Box
• Grey Box

Is this wrong? No, it is not wrong.

There was a need to explain for one how to visualize -- how a person would interact and interpret the software system when testing.  The analogy of these boxes helped and helps.  These boxes are mindsets.  In a way, these boxes are like models to interpret the different ways and approaches we take in Software Testing.

I see, we are seeing Black Box in every other box.  Maybe, this is limiting one not to think to learn software testing in other than a black box mindset.

If you ask, are we not automating? Is not that a Grey Box?  Very much, we are a black box mindset when writing automation as well.  I include myself here.  I'm exploring how to break out of this and see the Software Testing.

Do programmers think of their code as different boxes?

• A programmer reads, writes, deletes, and views the data. events and state
• The programmer as well cannot see what's happening between the binaries on the electric circuits
• The programmer evaluates her/his code's testing via logs, debugger, and assertion for the data, state, and events
• Is this a white layer or box?
• Is it called white because one can see through the logs, debugger, and assertion?
• But that is still not a sight of binaries on the electric circuit, right?
• If one could see through the binaries, we should not be having race conditions, out-of-memory, and unhandled exceptions
• Isn't so?

This makes me feel, there is a Black Box in every box and we are largely confined to this Black Box.  

Exploring to step outside this box helps to understand the testing, software testing, system [and software] under test, and what is needed to test better.

Test Automation Pyramid and Its Multiple Misinterpretations

 

I Decided to Write this Post

I thought over a few months, should I write this post. I said, yes to myself for writing this post.  Before, I proceed, I want to express this:

Hello Mike Cohn, thanks for giving Test Automation Pyramid visualization to the software engineering community.  I see your thought process, experience, and work, in this model. This model's assistance to understand software systems is evident today.  

Again, thanks for giving this model as a metaphor for us.

What I See?

I see a risk for the Software Testing community especially when we use the word The Test Pyramid without knowing what it is and does it make sense. 

To keep you till the end of this post so that you get what is my point, I tell you this:

One of the common entities among Testing and Automation is the tests.  How the test's intent is expressed and executed in the testing and automation, is not similar; they are different and have to be different.  If similar, then it leads to misunderstanding -- automation as the only way of testing; or, the execution of tests by a human as the only way of testing.

Why did I Say "Yes" to Myself?

• The metaphor which Mike Cohn came up with is helpful
• I see, Mike Cohn knew what he is trying to say and it makes very much sense in today's Software Engineering
• His book Succeeding With Agile got published in 2010
• It has a chapter with title Quality
• I insist, that anyone who is involved in software development should read this chapter if cannot make the time to read this book
• This chapter talks about the pyramid
• The Test Automation Pyramid
• Mike Cohn came up with this concept
• This acts as a helpful metaphor and a heuristic
• In 2010, I don't know how much did it help then; but this model is a beautiful and simplistic explanation of how to categorize the software system's layers for test code
• In 2023, it helps everyone who is involved in the development of software system
• I have witnessed how software engineering has grown in the last 23 years
• Reading what Mike Cohn has to say about it, helps to clear the misunderstanding of The Test Automation Pyramid to The Test Pyramid, for first

Today the word "The Test Pyramid" is very common among people who are into software development. For this reason, is it misread and communicated?  Maybe!  The software engineering community refers to this model as The Test Pyramid and not as The Test Automation Pyramid.

I want to share my thoughts on the risks of calling it The Test Pyramid.

• I strongly see the software engineering community and importantly the software testing community need to know it

So, I have decided to write this post.

Quality Is a Team Effort

Mike Cohn, said this in 2010 in his book Succeeding With Agile. 

Today, who does not claim "We are Agile"?  Are your team and organization, Agile?

Yet, we see discussions on who should be owning and responsible for the quality while we are in 2023.  Especially the software test engineers ask this as they are said responsible for the quality of the software systems they are testing.

The chapter Quality has a section "Quality Is a Team Effort"; this section has the below lines:

Acquiring new testing skills, learning how to apply them within the strict timeboxes of Scrum, and paying off technical debts are the responsibility of the whole team.  These are not challenges to be sloughed off onto the testers.

Quality is everyone's effort and responsibility.

Test Automation Pyramid

Automation is one of the ways we do the testing.

Here are some of the tests (actions) that can be done through automation and asserted:

• Which are repeatable and can be asserted
• Which are run to setup, tear down, configure, create, edit, read, and delete the resource 
• Which are monitoring to inform the change in state/event/data
• Whose execution is in a defined order and trying to assert the expected outcome

Some tests are better evaluated via automation and with the help of automation.  Some tests are better evaluated by the execution of a human.

• Both are important and necessary in today's Software Engineering and Software Development

Back in 2010, the boxes were monolithic; the microservices were a theory.  Today, we are thinking beyond microservices, containers, serverless and more.

Given the complexity of  Software Systems, today, we need a model of how we can isolate it into categories and layers.  So that it is simple to learn, understand, to categorize the development of code & test, and execute the testing of same.

This is where I see the usefulness of a model given by Mike Cohn.  It shows the layered architecture pattern in a way, for a [part of] software system:

• That we are about to understand, and 
• Figure out the strategy for testing using automation in respective layers

Now, do you see the importance and help of the Test Automation Pyramid model?

Mike Cohn expressed the three layers in Test Automation Pyramid. It is as below.  I have copy pasted this below image from his book Succeeding With Agile. That way, I keep the original thought and picture of the model as given by the author.

Pic: The Test Automation Pyramid from Mike Cohn

Pic credits: Mike Cohn

Benefits of Test Automation Pyramid

• It helps to relook into the automation strategy
• It says, we need a better strategy to automate by showing us to categorize the tests in a layered pattern
• It helps to learn how the two different layers can communicate and can be used together in automation 
• For example, the service and UI
• So that tests are isolated and maintenance is manageable, while the deterministic feedback obtained from the tests automated is value-adding.

I have heard Dorothy Graham say she did automation when she started testing.  I see she has 40+ years of experience in Software Engineering & Testing.  So, automation is not new.  It is evolving.

This Test Automation Pyramid describes the automation which we talk about in day-to-day engineering to be more understandable and communicative.

Is This a Test Pyramid?

I understand, Test Automation Pyramid is not The Test Pyramid.

It is a pyramid for the tests carried out through automation.  

Test Automation Pyramid helps to identify, design, categorize, relate, map, and use the outcome of tests from another layer.  This benefit can be sought in both, that is, testing through automation and testing by a human.

This is a risk I see

• The Software Engineering and Software Testing communities are learning that automation is the only testing and the only way to test.
• No, it is not the only way to test and it is not what is testing all about
• Automation is one of the ways we do our testing and has its limitations
• Automation is a subset and a part of Software Testing
• Automation exists to assist Software Testing and to do better testing as a whole; not to replace the testing; not to claim it is the only testing and way to test in every dimension and context
• This is a problem for the upcoming generation of software engineers, and those engineers who take Software Testing as their full-time practice and job
• This sends a message that automation is testing and testing is automation
• This will be the problem to the practices of automation and testing in Software Testing
• For example, when I talk to test engineers and say, Test Automation Pyramid, I hear they are not aware of it; all they are aware of is The Test Pyramid of three layers used for automation
• As a result, there will be shallow results and benefits from both automation and testing with costs and short of value
• Here, automation and testing cannot complement each other effectively when used together

Calling it a Test Pyramid and referring it to just to talk about automation, is not right technically and logically.

I see Mike Cohn knows the differences between the value of Testing and the value of automation in testing, as different activities from his work and experiences.  I believe, for this reason, he labeled this model explicitly as The Test Automation Pyramid.

I did not find any references for Mike Cohn renaming this model as The Test Pyramid.

Also, calling it a Test Pyramid and the top of a triangle having a scoop or cloud with the name "Manual Testing" is misleading.  I see this is one more cause of confusion.  Does this triangle say to automate at every layer and just involve a human at the GUI interface for testing?

• No! We, humans, test at the service layer as well and also automate here
• At the unit layer,
• a human manages to setup/configure at the seam or/and use test doubles, stubs, mocks, and what is needed to execute the tests via code
• If you see, one pyramid, misinterpreted and misrepresented in multiple ways.

I learn, as a community, we are misinterpreting the Test Automation Pyramid model [work] and calling it by another name.  Is this right?

If one sees no difference between testing and automation, fine!  Then everything is applicable and looks alike.  Then calling Test Pyramid and Test Automation Pyramid does not make any difference.

But, actually, it is not the same [and similar].

The common thing between automation and testing is the tests.  The purpose, nature, procedure, design, strategy, and intent of the tests differ in automation and testing.

It is Test Automation Pyramid

We Software Test Engineers, think [of] [the] tests as automation most times because we believe and are said the programming and code are everything in the software system.

• The code of a software system is one of the byproducts of developing a Software System
• So are the tests.
• The test code is one of the byproducts of automation and not the whole software system

A part of the test is automatable. Which part? 

• It depends on how I express my test with its intent
• So that I will know which part and aspect of a test can be automated

Automation in testing is one of the subsets of Testing, but NOT similar to Testing in every dynamics, dimension, and aspect.

Automation has its own subsets of dynamics, dimensions, and aspects of testing.  If we practice by learning Testing and Automation are the same and if we remain in this understanding,

• We cannot identify
• The differences between Testing and Automation
• The differences between the tests identified for Testing through Automation and Testing by human
• The values each offer along with its limitations.

Referring it to as a Test Pyramid and just talking about automation out of it, is not Test Pyramid.

I call and refer it to as Test Automation Pyramid and not The Test Pyramid.  Again, I thank Mike Cohn for giving this clarity in his model and its name.

Let us refer to Mike Cohn's model by the name and understanding to which he refers it with.

If you ask/say, "it is a model; can't we improvise it to one's context?" 

• Yes, we can and a model is not fit for all contexts
• Showing the same model picture along with the same name for its attributes and expressing a different thought is a problem
• It misleads one if she/he takes this model as a reference
• This will have chains and it continues

To end,

• Embrace programming
• Practice programming
• Embrace automation
• Practice automation
• Implement automation
• Call and refer to automation as automation
• Know what is testing in automation
• Know what is testing
• Call and refer to testing as testing
• Practice testing
• Know your testing in Software Test Engineering
• Call and refer to the model with its name given by its creator/author
• That name has a purpose and intent
• Know it; understand it; learn it

References

• The book "Succeeding With Agile: Software Development Using Scrum" by Mike Cohn
• https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
• https://martinfowler.com/articles/practical-test-pyramid.html
• https://medium.com/tide-engineering-team/the-practical-test-pyramid-c4fcdbc8b497

Maven For Software Test Engineers - 101

 

Why 101s?

In the initial days of my career, I looked and referred to multiple sources to understand the technology stack.  As I progressed, I looked for someone who can help me in understanding what I'm reading and to clarify the questions I witnessed.  The understanding of concepts took time when I actually started using it and especially when debugging.

Today, if I look at that, I ask myself

• How can I make my learning experience a happy one?
• How can I capture the fundamentals and have a smooth revision of it when in need?
• What can I have for my reference when I'm implementing the fundamentals?
• How can I increase the speed in which I do all these?

If seen, I want to help myself to be better and effective when solving a problem by implementing what I'm learning.

Maybe, other software test engineers too have same question.  Don't you?

I'm learning new stuffs, and also unlearning what I'm learning.  I create the 101s for myself.  I see the software testing community can also make use of these 101s that I'm building.

Why You Need Maven 101?

• Today, it is a common sight that one quickly picks up a boilerplate code and run it
• Especially with the software test engineers, who wants to start/practice automation
• Pick a framework and run a command
• Execute the program successfully
• Or, get stuck in between and do not know why is that command
• Or, see an error with that command and not sure what that command is doing
• What is the difference between these commands and what does it do?
• As this, one can get into different contexts where one gets trapped; not blocked
• How to help self to navigate out from here?
• I see this 101s can come to help

Anytime, you picked a Maven project and built it?  For example, a Selenium code written as a Maven project.  Do we know what Maven is and why the Maven project here?  And, what these different Maven command means and what it does?  For having this understanding, this Maven 101 comes to your help.

Maven for Software Test Engineers 101

I have my Maven 101 here.  I refer to this source when I revise my fundamentals.  This can be of help to you as well and also can be a start point.

This has helped me to understand the different build life cycles and how a Maven command is all about the plugins.

I have provided the Reference section which is also a credit that I give them for helping me to unlearn and learn better.

When to Start the Automation in Software Testing?

The Question of a Decade?

Today, when I draft this post, the calendar date is 14th February 2023.  If I look back to 10 years ago and ask myself what are the questions in and around Software Testing and Automation, I see this question.  What is that question?

When to start the automation?

We answer, hear, read, and discuss this question, today too!

Often, the opinion that comes out is, ".... to automate when it is stable".  Note that it is an opinion, not an answer or a fact accepted universally.

To Start Automation when it is "Stable"!?

My learning is,

Do not think of starting the automation when it is "stable".

The "stable" is an assumption we tend to believe by the outcome of using the system.  The binaries are never "stable".

The binaries appear to not show any risks and problems for the way one is using it in a context.  To be more precise, we are not seeing the risks and problems that binaries are showing us in other dimensions.  That is, the dimensions that we are not aware of or the dimensions that we are not focusing on.

When to Start the Automation?

I learn,

Start the automation, when the system is testable!

This leads to me the questions:

• What is testable?
• When it is testable?

Understanding testability helps me to learn and identify its child attribute -- automatability.  That is, understanding testability helps me to learn the order of "testable".

Testability does not mean "stable".  Testable does not mean "stable".

But the assumption "stable" means there are some characteristics of testability, automatability, and order of testable.

Automate when you learn, it is testable, and identify a layer of testability.  This helps to pick the better seam [that is the appropriate layer(s)] for automation in a given context.

Keep the automation structure ready, so that intent of a test can be expressed via code as we identify [a layer of] testability.

Maybe, this is what people say LEFT or SHIFT LEFT or START LEFT.  Or, could be out of the SHIFT LEFT BOX!

Testability Revisited

 

I read the below question on The Test Chat's Telegram group.

When you start working on a project, what steps do you take to establish the testability of the product?

This question is helpful in learning how we see the Testability of a product.  It is a common perception to see the product with testability and then to test the product using the testability.

But, in reality, the testability is associated more with the tests; the tests which are used to test a product being developed or developed.

So, when we talk about testability, we need to be more aware of the test that we will be designing to test the software.  This test should be quick and easy to execute with the help of the programmed or available testability factors and their attributes.

You can find more blog posts in and around testability here in Testing Garage.  Testability in software engineering and systems is one of my research areas.

Testability

I understand Testability as

• How easy it is easy to test by an engineer
• In a given context and skills of an engineer
• With the being used test approach and strategy

Note: The context can keep including factors as we add more and continue to test

It is not about if one can test the software or not.  It is all about software that is easily tested.  How easy?  That is one of the testability factors in software design and programming.

Test and Testability

Unless I know the test, I will not be certain about the Testability.  Testability does not drive tests.  It aids the execution of the test and it is a heuristic.  If the test is designed well to the context and if the testability is used well in the test's context, the execution of a test can be quick and easy.

The tests

• make use of available Testability
• helps to strengthen the Testability
• add more Testability in different seams/layers of the engineering and product

From here it will be two ways; the tests and testability will complement each other.  Further, it leads to developing and including more specific and deterministic tests and testability types in respective seam/layers.

Testability and Automatability

Testability can be classified further into several categories.  Based on the purpose and what to evaluate we will have to identify Testability in respective categories and need to use it.

As a software engineer one is bound to think testability with software programming and infrastructure.  But, testability in software engineering is not just bound to software programming.

The testability is diverse and available across engineering activities.  It is used in all engineering activities.  Maybe, for a software engineer who is hands-on with programming and testing, they infer Testability most times with programming and infrastructure.

I see, the Testability always exists to an extent.  But, can it be identified and used in the way I approach, design, execute and evaluate my test?  That is the point to explore.

If it is testable to some extent, then we are using some Testability attribute(s).

If there is Testability in a seam/layer, then there is an Automatbility in that seam/layer to an extent.

If it is automatable then there is some attribute of Testability in that seam/layer.  Again, the question comes to knowing and learning -- What am I testing and automating? Why? How? When? Where?

This discovers seams/layers to test and automate.  It leads to identifying the tests.  Then, to identify and build more Testability and Automatability.

A written program feature essentially will have an automatable characteristic and space.  If it is automatable, then it is making use of and extending the testability.

In summary,

• Know the test to know and identify the Testability better
• Know the Testability to automate better
• Know the Automatability to assist your testing better.

Context-Free Questions to Identify Testability

To know and have better Testability, here are a few things that I will want to know:

1. What is the test?
• What am I testing and what am I supposed to test?
• How is this test designed to learn and evaluate the system?
• What are the data, states, and events that I'm experimenting, exploring, and experiencing as I test this system with help of this test?
• How can I make this testing quick and easy?
• What should I use to make my testing quick and easy with this test?
• How should I use it to make my testing quick and easy with this test?
• When and where should I use it to make my testing quick and easy with this test?
2. Why am I testing this?
3. What happens if I test this and do not test this?
4. What is the value loss I will incur if not tested?
5. What is the value loss I will incur if I do not understand and learn the outcome of the test?
6. What changes the dimensions of my tests?
7. How can I learn the product better from this test?
8. What information am I learning from this test?
9. What information, heuristics, and Oracle help me and stakeholders to analyze and decide better?
10. Do I actually know the product from the perspectives of
• tech
• business
• user
• risks
• problems
• protocols
• guidelines
• environment
• money
• benefits
• exploitations
• team developing it
• and, more that I can add to the context of the product and project

To summarize, know the test and know how the test is designed.  It helps to identify better testability at the right layer/seam of the software system and engineering.  If there is no effective testability at that seam/layer, it helps to build one.  That way, the automatability also gets built in that seam/layer if the team collaborates well.

WebDriver: Tracing the Interface WebDriver - Part 2

 

In the previous post of this WebDriver series, I shared a gist about what WebDriver does and how.  In this blog post as Part 2 of this series, I'm sharing a bit more details on WebDriver and RemoteWebDriver.  

From there, we will see how AppiumDriver is related to WebDriver -- which extends the interface SearchContext.

This blog post is written as part of 21Days21Tips from The Test Chat.  The tip shared in this post is to know more about WebDriver internals and how it associates with RemoteWebDriver and AppiumDriver.

This should help in understanding the Selenium APIs better and from where it comes.  This helps in having a better mental model of the Selenium WebDriver and how we want to structure the instructions in the tests and utilities we write

SearchContext and WebDriver

Picture: Representation of SearchContext and hierarchy of WebDriver

• The SearchContext is the parent interface in the WebDriver hierarchy
• The subinterfaces of SearchContext are
• WebDriver
• WebElement
• This SearchContext defines two methods
• findElement(By by)
• Modifier and Type is: WebElement
• It finds the first WebElement using the given method
• findElements(By by)
• Modifier and Type is: java.util.List<WebElement>
• It finds all elements within the current context using the given mechanism
• Note: I'm referring to Java APIs of Selenium in this blog post
• More details of this can be found here.

Note: Selenium's Ruby client describes the Interface SearchContext as this.  

The WebDriver provides the below methods:

• close()
• findElement(By by)
• findElements(By by)
• get(java.lang.String url)
• getCurrentUrl()
• getPageSource()
• getTitle()
• getWindowHandle()
• getWindowHandles()
• manage()
• navigate()
• quit()
• switchTo()

More details of these methods can be found here.

RemoteWebDriver and AppiumDriver

Further, we see the class RemoteWebDriver implements the interface WebDriver.  Today, the WebDriver and RemoteWebDriver communicate using standard W3C specifications.

That way, all the modern browser which adheres to W3C specification should not have (much) trouble when using WebDriver and RemoteWebDriver to mimic the user action on them.  We see the ChromiumDriver(), ChromeDriver(), FirefoxDriver(), Edgedriver, SafariDriver(), and OperaDriver() extending the RemoteWebDriver.

This hints us to know and learn:

1. Why do we initiate the WebDriver for first
2. And, then we instantiate the browser's driver
3. Later how we use WebDriver's instantiation to drive action (mimic the user action) on the browser using the respective browser's driver

When we want to automate using Selenium Grid, we make use of RemoteWebDriver to drive the action between the client and server.

The class AppiumDriver extends the WebElement and RemoteWebDriver from the project Selenium.  And further, it has its own methods to interact with the mobile elements.  More details about the Java Client of AppiumDriver can be found here.

The subclasses of AppiumDrivers are:

• AndroidDriver
• iOSDriver
• WindowsDriver

21 Days 21 Tips -- #day17

Here are my pointers to fellow test engineers

1. Interface SearchContext is top in the hierarchy of the WebDriver interface
2. Interface SearchContext defines
• Should I want to search for the element in the whole page
• using WebDriver object
• Or, should I search within a containing element
• using WebElement object
• We can notice methods returning the type WebElement
3. RemoteWebDriver implements the interface WebDriver
4. The modern browsers drivers extends the class RemoteWebDriver
5. AppiumDriver extends the class RemoteWebDriver and interface WebElement
• AndroidDriver extends AppiumDriver
• iOSDriver extends AppiumDriver
• WindowsDriver extends WindowsDriver

For more understanding of the SearchContext and WebDriver, refer to below git repository of SeleniumHQ:

• SeleniumHQ Repository -- https://github.com/SeleniumHQ/selenium
• Selenium Java Client
• SearchContext
• https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/SearchContext.java
• WebDriver
• https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/WebDriver.java
• RemoteWebDriver
• https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/remote/RemoteWebDriver.java
• ChromiumDriver
• https://github.com/SeleniumHQ/selenium/blob/trunk/java/src/org/openqa/selenium/chromium/ChromiumDriver.java

The below understanding should give a mental model of how the call happens in Selenium's library:

• WebDriver and browser's driver instantiation
• The order in which it is instantiated and used in programming to automate actions on the  browser

If noticed, the automation we do is more of programming and not of Selenium's library.  We extend and implement the Selenium library in our programming to mimic the action on the browsers and mobile apps.

Testability: More About it from the Programming Literature

 

My friend Parimala Hariprasad gifted me the book Essential Skills for The Agile Developer, authored by Alan Shalloway, Scott Bain, Ken Pugh, and Amir Kolsky.  Thank you, Parimala, for gifting this book.  I'm experiencing the value of this book and using it.

In this post, I'm sharing the content shared in Chapter 3 of this book. It is about Testability and how it improves the code quality.  

Why this Blog Post?

I continue to read Software Testing literature.  I understand the below as one of the primary key skills for a Software Test Engineer practice:

• Identifying the Testability attribute in the system
• Mapping and classifying how the available Testability attribute can be used in Tests
• Asking for the Testability attribute

With that, I understand "how easy it is to test by a test engineer in a given context" as Testability.  If noticed, this is from Software Testing literature.  And, I see it has these three elements which tell the prominence of each:

• How easy it is to test?
• what factors make it easy to test?
• how does it make it easier?
• how does it bring the deterministic character?
• how can I isolate the observations with my analysis with the help of deterministic character and aid added?
• By a Test Engineer's
• awareness, experience, learning, applying the skills, and more
• In a given context
• time, people, environment, availability, and more

If any of these three elements has trouble, it has its effects on the test and testing.  If you ask what effects, I don't know.  If I pick from my case to share one of the effects, I say, I was not very sure what was happening though the product looked to do what is expected.  But will it continue to do what is expected to do and in what all ways? I had no answer for in what all ways and in what contexts. This is one such case of how the absence or not using the Testability can influence the tester to be unsure about the learning made with help of a test.

The book I mentioned here gives another perspective from the Computer Programming literature.  It talks at the fundamental level and I see this is important to understand for we Test Engineers.  Soon in the coming days, we Test Engineers will be working and testing in these layers of product development. 

In the next section, I will share the lines from the book as is in italics and blue font color word.  The credits are to the authors of this book. I'm taking the text as it is from this book.  And, I will share my interpretation for the same and see the relativity of Computer Programming and Software Testing literature.  

Note: The credit is to James Bach for the Testability definition used above.  I added "the tester and context" to it as these two influence the Testability and outcome of using the Testability to a greater extent.

Testability and Code Quality

The authors of the book say, "testability is highly correlated to the code qualities we want to manifest, in particular, loose coupling, strong cohesion, and no redundancy."  Further, they illustrate how one remarks at the start of testing one's code by saying the below:

I can't test this code; it does too many things that are so interwined -- weak cohesion

I can't test this code without access to dozens of other things -- excessive cohesion

I can't test this code; it's been copied all over the place, and my tests will have to be duplicated over and over again -- redundancy

I can't test this code; there are too many ways for external objects to change its internal state -- lack of encapsulation

Then I read this line from the authors, "Gee, I wish they had thought of how this code was going to be tested while they were writing it!".  That's a question that every one of us has to ask ourselves for the work we deliver and not just for the programming.  

Alan Shalloway says he is kind of slow sometimes because it took him some time to realize this -- I should consider how my code is going to be tested before writing it! 

Testability is related to loose coupling, strong cohesion, no redundancy, and proper encapsulation.  Another way to say this is:

• the tighter your coupling, the weaker your cohesion; 
• the more your redundancy and the weaker your encapsulation, the harder it will be to test your code

Therefore, making your code easier to test will result in the loose coupling, strong cohesion, less redundancy, and better encapsulation.  This leads to a new principle -- Considering how to test your code before you write it is a kind of design.

Since testability results in so many good code qualities and since it is done before you write your code, it is a very highly leveraged action.  That is, a little work goes a long way; it is a great trim tab.

I and Testability

I try to understand and learn about Testability every day in my practice.  When I started my career 15 years back, I learned from my network, that one of our fellow testers in the community that is Meeta Prakash did her Ph.D. in Testability.  I wanted and still want to read the thesis of Meeta Prakash.  I hope she will find it and give me soon, one day.  In those days, I referred to the slides of James Bach on RST; that legacy slides that had contents filled with blue color. 

From there, I tried looking into the testability in what I test and what programmers deliver to me.  When I worked with Moolya in 2012, I realized from my practice -- context and the skill sets of a tester matter to make use of the available testability and to identify if it is present or not, and to what extent. I added this to the definition of James Bach and I shared the same with my fellow testers with whom I was mentoring and working together.

Relating the Literature and Interpretation

When the programming is talking about testability, I see it is talking about:

• internal aspects of how it is programmed and to test the same easily in isolation, and in integration for the context while being deterministic

The words used to express in programming literature are more programming oriented.  Whereas, what we see in the Software Testing literature, it is more of a common man's words.  But, what both means is the same and the difference between them is to which layer and aspect they are referring and how, and why.

The Weak Cohesion

• It will be an obvious experience to a tester when it is difficult to speculate and pull a particular observation with more information for a feature or a user flow
• For example, if the Refresh Token is used along with Auth Token everywhere, then it will be tough to isolate when Refresh Token is used and when the Auth Token is used

I feel the same when wanting to test a piece of code in isolation from other code.  I have experienced this when testing one aspect of utility or a complete utility in isolation from the rest of the automation code.

The Excessive Cohesion

• I could not test the mobile apps as I needed data
• Certain data came from a portal that is also under development and depends on APIs to work
• APIs would be under development till the last day of release and did not deliver the endpoints to the portal and mobile apps team

So how could the test team create data to test for mobile apps, web portal, and for APIs themselves?  If you see this is excessive cohesion at the product development level.

The Redundancy

• In one project, I had to login each time to see the status of a session
• All tests were programmed in a way that I should login each time
• The test team used the login function in every test and it was duplicated
• When signed in, the Auth token got changed which lead to difficulty in debugging and isolating the problem
  

This complicated the test code and also messed up debugging.  The tests could not be deterministic here.

I see a static Auth token or one-time login and using the same Auth token in all other tests in the suite could have helped to debug the problem and where it occurred.

The Lack of Encapsulation

• My team had a tough time when started to use an existing automation
• It had public access modifier for all methods in all packages
• The team picked up and authored more tests that changed the data and states
• This led to any object of a method to modify the data or state; it was not supposed to be modified at all
• The debugging led us here and it was not a problem with the product
• It was the problem with the product's automation code and how the tests changed data and state;  it was in turn used in other tests

This led to much more chaos as the automation and testing environment were the same.  The invalid bugs, meetings that got scheduled to discuss and time went into the meeting that ended with no use, and a couple of releases came into a decision should we deploy or not, and more.

Continuing the Unlearning and Learning of Testability

If you see, Testability has got multi-dimensions in the dynamics of software development.  Testability is not just about Programming and Testing.  It can be from the environment, project, people, what we understand and how we use it further in work, and the business itself.

I continue to unlearn and learn testability every day as I practice testing and automation.