Testability Revisited

 

I read the below question on The Test Chat's Telegram group.

When you start working on a project, what steps do you take to establish the testability of the product?

This question is helpful in learning how we see the Testability of a product.  It is a common perception to see the product with testability and then to test the product using the testability.

But, in reality, the testability is associated more with the tests; the tests which are used to test a product being developed or developed.

So, when we talk about testability, we need to be more aware of the test that we will be designing to test the software.  This test should be quick and easy to execute with the help of the programmed or available testability factors and their attributes.

You can find more blog posts in and around testability here in Testing Garage.  Testability in software engineering and systems is one of my research areas.

Testability

I understand Testability as

• How easy it is easy to test by an engineer
• In a given context and skills of an engineer
• With the being used test approach and strategy

Note: The context can keep including factors as we add more and continue to test

It is not about if one can test the software or not.  It is all about software that is easily tested.  How easy?  That is one of the testability factors in software design and programming.

Test and Testability

Unless I know the test, I will not be certain about the Testability.  Testability does not drive tests.  It aids the execution of the test and it is a heuristic.  If the test is designed well to the context and if the testability is used well in the test's context, the execution of a test can be quick and easy.

The tests

• make use of available Testability
• helps to strengthen the Testability
• add more Testability in different seams/layers of the engineering and product

From here it will be two ways; the tests and testability will complement each other.  Further, it leads to developing and including more specific and deterministic tests and testability types in respective seam/layers.

Testability and Automatability

Testability can be classified further into several categories.  Based on the purpose and what to evaluate we will have to identify Testability in respective categories and need to use it.

As a software engineer one is bound to think testability with software programming and infrastructure.  But, testability in software engineering is not just bound to software programming.

The testability is diverse and available across engineering activities.  It is used in all engineering activities.  Maybe, for a software engineer who is hands-on with programming and testing, they infer Testability most times with programming and infrastructure.

I see, the Testability always exists to an extent.  But, can it be identified and used in the way I approach, design, execute and evaluate my test?  That is the point to explore.

If it is testable to some extent, then we are using some Testability attribute(s).

If there is Testability in a seam/layer, then there is an Automatbility in that seam/layer to an extent.

If it is automatable then there is some attribute of Testability in that seam/layer.  Again, the question comes to knowing and learning -- What am I testing and automating? Why? How? When? Where?

This discovers seams/layers to test and automate.  It leads to identifying the tests.  Then, to identify and build more Testability and Automatability.

A written program feature essentially will have an automatable characteristic and space.  If it is automatable, then it is making use of and extending the testability.

In summary,

• Know the test to know and identify the Testability better
• Know the Testability to automate better
• Know the Automatability to assist your testing better.

Context-Free Questions to Identify Testability

To know and have better Testability, here are a few things that I will want to know:

1. What is the test?
• What am I testing and what am I supposed to test?
• How is this test designed to learn and evaluate the system?
• What are the data, states, and events that I'm experimenting, exploring, and experiencing as I test this system with help of this test?
• How can I make this testing quick and easy?
• What should I use to make my testing quick and easy with this test?
• How should I use it to make my testing quick and easy with this test?
• When and where should I use it to make my testing quick and easy with this test?
2. Why am I testing this?
3. What happens if I test this and do not test this?
4. What is the value loss I will incur if not tested?
5. What is the value loss I will incur if I do not understand and learn the outcome of the test?
6. What changes the dimensions of my tests?
7. How can I learn the product better from this test?
8. What information am I learning from this test?
9. What information, heuristics, and Oracle help me and stakeholders to analyze and decide better?
10. Do I actually know the product from the perspectives of
• tech
• business
• user
• risks
• problems
• protocols
• guidelines
• environment
• money
• benefits
• exploitations
• team developing it
• and, more that I can add to the context of the product and project

To summarize, know the test and know how the test is designed.  It helps to identify better testability at the right layer/seam of the software system and engineering.  If there is no effective testability at that seam/layer, it helps to build one.  That way, the automatability also gets built in that seam/layer if the team collaborates well.

The Testability and a Tester

In one of the recent discussion with fellow testers, the topic was "testability".  While fellow testers asked, "what is testability?", I had a question which I shared in the discussion -- "What is codability?".


Codability and Testability

I see both of these are inter related.  If it is codable then it is testable to a degree.  What is that degree? That's the question of interest.  But, what is that one will accomplish by knowing the order of degree, here?  This forms the base in having the curiosity or wish to know about testability.

I have a product's feature which I have to implement. The feature has cases which I need to handle in run time for different situations which a user can encounter. Well, I can code for situations which I can think and which is of priority from point view of using the product. Then for other situations which I'm not seeing or not thought off, will it be a hit for my product?  What is the complexity level of the code I'm writing for this feature and what are the risks that are subtle in code I have written? The written code will always have the risk that is tied along with it.

Now, can I tell that testability is influenced by multiple factors and not just on the basis of the test identified? Well, after knowing this, it is essential to understand -- "testability is the easiness at which I can test the system in a given situation.".


Factors influencing testability

The few factors which will potentially influence the order of  easiness -- in learning the test challenge, identifying the test, approaching the test and executing it, are as below:

• The testing skills of a tester
• The programming skills of a tester
• The technology skills of a tester
• The experience of a tester in testing such systems
• What is know by tester about the product -- purpose of product; people who are building the product; people who will use or who are using the product; the features and functionality of product; technology of the product; code written for the product; how the product is built with limitations from business team; information available about the product; process, practice and culture followed by teams in building product; availability of the system and its present state
• Time available for system development
• Time available for testing
• The level or order to which the testability aid is built and can be provided in system to test, and the complexity or simplicity level of same
• Measures taken to improvise and include testability aiding stuffs in system
• Existing tools or utilities for testing the product or code, or to build one such utilities
• The freedom to tester and testing to control and guide the testing activity
• Information expected out of testing and priority of it
• Relationship with fellow testers, fellow programmers, business teams and any other who are involved and interested in the system being programmed and tested
• The audience interested in test report or outcome of testing and their expectation

Testability and Test Coverage

The test coverage accomplished by a tester in given context is relational to the skills set of tester and the testability factors of the system and environment.

While I have understood this, I tried to learn the scale of testability keeping the test coverage as base. Here is what I have observed for now.

By keeping the test coverage as a base scale might mislead me to learn what I can accomplish in coverage. Because, the testability might be available and easy one for me to accomplish the 10th level of coverage with my skills and other influencing factors of testability. But to cover and accomplish the first three levels can be not the easy one given my skills and other influencing factors of testability. The vice versa is also possible.  Here, I can take longer time to learn and know how could I have cut short and still accomplish the same coverage given the testability level to me in a context, if I had learned about the complexity level of of it. While I reach to a intended level of coverage, I would have done the moves which does not favor business timeline, learned from them and repeated few stuffs.  This is not wrong but I have consumed time which I could have invested in testing to accomplish much more coverage.  The same is represented in below image.

Keeping the test coverage in base and assessing the testability on vertical scale marking the breadth and width of coverage spread out in parallel which  I want accomplish or what I can accomplish, here is what I observe for now:

Understanding the test coverage I want to reach by my testing and automation, so I provide the information expected is a challenge each time.  The reason for simple to quote for first is the 'testability' for that coverage mark.  Once I get an idea of what is the skills that I need to build and use in such situations, it will help me to make strategic decisions in testing execution and its management.  The is very important as per me from point of view testing strategic base.  This is helping me and allows me to test the perceived level of testability itself for marked coverage boundary or milestones.

This base of strategic thinking in learning and identifying the testability can be used in timelines of a project. Not just during the execution of testing, also in pre-execution and post-execution, it can be used. It gives an idea of how the tester has to be equipped in changing needs of testing and engineering. For me this is working in context where I'm exposed and in few cases I had to add few vectors along with test coverage such as technology and programming factors specific. I'm experimenting this in varied projects having different technology and skill challenges.


Visibility out of testability

With this, for now I see,

• Testability is related with codability, programmer, tester, environment, process, people, situation, priorities
• Codability gives the first hint on testability and skills needed to test the same.
• Testability influences the Test Coverage
• Testability influences the time taken by a tester to test the system in a given context
• Like codability requires the skills of a programmer, testability requires skills of a tester

Note: Automation when leveraged with testability, wonders can be done via automation in assisting the testing.

Testability: More About it from the Programming Literature

 

My friend Parimala Hariprasad gifted me the book Essential Skills for The Agile Developer, authored by Alan Shalloway, Scott Bain, Ken Pugh, and Amir Kolsky.  Thank you, Parimala, for gifting this book.  I'm experiencing the value of this book and using it.

In this post, I'm sharing the content shared in Chapter 3 of this book. It is about Testability and how it improves the code quality.  

Why this Blog Post?

I continue to read Software Testing literature.  I understand the below as one of the primary key skills for a Software Test Engineer practice:

• Identifying the Testability attribute in the system
• Mapping and classifying how the available Testability attribute can be used in Tests
• Asking for the Testability attribute

With that, I understand "how easy it is to test by a test engineer in a given context" as Testability.  If noticed, this is from Software Testing literature.  And, I see it has these three elements which tell the prominence of each:

• How easy it is to test?
• what factors make it easy to test?
• how does it make it easier?
• how does it bring the deterministic character?
• how can I isolate the observations with my analysis with the help of deterministic character and aid added?
• By a Test Engineer's
• awareness, experience, learning, applying the skills, and more
• In a given context
• time, people, environment, availability, and more

If any of these three elements has trouble, it has its effects on the test and testing.  If you ask what effects, I don't know.  If I pick from my case to share one of the effects, I say, I was not very sure what was happening though the product looked to do what is expected.  But will it continue to do what is expected to do and in what all ways? I had no answer for in what all ways and in what contexts. This is one such case of how the absence or not using the Testability can influence the tester to be unsure about the learning made with help of a test.

The book I mentioned here gives another perspective from the Computer Programming literature.  It talks at the fundamental level and I see this is important to understand for we Test Engineers.  Soon in the coming days, we Test Engineers will be working and testing in these layers of product development. 

In the next section, I will share the lines from the book as is in italics and blue font color word.  The credits are to the authors of this book. I'm taking the text as it is from this book.  And, I will share my interpretation for the same and see the relativity of Computer Programming and Software Testing literature.  

Note: The credit is to James Bach for the Testability definition used above.  I added "the tester and context" to it as these two influence the Testability and outcome of using the Testability to a greater extent.

Testability and Code Quality

The authors of the book say, "testability is highly correlated to the code qualities we want to manifest, in particular, loose coupling, strong cohesion, and no redundancy."  Further, they illustrate how one remarks at the start of testing one's code by saying the below:

I can't test this code; it does too many things that are so interwined -- weak cohesion

I can't test this code without access to dozens of other things -- excessive cohesion

I can't test this code; it's been copied all over the place, and my tests will have to be duplicated over and over again -- redundancy

I can't test this code; there are too many ways for external objects to change its internal state -- lack of encapsulation

Then I read this line from the authors, "Gee, I wish they had thought of how this code was going to be tested while they were writing it!".  That's a question that every one of us has to ask ourselves for the work we deliver and not just for the programming.  

Alan Shalloway says he is kind of slow sometimes because it took him some time to realize this -- I should consider how my code is going to be tested before writing it! 

Testability is related to loose coupling, strong cohesion, no redundancy, and proper encapsulation.  Another way to say this is:

• the tighter your coupling, the weaker your cohesion; 
• the more your redundancy and the weaker your encapsulation, the harder it will be to test your code

Therefore, making your code easier to test will result in the loose coupling, strong cohesion, less redundancy, and better encapsulation.  This leads to a new principle -- Considering how to test your code before you write it is a kind of design.

Since testability results in so many good code qualities and since it is done before you write your code, it is a very highly leveraged action.  That is, a little work goes a long way; it is a great trim tab.

I and Testability

I try to understand and learn about Testability every day in my practice.  When I started my career 15 years back, I learned from my network, that one of our fellow testers in the community that is Meeta Prakash did her Ph.D. in Testability.  I wanted and still want to read the thesis of Meeta Prakash.  I hope she will find it and give me soon, one day.  In those days, I referred to the slides of James Bach on RST; that legacy slides that had contents filled with blue color. 

From there, I tried looking into the testability in what I test and what programmers deliver to me.  When I worked with Moolya in 2012, I realized from my practice -- context and the skill sets of a tester matter to make use of the available testability and to identify if it is present or not, and to what extent. I added this to the definition of James Bach and I shared the same with my fellow testers with whom I was mentoring and working together.

Relating the Literature and Interpretation

When the programming is talking about testability, I see it is talking about:

• internal aspects of how it is programmed and to test the same easily in isolation, and in integration for the context while being deterministic

The words used to express in programming literature are more programming oriented.  Whereas, what we see in the Software Testing literature, it is more of a common man's words.  But, what both means is the same and the difference between them is to which layer and aspect they are referring and how, and why.

The Weak Cohesion

• It will be an obvious experience to a tester when it is difficult to speculate and pull a particular observation with more information for a feature or a user flow
• For example, if the Refresh Token is used along with Auth Token everywhere, then it will be tough to isolate when Refresh Token is used and when the Auth Token is used

I feel the same when wanting to test a piece of code in isolation from other code.  I have experienced this when testing one aspect of utility or a complete utility in isolation from the rest of the automation code.

The Excessive Cohesion

• I could not test the mobile apps as I needed data
• Certain data came from a portal that is also under development and depends on APIs to work
• APIs would be under development till the last day of release and did not deliver the endpoints to the portal and mobile apps team

So how could the test team create data to test for mobile apps, web portal, and for APIs themselves?  If you see this is excessive cohesion at the product development level.

The Redundancy

• In one project, I had to login each time to see the status of a session
• All tests were programmed in a way that I should login each time
• The test team used the login function in every test and it was duplicated
• When signed in, the Auth token got changed which lead to difficulty in debugging and isolating the problem
  

This complicated the test code and also messed up debugging.  The tests could not be deterministic here.

I see a static Auth token or one-time login and using the same Auth token in all other tests in the suite could have helped to debug the problem and where it occurred.

The Lack of Encapsulation

• My team had a tough time when started to use an existing automation
• It had public access modifier for all methods in all packages
• The team picked up and authored more tests that changed the data and states
• This led to any object of a method to modify the data or state; it was not supposed to be modified at all
• The debugging led us here and it was not a problem with the product
• It was the problem with the product's automation code and how the tests changed data and state;  it was in turn used in other tests

This led to much more chaos as the automation and testing environment were the same.  The invalid bugs, meetings that got scheduled to discuss and time went into the meeting that ended with no use, and a couple of releases came into a decision should we deploy or not, and more.

Continuing the Unlearning and Learning of Testability

If you see, Testability has got multi-dimensions in the dynamics of software development.  Testability is not just about Programming and Testing.  It can be from the environment, project, people, what we understand and how we use it further in work, and the business itself.

I continue to unlearn and learn testability every day as I practice testing and automation. 

When to Start the Automation in Software Testing?

The Question of a Decade?

Today, when I draft this post, the calendar date is 14th February 2023.  If I look back to 10 years ago and ask myself what are the questions in and around Software Testing and Automation, I see this question.  What is that question?

When to start the automation?

We answer, hear, read, and discuss this question, today too!

Often, the opinion that comes out is, ".... to automate when it is stable".  Note that it is an opinion, not an answer or a fact accepted universally.

To Start Automation when it is "Stable"!?

My learning is,

Do not think of starting the automation when it is "stable".

The "stable" is an assumption we tend to believe by the outcome of using the system.  The binaries are never "stable".

The binaries appear to not show any risks and problems for the way one is using it in a context.  To be more precise, we are not seeing the risks and problems that binaries are showing us in other dimensions.  That is, the dimensions that we are not aware of or the dimensions that we are not focusing on.

When to Start the Automation?

I learn,

Start the automation, when the system is testable!

This leads to me the questions:

• What is testable?
• When it is testable?

Understanding testability helps me to learn and identify its child attribute -- automatability.  That is, understanding testability helps me to learn the order of "testable".

Testability does not mean "stable".  Testable does not mean "stable".

But the assumption "stable" means there are some characteristics of testability, automatability, and order of testable.

Automate when you learn, it is testable, and identify a layer of testability.  This helps to pick the better seam [that is the appropriate layer(s)] for automation in a given context.

Keep the automation structure ready, so that intent of a test can be expressed via code as we identify [a layer of] testability.

Maybe, this is what people say LEFT or SHIFT LEFT or START LEFT.  Or, could be out of the SHIFT LEFT BOX!

Architecture: The Common Shared Understanding -- Part 1

 

When we are developing a software system, the requirement from a stakeholder is not 'Fast' or 'Scalable'  or  'Responsive'.  But, the stakeholder needs it and expects it.  If you see, on a larger picture, the software system development and maintenance is a job of balancing too.

When a Software Architect [Technical Architect and Test Architect], works on architecting the software system and testing for the same,

• It is about balancing the technical aspects with the business's requirements from stakeholders.  
• Do you see that?

Knowing the architecture of a software system and testing of same is one of a primary task for engineers on the project.  Because, we software engineers have to balance it well.  Balance, what? Balancing the technical aspects together with business's requirements from stakeholders.

This blog post is part of 100 Days of Skilled Testing series.  The second question posted for season 2 is,

How important is the understanding of application architecture to do performance testing better?

 

What is an Architecture?

In context of Software Development & Engineering, the word "architecture" is one of the ambiguous words among the teams in a project and an organization. 

As a test engineer,

• Did you ever had a discussion or arguments or debate with programmer and architect?
• I had such discussions and I continue to have it today as well in the projects that I work.
• This is to know and understand
• What I should be doing as an engineer for first and as a Test Architect in the role?

The outcome of this discussion showed me,

• We all did not have a common understanding of it
• We did not share,
• "What I understood for the architecture and this architecture?"

The primary goal of a Software System's Architecture is,

• We all engineers on a project have a same understanding of it, in the aspects it exists for.
• This understanding is arrived after we have put our thoughts into scrutiny and decided that we stick on to it, so that,
• We can balance well between the technical requirements and business expectation.

Are you with me, so far?

A Software System's Architecture is,

1. A common shared understanding of what we all have for,
•  What we are developing, testing, and to about maintain?
• And, Why? Who? When? Where? How?
2. Represents the boundaries and interfaces of what matters,
• That is [to be] orchestrated, designed, implemented, how it communicates, and what it will have, and not.
• It also can show how the teams are structured and how the team and organization is organized.
• For example, in the Service Oriented Architecture, the teams are built and structured with respect to the service they offer.
• It is a model that is better than other models in a given context of technical requirements and business needs
3. The context and awareness for,
• Why it is the way it is?
• The cost and value for being so.
• What to do when it has to be changed? Where to change?
• How simple and quick to change?
• What are the cost and value for being so?
• How can I monitor and observe all these consistently?
4. The Gateway of Testability - it tells what is the Testability available for,
• Letting know what is critical and priority to test
• Where, How, When, and What tests can be framed, designed, and executed? To what extent?
• Why these tests?
• If an architecture does not talk about and do not have the Testability, we have a serious problem!  This has to be fixed for first on priority.
• An architecture has to provide the Testability and Programmability scope and opportunities to develop a software system that is of value!

For today, this is my understanding for the "Architecture".

I'm a Test Architect in the role and I expect myself to be an hands-on engineer for first.  It is a necessity for an architect to be an hands-on engineer.

Note: Read the Part 2 of this blog post here.

Testers' Arena, will testing challenge help to voice the opinion than just a blog post?

I'm writing this post as a request to STeP-IN Forum and Testers' Arena contest's judging panel. What I'm writing is my opinion and does not indicate any other person's opinion. I want to share this so I can share the perspectives of how the Testers' Arena Contest 9 task can also be seen and possible effects of it.



To,
Respected Judging Panel of Testers' Arena Contest and STeP-IN Forum,

I have respects for what you are giving to us (Software Testing community) via Testers' Arena contests. You are helping us to see lot other testers and their thoughts, which is very much required for a practicing tester.

With the Testers' Arena contest 9's task, I do not know why you decided to give the task statement as below and it surprised me. And the same I have mentioned in this post.

Read the links at http://www.huibschoots.nl/wordpress/?page_id=1771 and come up with your blogpost highlighting why ISO 29119 standard is bad for software testing.

The blog can be on your website, blogger or Wordpress. Share the link for evaluation.

In a view, to me it looked as, the judging panel had already decided that "it is bad" and was looking to hear the voice of testers to support that opinion.  If so, why? You are asking us to prove than asking us to come out with analysis, reasoning, investigation and produce information to you with context and say, in this contexts the ISO 29119 appears to be needed or useful or not needed or not useful or uncertain or good for system or not good for system.

I did not sign the petition ISO 29119 because, I'm not yet sure what is inside ISO 29119 nor I spoke to WG26 team. Also, I do not support ISO 29119, as I believe its testability has be to strengthened for the present context around it and this is mentioned in below blog posts.

1. Hot Cold Topic "Standards" : Building the Stand -- Part 1
• http://testinggarage.blogspot.in/2014/09/hot-cold-topic-standards-building-stand.html
2. Hot Cold Topic "Standards" : The word 'Testing' between - International, Society, Organization, Standards and I -- Part 2
• http://testinggarage.blogspot.in/2014/09/hot-cold-topic-standards-word-testing.html
3. Hot Cold Topic "Standards" : I'm not a Judge; I'm an advocating Tester -- Part 3
• http://testinggarage.blogspot.in/2014/09/hot-cold-topic-standards-im-not-judge.html
4. Hot Cold Topic "Standards" : Testability of ISO 29119 and Dr. Stuart Reid replies to petition. -- Part 4
• http://testinggarage.blogspot.in/2014/09/hot-cold-topic-standards-testability-of.html

For the Judging Panel

I wish to say a small story that makes me to think. Hope it also makes you to think.

A mother feeding the food to her kid showed the place to which kid had not been so far in dark. And she said, "That is dangerous place. If you don't have food now, someone will come from dark place and will eat away this food. Sweet heart, eat the food."

The kid replied, "dark place? It is dangerous? It is bad, huh? I will eat my food before danger come." The kid believed the words 'that is dangerous'  Later, the father was playing with kid and he took him to show the stars in the sky. 

Father said, "See the shining stars and I will catch one for you." The kid replied, "It is dark sky and danger. Anything that is dark is danger. I don't want star. The star you catch is danger to me." 

Father did not have any words listening to the kid.

If mother or father or guardian could have said why dark appears to be dangerous but still one has to get into dark to learn if it is true or lie. If not the kid may start to assume and believe it is dangerous. I see the same here. We will be having testers who are starting their career and testers who are working to build their career in testing. But not necessarily, they will challenge or question Contest 9's task to learn why did you as that.

But if said, don't go there at all, few kids might not even go and explore trusting the words said to them. Now, do you want to see any testers as this in practicing?

This is not a mistake at all for having that contest 9's task statement. It's okay, this is how practicing tester learns and I believe judging panel also has the practicing tester. But please consider to get the thoughts, if the motive of the task given is with words "highlight" "good", "bad" and on such serious context happening around it, because it can also be taken as "prove it so". Hope I clarified my thought on this.


For STeP-IN Forum

STeP-IN forum, you are gaining the much more visibility via this contest and you might also agree to this. Now, when you are gaining the visibility, you are also building the credibility for your intention and contribution to Software Testing community. Below are the questions which I have for you with respects.

1. Do you support only one philosophy or one group of testing practice? Or you support the forum of test practitioners from different philosophies of testing practice?
2. If you support forum of software testing practitioners from different philosophies, then people who have come up with draft of ISO 29119 are also from one group? So, you are saying their work or idea or efforts is useless by just giving contest 9's statement as ".... come up with your blogpost highlighting why ISO 29119 standard is bad for software testing."
• Now, you give us a view point how this should be considered if you are the forum for software testing and practitioners.
3. Yes, I believe the judging panel has the rich set of skills and experience in the craft. But do they know the intention of your platform from where the Testers' Arena contest is held?
• Because, the work of both can get questioned or challenged if stuffs such as ' bad for software testing' come out without asking the analysis work and testing challenges to show that.
• Hope you stand impartial to any group or any group of practitioners, if said as Software Testing forum and not in favor of any one group or philosophy of testing.  Software Testing  forum is to all, yes?
4. When you claim as a Forum for Software Testing, can we eliminate anyone who do not fit into one's thought or ideas of testing?
• By doing this, we are keeping away testers who participates in the contests if one believes I should not participate where my thoughts are not respected than discussing. Yes?
• Does this affect the credibility of yours?

Testers' Arena Contest 9 contestants blog post are available here. Giving here is okay but it also raises one more question now. 

1. Is the judging panel a member of ISST?
2. If yes, Testers' Arena contest challenge is being made use for STOP ISO 29119?
3. STeP-IN forum agree for this?

These are few questions which comes straight out into my mind. If you support or don't support it is your choice. But, please let us not misguide the testers by your opinion of good or bad and asking to highlight the points for that. Hope you understand my humble request. 

I care for Software Testers skills which are needed for testing as you care, before any standards or philosophies of testing. I wish, the skill has to be sculptured each time than being mirror to someone else opinion.

If you ask me how is this trouble, please do read the contest's task statement which ends up saying "... highlighting why ISO 29119 standard is bad for software testing." Did STeP-IN forum panel or all board members go through these standards on purchasing it? If not, how can STeP-IN forum's contest come up with such task statement and give the contestants blog post(s) to ISST? I have taken the print of this page on 24th September 2014 1:25 AM IST and it can be found here and the page on ISST can be found here.  

This indicates, probably the judging panel could have the member of ISST and taking the benefits of 9th contest. If you ask me why you think it as benefit, "... highlighting why ISO 29119 standard is bad for software testing." hints me this on seeing the petition STOP ISO 29119.  If the task's challenge was to analyze and come up with reasoning why you think it is good or bad, could have been fairly better.

What if there is another contest or next contest's task is to come up with a blog post highlight why ISO 29119 is useful for testing? What do we see? Any ideas? What should I say, if I have said it is bad for testing, in my previous blog post for Testers' Arena?

One of the ways in which we could have approached

STeP-IN Forum and Testers' Arena judging panel, you could have built beautiful stage (which is not built so far) for discussion of the software testing practitioners groups who support and don't support the standards. The Testers' Arena contests statement having the elements which asks to

• demonstrate and requires the skills which are not mentioned (or probably cannot be mentioned) in standard documents, and 
• demonstrate and requires the information which are mentioned in standards, let us say,

With this, I believe, the supporting group and opposing group has not met on a stage for discussion on ISO 29119. You could have created one by giving the contestants testing work to these two groups --  who supports and don't support ISO 29119. With this you could have initiated for the open discussion on a stage. This would have brought more credibility to you, to the judging panel and to our work that we submitted for Testers' Arena. As well it could have got more testers to participate and know what's happening in Software Testing.


Closing Thought

With this, I have tried to convey what I see. If you feel it is meaningful to your context, kindly pick it up and see what is going on. If you feel, it is not meaningful to your context, ignore and one day we will all will realize it and wish it will not be too late.

Either ways, challenge us with the contest than asking us to highlight or prove. Awaiting for the Testers' Arena Contest 10.

Hot Cold Topic "Standards" : Testability of ISO 29119 and Dr. Stuart Reid replies to petition. -- Part 4

Will the software testing business get affected by the ISO 29119? Before picking up this, the question to be picked is, are there any damage that comes to subject and integrity of practitioners and their groups? If subject exist then the business.

Business and ISO 29119

1. Will the reputed software product companies make use of ISO 29119 standards?
• I don't know. Do they disclose it if did so with case study? At least I wait for it curiously. Will there be any legal problems for this product company for not using ISO 29119 from customer who brought the product saying this could have avoided if standards were used? What will the software product companies say to world?
2. Will the software service companies follow these standards?
• I don't know what legal documents will be signed in between service provider and service seeker. If service seeker brings in ISO 29119 and says it has to be strictly adhered believing it helps her/his product, will the work be highly usable? I don't know what happens. This could be one of the major trouble legally, if the outcome is not acceptable by stakeholder.
• If the service provider then claims, it is because of IS0 29119, then how this will be received by the WG26 and are we ready to accept it?

People asking for solution will come and business runs, but, the legal problem with testing standards will be solved? Hence I feel, there should be a common understanding, acceptance or denial of standards in Software Testing industry from the academic and industry practitioners groups being one family.

Dr. Stuart Reid responds for petition STOP ISO 29119

The reply from Dr. Stuart Reid can be found here. I have respects for WG26 team and for their accomplishments and skills. Likewise, I have respects for people or group who are opposing ISO 29119, for their accomplishments and skills. But this does not solve the problem. Forget about solving the problem, it will not help in learning the problem itself.

Dr. Stuart Reid, says

1. WG26 team is well skilled and have rich experience.
2. Six years have been waited to get the consensus.
1. I feel,
• But, why people did not start to oppose as this six years back? Or if started why it was not highlighted then, as now?
• This gives the hint of, probably not all were aware and hence it did not get the opposing traction?
• With this, I get the question, did practitioners not bothered to know then? And now it is known, because of Social Media, advancement in use of web,  than the conferences? 
• If yes, ISO you can see where to make the consensus process public and invite for the review of it.
3. He himself wish and want to give all standards for free, but he has no enough power.
• I feel,
• Dr. Stuart Reid, you can still influence ISO for this to make ISO 29119 free till the Software Testing industry, academic and practitioners groups, come to an common opinion what should be in standards and how it should be, it at all if standards is required.
4. Any activities in ISO 29119 is not politicized to out rule other player in industry.
5. Changes to standards will be done consistently based on the feedback received from the use of it.
6. He says, any standard cannot be mandated as compulsory unless it is enforced by the organization. 
• I hope, by the 'organization' mean, the organization which makes use of the ISO standards for the business.
7. Testing standards is for testers who want see a definition of good practice and to see how close or far they are from it. They are free to tailor it to their context on seeing this.
8. There is no certification scheme associated with Testing Standards ISO 29119. No where  the ISO/IEC/IEEE testing standards is linked with ISTQB certification scheme. If ISTQB adheres to ISO 29119, it is decision left to ISTQB.
9. Exploratory Testing is explicitly included as a valid approach to testing in testing standards.
10. Testers around the world were invited to take part in testing standards development and it is presented in the conferences.
• I feel,
• This should reach to people who work using standards than just saying it for people who manages knowing what is standards.
11. There are number of options for people outside the WG26 to participate and provide feedback.
• I feel,
• This needs to be publicized by ISO very much and by WG26 team.
• Should give the invitation respectfully to group or practitioner if one has different thoughts.
12. Agrees with 7 principles of Context Driven Testing. But unhappy when testers not belonging to CDT practicing group are assigned to deprecated schools by creating one.
13. Jon Hagar supporter of CDT and with other WG26 members ensured many of CDT perspectives to be considered.
14. Has no problem in following risk based and context driven approach simultaneously.

What should I say now after reading these words of Dr. Stuart Reid? Where is the problem? This again confuses for tester as me who has not read the draft content in detail, not spoken to WG26 members, and reading the tweets, FB sharing, blog posts which 

• opposes the standards (just) saying it affects the craft and subject, Software Testing. How it affects? Why it affects? To whom it affects?
• supports the standards (just) saying it helps the craft, Software Testing. How it helps? Why it helps? To whom it helps? When it will help?

Now should I say ISO 29119 as good or bad?  Until, there is a common stage where Software Testing's academic and practitioners groups come forward and discuss for betterment of subject, I hope this will not be solved. I have taken the print of Dr. Stuart Reid response and it is here.

ISO 29119, Good or Bad ?

After learning and expressing what I'm seeing in Part 1, Part 2, Part 3 and Part 4 (this post), I see Software Testing ISO 29119 standards is not yet reached to place where it can be considered for evaluating it further, so the interested people can take decision of it is good or bad. 

It has much more scope to get stronger in terms of testability. Yes, testability of ISO 29119 has to be improved to study if it is required or not and if required what it should have in it.  How to get it here? Working together and supporting each others in expressing and by leaving as one family.

Does a picture speaks all 4 parts of blog posts?

Different interpretations of "What a standard is?"
Photo Credit: @standardsforum

Concluding Part 4

1. If the skill is so much emphasized in Software Testing, I want to be skilled than repeating or tweeting or agreeing to others opinion.
• Hence, in the time I had, I have tried to analyze by learning what is standard, governing bodies of standards, and with available content of ISO 29119
2. As a tester, I have not said it is good or bad. I have said, it is not yet in testable state if you are expecting me to provide the information to help you in making the decision it is good or bad.
• When it reaches to the testable state?
• Refer the previous section of this blog post.
• What factors are not strengthening the testability of ISO 29119?
• Refer the blog post series Part 1, Part 2, Part 3 and Part 4, 

-- End of Part 4 --

-- End --

The Mindset to Embrace and Encourage Automation in Software Testing

 

I was brainstorming about the automation practice setup. Before getting technical, I was looking at the mindset needed. Besides this, I was with a mentee Chidambara and answering the questions.

I'm sharing the gist of pointers that we discussed and which are essential for having the encouraging mindset towards getting started and value adding automation in testing.

 

• Build the culture that supports and encourages the Testing and Automation

• Communicate and Collaborate

  • With testers, developers, product owners, business teams

  • Build the relationship and trust; understand the system better; seek help when needed

  • Help the team to know the value they bring from their testing and automation

• Learn the problem; understand the problem; step up to solve the problem

  • Ask, is this a problem worth solving for the business

• Human is a key center of the engineering

  • Human is needed for developing

  • Human is needed for testing

  • Human is needed for automation

  • Human is needed for the maintenance of all above said

  • Humans cannot be removed from action saying automation takes care of all

• Automation is not a replacement for Testing

  • Automation exists to assist the Testing and not to replace

• Automation involves programming and is about programming to leverage the testing

  • The libraries as Selenium, Appium, Cypress, etc., are 30% to 35% of code in automation what we write

  • The rest of the code is about how we program to build, organize, scale, and maintain the automation

  • Practice programming to write better automation

  • Embrace Programming

  • Practice Object Oriented Programming and Design Patterns

• Automate which has to be automated

  • Automate first what has to be automated

  • Do not automate everything – yes not to automate everything!

  • What needs to be tested by humans, has to be tested by human

• Build the skills which will help you to do the Automation in Testing

  • This is very important and not to ignore it

• Know and confirm what is expected out of automation from business and stakeholders

  • This helps to think and define the value return from the automation

• Know what to automate, where to automate, and how to automate

  • Automating 100% is good to hear but it is not practical and logical

  • Figure out what to automate i.e. what cases or ideas to automate

  • Identify the right layer where it is appropriate to automate

    • Automating at UI layer is a cost; automate minimal at UI layer

    • Automate at service layer i.e. API layer

    • Encourage the Unit Testing practice and culture in the engineering

• Automate at the lowest layer which returns the maximum value and that has a low probability of failing

• Know what should be covered from Automation in Testing

• Understand the system being tested and its internals

  • Without understanding the use cases and business expectations out of a system, writing automation might not return the expected value

• Have the automation strategy in place than just having/working on the proof of concept

• Know and understand the tech stack of the product

  • Learn the architecture of the product

  • Understand the DevOps operation of the product

• Automation is a separate project within a project that builds the product

  • The time needed for thinking, setup, configuring, coding, maintaining, and collaborating within teams are unique tasks

  • This needs its time and it cannot be adjusted with the time available for Testing

    • Note that, Testing and Automation are two different activities though automation exists to assist the testing

  • Testing and Automation are two different activities that overlap

• Have the deterministic testability and automatability attributes in the product to leverage the value of automation and Testing

  • Ask for it with product owners and developers

• Be informed about the value returned from the automation and its limitation

  • Keep the stakeholders informed about the same

• Testability is the foundation in a system for Testing and Automation in Testing

  • Automatability attribute together with Testability makes the system very responsive for testing and automation

• Do not be scared of automation; encourage the team to start thinking about it and practice it

  • Listen to all groups who are for automation and who are not for automation

  • Understand what they mean for what is being said by them

  • Learn the context in which they are talking

  • Pick what works for you, but do not ignore and be away from the practice of automation

• Automation is a need today and for coming days; it is a skill looked for and it helps a Test Engineer very much

• Embrace automation!

Performance Testing - The Unusual Ignorance in Practice & Culture

 

I'm continuing to share my experiences and learning for100 Days of Skilled Testing series.  I want to keep it short and as a mini blog posts.  If you see, the detailed insights and conversations needed, let us get in touch.

The ninth question from season two of  100 Days of Skilled Testing is

What are some common mistakes you see people making while doing performance testing?  How do they avoid it?

Mistakes or Ignorance?

It is mistake when I do an action though I'm aware that it is not right in the context.

I do not want to label what I share in this blog post as mistake.  But, I call it as ignorance despite having or not having the awareness, and the experience.

The ignorance said here is not just tied to the SDLC.  It is also tied to the organization's practice and culture that can create problems.

To this blog post's context, I categorize the ignorance in these categories -- Practitioner and Organization.

1. Practitioner's ignorance
• Not understanding the performance, performance engineering, and performance testing
• When said performance testing, taking it as - "It is load testing"
• No awareness on what is performance and performance engineering
• Going to the tools immediately to solve the problem while not knowing what is the performance problem statement
• Be it web, API, mobile or anything,
• Going to one tool or tools and running tests
• No much thinking on how to design the tests in the performance testing being done
• Ignoring Math and Statistics, and its importance in Performance analysis
• No idea on the system's architecture, and how it works
• Why it is the way it is?
• The idea of end-to-end is extended and used in testing for performance and having hard time to understand and interpret the performance data
• How many end-to-end your tests have identified?
• Can we test for performance to all these identified and unidentified end-to-end?
• Relying on the resource/content in internet and applying or using it in one's context without understanding it
• No idea on the tech stack and how to utilize the testability offered by it in evaluating the performance
• Not using or asking for testability
• Getting hung to most spoken and discussed 2 or 3 tools on the internet
• Applying tools and calling out it as performance testing
• No attempting to understand the infrastructure and resources
• How it impacts and influences the performance evaluation and its data
• Idea on Saturation of resources
• Thinking it as a problem
• Thinking it as not a problem
• Not working to identify where will be the next bottleneck when solving a current bottleneck
• What to measure?
• How to measure?
• When to measure?
• What to look when measuring?
• Not understanding the OS, Hardware resources, Tech Stacks, Libraries, Frameworks, Programming Language, CPU & Cores, Network, Orchestration, and more
• Not knowing the tool and what it offers
• I learn the tool everyday; today, it is not the same to me compared to yesterday
• I discover something new that I was not aware of what it offered and exist
• I learn the new ways of using the tool in different approaches
• No story in the report with information/image that is self-describable to most who reads it
• And, more; but the above said resonates with most of us
2. Organization's ignorance
• At the org level, for first and to start, it is ignorance in Performance Engineering
• Ignoring the practice of performance engineering in what is built and deployed
• Thinking and advocating, increasing the hardware resources will increase and better the performance
• In fact, it will deteriorate over a period of time no matter how much the resources are scaled up and added
• Ignoring the performance evaluation and its presence in CI-CD pipeline
• The performance tests on CI-CD pipeline should not take beyond few minutes
• What is that "few minutes"?
• Not prioritizing the importance of having the requirements for Performance Engineering

Recently, I was asked a question - How to evaluate the login performance of a mobile app using a tool "x"?

In another case, I see, a controller having all HTTP requests made when using web browser. Running these requests and trying to learn the numbers using a tool.

I do not say this is wrong way of doing.  That is a start.

But, we should NOT stay here thinking this is a performance engineering and that is how to run tests for learning a performance aspect[s].

To end, the performance is not just - how [why, when, what, where] fast or slow?  If that is your definition, you are not wrong!  That is a start and good for start; but, do not stick on to it alone and call performance.   It is capability.  It is about getting what I want in the way I have been promised and I expect; this is contextual, subjective and relative.  The capability leads to an experience.  What is that experience experienced?

Sometimes, serving the requests by what you call as slow, is a performance.  What is slow, here?

The words fast and slow are subjective, contextual and relative.  It is one small part of performance engineering.

That said, let me know, what have you been ignoring and unaware in practice of Performance Engineering & Testing?

Stroking the Gestures in Gesturs 1.1

I practiced testing on the gestures stroke in the application Gesturs 1.1. This was the application provided to test as part of STePIN Testers Arena's 4th contest. I covered various aspects of tests on different quality criteria.

The tests conducted for gesture stroke is interesting and it gave me new perspective of learning. It was a two hour session where I touring the product to understand and how it calculates and executes the gestures stroked by me.  The coverage area spanned from install, operation and post uninstallation.

The observations I made in a session is shared in this report. It spans to coverage area from pre-installation, installation, configuration, in-built gestures, customized gestures, gesture stroking, stroking velocity, stroke angular details, stroke length, gesture reversal and gesture threshold area.  Explored the product on functionality and response time criteria. 

From the experience part of use, I felt very much uncomfortable to use Windows in the usual way I do using the'Windows' key in keyboard. The Windows key was not functional when Gesturs process is active in Activity List of Windows.  Using the product in laptop on trackpad with no mouse, is another area where the experiences varies on different versions of Windows OS. 

That is, Windows 7 and Windows 8 has different uncomfortable experience when used it with Gesturs 1.1. Whatever it is, it was not helping me to make my task quicker. I have worked on how the Gesturs can be designed to make the experiences better. While I worked on this, I worked on the Test Architecture of the same and it is impressive for me when I tested it under the perspectives I thought of at the time of testing it. As known, the product is still in the primitive state of development and it will get better as it evolves. Wishing the best to development teams.

The testability factor for gesture testing was not straight simple. But, it had clues to learn and test about it. I used few tools available in Windows OS to make my learning quicker during the session. There was a need to build second layer of testability to understand Gestur's gestures functioning. Once I did that, it became easier for me to proceed with the testing.

The interesting test which I did not carry out is working on different screen resolution and varied hardware acceleration for gesture stroking and response.

MVQT: The Testing and Tests with a MVP's Perspectives

I was leading multiple teams and its delivery in a testing service company.  Then, I came up with this thought -- Like MVP, I also have the MVT (Minimum Viable Tests) for a MVP.

Further, I expanded this thought in my day-to-day practice on tailoring to different contexts. I'm observing that it is applying well to the different contexts when I tailor it to the contexts.  After experimenting it for 10 years, I'm sharing this as a blog post.

What is a MVP?

I take this from Eric Ries.  It looks simple and precise to me.

The Minimum Viable Product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.

I see this technique [and a concept] can be applied to anything when I'm developing.  As a test engineer, I develop the tests and test code in major as part of my testing.  On applying the idea of MVP to my testing and deliveries, I see the value and result.

Reading this blog post of me to know who is a developer.

Testing, Tests, MVP and MVQT

In software test engineering, I see the MVP as Minimum Viable Questioning Tests.

The Minimum Viable 'Q' Tests (MVQT) for a focused area of a feature [or to a feature]

• Helps me to identify the priority tests that should be executed for first
• Allows me to learn information on priority which matters critically to product and stakeholders
• So that a informed decision can be made.

The Q in MVQT stands for "questioning".  I read it as Minimum Viable Questioning Tests.  I see the "Q" as a placeholder for the Quality Criteria.  That is, MVFT means Minimum Viable Questioning Functional Tests to a feature or a workflow.

The MVQT are key to know:

• Have I identified and designed the priority tests?  How do I know that I have got them?
• Did stakeholders get the information which they wanted to know on priority?
• Did MVQT help me to
• Explore and know what I wanted to know about a feature or a workflow?
• How fast I was here to know and learn this?
• How did I develop my tests incrementally?  Did I?  If not, then, is it a MVQT?
• Did MVQT help me to know
• Am I aligned and in sync with expectations of my stakeholders and customers who are using the software product I'm testing and automating?
• Did the MVQT help me 
• In collecting the critical information in a given context for the scope of testing and automation?
• Do the learning and outcome from this MVQT help to reinforce the validated learning of customer?
• Do MVQT result support the outcome of Unit Testing result?

The tests in MVQT has to be consistently revised and evaluated to keep it as a MVQT.  Note this, not all tests are MVQT.  If the number of MVQT is growing to a part of feature or to a feature, it is time to think about what is MVQT for you.

The "minimum" tests are highly effective and it helps me learn and test better technically and socially.

MVQT and Testing

• Sanity or Smoke Tests
• The set of MVQT which helps me learn can the build be taken further testing
• MVFT - Minimum Viable Questioning Functional Tests
• Apply this to a feature or a workflow or to that part which can be evaluated with minimum tests for its functionality
• To update is this aligning to the validated learning of customer [stakeholders]
• MVPT - Minimum Viable Questioning Performance Tests
• MVUT - Minimum Viable Questioning Usability Tests
• MVAT - Minimum Viable Questioning Accessibility Tests
• MVTxT - Minimum Viable Questioning Tester's Experience Tests
• MVST - Minimum Viable Questioning Security Tests
• MVAF - Minimum Viable Questioning Automation to a Feature
• MVLT - Minimum Viable Questioning Localization Tests
• MVUIT - Minimum Viable Questioning UI Tests

You add more of this to your list and context.

In a way, MVQT should ask and look for the testability, automatability and observability.  If this is not happening, then there is no possibility of saying I have got my MVQT.

More importantly, in the CI-CD ecosystem, MVQT pays a major role.  If I should have my tests in the  CI-CD pipeline, then, the MVQT is the way and it focuses on a targeted area to evaluate it.  Else, it is hard, impractical and not possible to test in CI-CD eco system by delivering continuously.

Ask and Review for MVQT

Ask for MVQT, when you review these:

• test strategy, test framing, test design, test ideas, test cases, test plan, test architecture, test engineering, testing center of excellence, and test code

For example,

• What is the minimum viable questioning performance tests that you have got to test this feature?
• What is the minimum viable questioning performance tests that you have got to test this workflow?
• What is the minimum viable questioning security tests that you have got to test this feature?
• What is the minimum viable questioning GUI tests that you have got to test this feature?
• What is the minimum viable questioning contract tests that you have got to test this end point?

Likewise, What is the minimum viable questioning automation tests that you have got to test this feature?

Ask how these tests qualify as MVQT in this context of testing and automation?

This should help you to see how effective is the test strategy in a given context.

Importantly, the MVQT and its effectiveness is a testability to test your tests.

The Credit is to Me

I'm not sure if the idea what I'm saying here in this blog post is practiced by other test engineers.  I have not seen this being discussed about it in public forum.  I have not come across it in my awareness and to the exposure I have put myself.

Hence, I will take this credit to me.  Giving the credit honestly is not a common sight and practice.  I have not got my due credits for using the ideas, thoughts and work that I have come up with.

So, I make it as a open letter and call out that credit for this idea, thought, concept, and practice will be to me when you listen, use and practice it.

Test Coverage Ideas and Test Ideas – Part 1

In one of my practice session, I was brainstorming for the test ideas.  I got a question, what am I covering in the ideas?  The product being tested in the session was Triangle application for testers programmed by James Bach. With this question, I toured the application for 20 minutes to learn what it is.

While doing this, I understood, to have better and effective coverage, there needs to be the models of Triangle application and not just one model.  From here, I started to build the models. It took time and experienced the pauses frequently. Learning what I’m undergoing, the strategy for approaching the mission changed.

I love experimenting. This work is an experiment being carried out for making my testing useful, informative and valuable. The strategy now is to write the Coverage Ideas for first by touring the product.  Then use, the Coverage Ideas to build tests based on what I need to cover for the testing context.  Doing this I noticed the advantages and disadvantages. Advantage is, there is no need to sketch out the model in detail as I have the Coverage Ideas (detailed model guiding structurally) which shows me test ideas. Doing this I get the mental model for each types of coverage.  Disadvantage with this is, it might consume time in initial stage using this strategy for Test Engineering. On practice, the time might reduce depending on testing context, testability, tester skills and other factors of system being tested.

With this, I have got more than one Coverage Models to test and I made note of it consciously. And, I brainstormed for test ideas on each Coverage Model knowing the testing context. This is very interesting for me.  I learn the Test Coverage Idea as – an experimental question and the actions which helps in identifying and building the various coverage models (an idea) of a system for testing context. There by aiding to evaluate the coverage dimensions of the Test Ideas coming out of each Test Coverage Ideas on a reference.

In simple to summarize, Test Coverage Ideas --> Build Coverage Models (test ideas) with a reference to the testing context --> Brainstorm and identify (design) the tests; execute the tests; evaluate, observe and make notes.  These all activities can be executed in parallel as well.

To make the start, I decided to brainstorm and write test coverage ideas for one or two sessions. Then pick these test coverage ideas, execute and make note of test ideas coming out of from respective coverage model. This gives me structure for the test ideas and reference to evaluate it for knowing the coverage extent of it.  Meanwhile, I make note of the new test ideas as testing continues.

Drawing a Test Coverage diagram, it helped me knowing -- what I’m covering and to what extent, what I have covered and to what extent, and what is not yet covered and should be covering. The testing context and different models of system helped in seeing the dimensions that a test should take and can take.

Figure: Context Free Test Coverage Model

The above diagram shows, how little and/or effectively we cover the varied dimensions for a test. Bigger the three circle’s common intersection space, it is likely that tests carry the desired dimensions and variance when evaluated against a reference in testing context. The space when just two circle intersect can also be useful in terms of coverage, but it might not be effective with dimension attributes of another missing circle.

I will be sharing the test reports coming out of this model in next successive threads of this post.