Model, Oracle, and Perceived Quality

 

These words are one of the most repetitive words in my blog and also in the testing community's discussion.  Nevertheless, I write a blog post on it and share my interpretation here on the understanding of it.  I came across a discussion on these words in one of the testing communities. Here is how I see it as of today with my practice of Software Test Engineering.

Model

The "Model" can be seen as a representation.  In Software Engineering, we use models as a reference to build and develop the product.  Software Testing can be leveraged when we use the models to design, build, execute and interpret the tests.  How I build the models or see models can vary each time in my work.

Today, as I write this blog post, here is how I understand a model:

• Model is how I'm understanding
• Model is what I'm understanding
• Model is what I have understood
• Model is what I have to understand
• Model is also what I have not understood and what I am not aware
• But, this would not be included as a part of the picture or written in the model most times
• We tend to see the model as a working object (always?)
• In the working object, having something not understood and not aware is not a common practice
• A model can be a non-diagram and the organized documents or words
• I look at The Constitution of India as a model on which I live in the Indian society

Usually, one looks for a diagram on hearing the word "mode".  I did this, that is I looked for the diagram and continue to do so.  But the model need not be a diagram always.  But, the diagram helps better in relating and understanding.

Example of models:

• A representation understanding of me how I can broadcast YouTube live stream on Zoom
• Here I can have multiple models
• From a tech layer spanning to a UI layer of Zoom and then to people who are watching it on Zoom, YouTube or both
• And, many more models like this
• If I'm on a sales team, my model thinking on the same would be different
• Like which platform that is YouTube or Zoom caters the content to the maximum audience I'm focusing on

Oracle

In a context-free case, I will say Oracle as -- It is a reference which I refer to learn and interpret what I'm experiencing or about to experience.  It helps me to understand and in decision making.

That said, the Oracle is not a source of truth.  It is a reference and so it is a heuristic.  If it is a source of truth, can it be a heuristic?  This is the challenge and confusion one goes through in understanding and drawing the relationship between the oracle and heuristic.  When we understand oracle and heuristic, it is simple to draw the relationship between them and know when it can serve as the other.

When heuristic is taken as a source of truth, it can fail to be a source of truth at any time.  Heuristic is a fallible way of solving a problem or making a decision.  That said, not all heuristics are oracles; and, the oracles can be used as heuristics.

In software testing context, Oracle is quoted as -- An oracle is a way to identify the problem that we experience during the testing.  It helps to identify the problem.  Maybe we use the word "problem" as we are into the context of Testing.  Testing is expected to figure out for what it is commissioned; in most cases, we take it is to find the problems.  So that definition or quote for the word Oracle in software testing context has the word "problem" with it.

Example:

The 1000 INR currency note that we had in India, was a valid and acceptable currency.  If someone asked before demonization this was accepted as truth.  But today, it is not a valid currency and this is accepted as a truth.

The 1000 INR currency note was a valid currency until 8th Nov 2016.  This currency was used in daily life transactions by people.  This is a heuristic and as well oracle.  

Today, a 1000 INR currency note is not a valid currency.  This is an oracle and as well a heuristic.  

If I go with 1000 INR currency it will not be accepted in transactions.  People will identify it as a problem if this currency note is exchanged in a transaction.  

• Oracle: 1000 INR currency is not valid and accepted in transactions at the time of writing this post.  So not to tender 1000 INR currency in the transaction.
• This gives an example of an oracle as a heuristic
• Heuristic: Use the other available valid currency in a transaction.  Is there any Indian currency note that is invalid today?  What are the valid currency notes that I can use today?
• This gives an example of heuristic as an oracle

Perceived Quality

• We experience and experience is an emotion
• Quality is an emotion and an experience
• How I perceive the emotion and quality from testing or by any events/actions might not be the same to another person
• But it is still valid and authentic because that is what I perceive and experience

Coming to the text "Perceived Quality", I have two questions:

1. How do I perceive the quality as a tester?
2. How the business and stakeholders perceive the quality by referring to feedback from testing?

Whose opinion and perception of quality matters?

Here is my thought for now on this:

• As a Test Engineer, I try to provide information from my testing
• The information with a compelling and influencing story of my testing
• The outcome of my testing
• The potential consequences of the outcome as I perceive
• I have my words to share about my emotion and experience
• Any measures to be taken and the authority to change the direction in this regard are with stakeholders and the business
• What would stakeholders and the business perceive about quality from my testing story
• The outcome from this perception is crucial in the larger interest than what I perceive as a Test Egnineer

The compelling and influcing way of telling the testing story is important.  The perception what the stakeholders and business make from this story, is what they see as quality in the first sight.

I say that word "perceived quality" talks about how the stakeholders and business is perceiving the quality.  And, how as a Test Engineer, I'm influencing it with my advocacy.

Example:

Let us take a scenario where the Product Owner and Sales team are looking forward to data for making the decision. 

The product owner has a good feel about how the notification works. The notifictions are received by the intended interfaces at the time of testing. 

But, did it record in analytics about how many taps were made on notification?  No, it did not record.  The Product Owner together with the sales team needs this data to plan the business decisions.

The lack of this data will the lower quality experience of notification for the Product Owner and Sales team?  

What is the quality emotion and experience perceived here by two different teams and people?

Automation Strategy - How to Automate on Web UI Table for the Data Displayed in Table

 

Use Case to Automate and a Problem Statement

I read the below question in The Test Tribe's Discord community server.  As I read, I realized, we all go through it when we think of automating a use case.  The credit of this question is to Avishek Behera.

Picture: Description of a use case to automate and a problem statement

Here is the copy-paste of the use case and problem statement posted by Avishek Behera:

Hello everyone, here is a use case I came across while having a discussion on automating it.

A webpage has a table containing different columns,let's say employees table with I'd, name, salary , date, etc

It also has pagination in UI, we can have 20 rows in one screen or navigate to next 20 records based on how many total records present,it could go about 10 + pages and so on....

Problem statement:

How to validate the data displayed in table are correctly displayed as per column header , also correct values like names, amount etc. Use case is to validate data.

The data comes from an underlying service, different endpoints basically.

Now it's not about automation but about right and faster approach to test this data.

What are different ways can we think of?

I know this is a basic scenario but since I was thinking of different possible solutions.

One way my friend suggested to use selenium, loop through tables ,get values ,assert with expected. Then it is time consuming, is it right approach just to validate data using selenium?

 These are the useful attributes of this question:

1. It had the preset of context and the context information to the reader
2. The availability of context information gave an idea of
• What would API look like
• The request and type
• The response and details
• The consumer of this API
3. It helped to imagine and visualize how the data would be interpreted by consumers to render
4. I get to see what Avishek is looking for in the said context

Interpreting the Use Case and Problem Statement

What it is?

• Looks like the consumer is a web UI interface
• Mention of Selenium library supports this interpretation
• The response has a data which is displayed in the table kind of web UI
• There can be no data to multiple rows of data displayed in the table
• Pagination is available for this result in the UI
• Is pagination available in the API request and response or not, this is not sure from the problem description
• 20 rows are shown on one page of a table
• The number of pages in the table can be more than one
• The response will have a filter flag
• I assume that data displayed in the table can be validated accordingly
• The response will have the data on the number of result pages
• This makes the result in a page to be of fixed length
• That is 20 results on each page and I cannot choose the number on the UI
• The response will have the offset or a value that tells the number of records displayed or/and returned in the response
• Is it a GET or POST request?
• This is not said in the problem description
• But from the way the problem is described, it looks like a GET request
• But should I assume that it is an HTTP request?
• I assume it for now!
• I assume the data is received in JSON format by the consumer
• I assume the data responded by the endpoint or the service, are sorted and returned
• The consumer need not process the response, sort, filter, and display
• If the consumer has to process the response, then filter, sort and display, 
• it would be a heavy operation on the client and the client-side automation for this use case

If it is other than an HTTP request, it should not matter much.  The underlying working and representation may remain something similar to HTTP requests and responses unless the data is transferred in binary format.

Automation Strategy for the Use Case

The key question I ask myself here is:

What is the expectation from automating this use case?

How I automate and where I automate is important but it comes later on answering the key question. The key is in knowing:

• What is the expectation by automating this use case? 
• What am I going to do from the outcome of this automation? 
• What if the outcome of the automation gives me False Positive information and feedback?

These questions help me to see:

• How should I weigh and prioritize the automation of this use case?
• How should I approach automating this use case to be close to precise and accurate with deterministic attributes?
• What and whose problem am I solving from automating this use case?

It gives me the lead to the idea of different approaches for automating the same; helps in picking the best for the context

That said, the use case shared by Avishek Behera is not a problem or a challenge with the Selenium library or any other similar libraries.  Also, it is not a problem or a challenge with libraries used in the automation of web requests and responses.

Challenges in the Problem Statement

I do not see any problem in automating the use case.  But there are challenges in approaching the automation of this use case.

On the web UI, if I automate on data returned, filtered, sorted, and displayed, it is a heavy task for automation.  Eventually, this is a very good candidate for soon to be a fragile test.  

Do the below said are the expectation from automation of the use case?

• To have a fragile test
• To have high code maintenance for this use case
• To do high rework in the automation when UI of the web change
• To complicate the deterministic attribute of this use case automation

If these are not the expectations, then picking an approach that has lower cost and maintenance is a need.

The challenges here are:

• It is an Automation Strategy and Approaching challenge
• It is a sampling challenge
• Yes, automation at its best is as well a sampling, not just the testing
• It is about having better data, state, and response which helps to have accuracy in the deterministic attributes of automation
• To know if it is a:
• true positive
• false positive
• true negative
• false negative
• an error
• not processable
• The layer where we want to automate
• The layers which we want to use together in automation, and how much
• Automate to what extent for having information and the confidence -- if this sampling works then most data should work in this context of a system?
• The availability of test data that helps me to evaluate faster and confidently

Let whatever the system have in the underhood that is GraphQL, gRPC, ReST API, or any other technology stack services, one has to work on -- how to make a request; go through the response, and analyze it in context.  Like testing depends on context, automation as well depends on context.  In fact, context drives testing and automation better when it is included.

My Approach to Automate this Use Case

I will not automate the functional flow of this use case entirely on the web UI.  My thought will be to have those tests which are more reliable and the result influencing and driving the decision.

This thought has nothing to do with the Test Automation Pyramid and its advocacy that is to have the minimal number of UI tests at the UI layer and much more at the integration (or service) layer.  I'm looking for what works best in the context and where to have the tests that give me information and feedback so I have the confidence to decide and act.

To start, I identify the below functional tests for the said use case:

1. Does the endpoint exist and serve?
2. Assuming it is HTTP,  I see what HTTP methods this endpoint serves?
3. What does the endpoint serve when it has no data to return?
• The different HTTP status code this endpoint is programmed to return and not programmed but still returns
4. What inputs (data, state, and event) does this endpoint need to return the data?
5. In what format and how the input is sent in the request?
6. In what format the response will be returned from the endpoint?
7. Is the response sorted and filtered by the endpoint?
8. How does the response look when there is no data available for any key?
9. What if certain keys and their value are not available in the response?  How does it impact the client when displaying the data in a table?
• For example,
• No filter data is returned or it is invalid to a consumer to process
• No sorted data is returned or it is invalid to a consumer to process
• No pagination data is returned or it is invalid to a consumer to process
• The contract mismatch between provider and consumer for data returned
• What the web UI shows in the table data
• Any locale or environment-specific data format and its conversion when the client consumes the data that is returned by the endpoint
• The data when sorted by consumer and provider differs
• The data is sorted on a state by the endpoint and that might change at any time when being consumed by the consumer
• Is it a one time response or a lazy loading
• If it is a lazy response, does the response have the key which tells the number of pages
• and more cases as we explore ...
10. and more tests as we explore ...

Should we automate all of these tests?  Maybe no per business needs.  Imagine the complexity it carries when automating all these tests at the UI level.  But there are a few cases that need to be automated at the UI level.  

Then, should we to look at the table rows on different pages to test in this automation?  No!  But we can sample and thereby we try to evaluate with as much as minimal data.  This highlights the importance and usefulness of the Test Data preparation and availability.  While, preparing the test data is a skill, the using of minimal test data to sample is also a skill.

API Layer Test

I have a straight case here for first.  That is to evaluate:

1. The key (table header) and its value are returned as expected
• Is it filtered? 
• If yes, is it filtered on key what I want?
• Is it sorted upon filtering?
• There is no null or no value for a key that needs to have a value in any case
• The data count (usually the JSON array object), that is the number of rows
• The page index and current offset value
• The number of result pages returned by the endpoint
2. Can I accomplish this with an API test? 
• Yes, I can and it will be efficient for the given context
3. I will have five to ten test data which will help me to know if the data is sorted and filtered
4. Another test will be to receive more than 10 rows and how these data look on filtered and sorted
• Especially in case of lazy loading
• I will try to evaluate the filtering and sorting with minimal data
• I will have my test data available for the same in the system

UPDATE: I missed this point so adding it as an update.  I'm exploring the Selenium 4 feature where I can use the dev toolbar and monitor the network.  If I can accomplish what I can and it is simple in the context, this will help.

UI Layer Test

I have a straight case here as well to evaluate in the given context:

1. I assume the provider and consumer abides by the contract
• If not then this is not an automation problem
• It is a culture and practice problem to address and fix
2. I assume the data returned data is sorted on the filter; the web UI just consume it to display
• If not, I will understand why the client is doing heavy work to filter and sort
• What makes it to be this way?
• You see, this is not an automation problem; it is a design challenge that can become a problem to product, not jot just for automation
3. Asserting the data in the web UI table:
• I will keep minimal data on the UI to assert that is not more than 4 or 5 rows
• These rows should have data that tells me the displayed order is sorted and filtered
• Let's call the above 1 and 2 as one test
• To evaluate pagination that is number of result pages, I will use the response of API and use the same on the web UI to assert
• Let's call the above another test that is the second test
• Again, the test data will be the key here
• To see if the pagination is interactive and navigatable on UI, I make an action to navigate for page number 'n'
• If it is lazy loading, I will have to think about how to test table refresh
• Mostly I will not assert for data
  
• In testing the endpoint, 
• I would have validated for the results returned and its length
• I will assert for number of rows in the table now
• Let's call it a third test
4. I will not do the data validations and its heavy assertions on the web UI unless I have no other way
• This is not a good approach to pick either
• One test will try to evaluate just one aspect and I do not club tests into one test

Note: The purpose of the test is not check if the web UI is loading the same rows in all pages.  If this is the purpose, then it will be another test and I will try to keep minimal assertion on the web UI.

The Parallel Learnings

If observed, the outcome of automation and its effectiveness is not just dependent and directly proportional to how we write automation.  It is also dependent on:

• The design of the system (& product)
• The environment and maintenance
• The test data and maintenance
• The way we sequence the tests to execute in automation
• Where and how we automate
• The person and team doing the automation
• The organization's thought process and vision for testing and automation
• The organization's expectation from testing and automation
• How, why, and what the people, organization, and customers understand for testing and automation
• Time and resources for testing and automation
• The automation strategy and approach
• More importantly, the system having and providing
• Testability
• Automatability
• Observability

Note: This is not the only way to approach the automation of this use case.  I shared the one which looks much better to the context.