AMYQ: Approaching the Solution to Test Data Challenges of Shrini Kulkarni

 

In this post, I'm trying to reason in brief for the questions shared by Shrini Kulkarni in this AMYQ.  Here are the challenges/questions shared by Srini,

• #1 challenge - setting up data in upstream systems to suite test cases that need to be run. There is AUT and there are upstream systems. In a corp setup -- individual teams are setup for each application. Hence getting another team to set some data in other system often encounters lots of manual effort and red-tapism

• #2 Reserving test data created in AUT or upstream systems for specific team's use so that other teams do not change it.

My Understanding of These Challenges

1. AUT is in place and it has upstream systems.
2. There are multiple teams for each AUT in an org.
• Say teams A, B, and C for sake of learning here.
3. Team A has difficulty when wanting to create the Test Data in the space of Team B, and vice versa.
• The difficult for Team A can be as,
• Not permitted to create data
• No awareness and understanding of Team B's system
• Cannot make progress in testing unless there is data created for Team A
• The created Test Data by Team B is not shared to Team A for multiple reasons as data pollution and corruption which disturbs their test cycles
• And, more ...
4. How to reserve Test Data created in Upstream system for use of a specific team?
• How to make sure other team does not use it or edit it or delete it?

This is my understanding of Shrini's challenges.

Conway's Law and My Experiences

The Conway's Law says,

The architecture of a system will be determined by the communication and organization structures of the company.

Inverse of Conway's Law says,

The organizational structure of a company is determined by the architecture of its product.

The above stated both laws hold good for Microservices teams and the teams which consumes their services.  

I have experienced,

• Each microservices teams creating their own set of test data.
• These test data is not shared to any other team.
• The other teams are not allowed to create data in their space.
• No team is aware of what others are doing in tests and what they are using to test.
• The teams work in silos.

This leads to friction, aggression and unhealthy communications between teams.  Now, where is the possibility of having the Test Data in upstream system which can be consumed to run tests by every other teams?

Do you see the statements of Conway's Law and Inverse of Conway's Law here in how these teams are setup, orchestrated and communicates in building one product?

What's the Problem?

Here is my understanding of the Problem Statement from the challenges shared by Shrini,

How to setup Test Data in upstream system, to suite the test cases, that need to be run?  And, how to ensure Test Data created by one team is not modified by other teams?

If observed, the outcome described in previous section is not a technical problem.  

It is the collaboration and communication problem, that comes up in presumption -- other teams using one's test data it affects their team's work and delivery.

Yes, it can impact badly if the creation, editing and managing the test data is not done attentively by teams when everyone shares and consumes it.  So the respective team's engineering managers and directors show the resistance to create data in their team's space as it impacts their pipeline and delivery.

How I Solved It?

In the contexts where I work, I have different upstream systems to which I'm supposed to interact and get the data.  Then, use these data to test the service offered by the product.

But, creating test data in other team's space is not allowed!

There are multiple solutions that I approached with and solved this problem in different organizations.  The below said approach worked with most clients and so I share it here.  It is like a Design Pattern; I can use the structure of it in multiple teams to solve the similar problems.

I came up with an approach to create a suite having the endpoints of all these upstream systems.  And, I need not say how difficult and pain it was to get the endpoints and its details from teams.  It was a marathon circus of me to get it!

Here is how I approached the solution,

1. I understood my teams were comfortable with Postman.  
2. For this problem solving, I see Postman was simple and quicker than writing a automation project with libraries like RestAssured and request.
• Team can run the Postman Collection from their system in quick time.
3. I created an Test Data Inventory
• The inventory has the Test Data which the test teams engineered for their testing and automation
• These data is not just for functional testing; it had Test Data for security, performance and accessibility too.
4. I built the Postman Collection having the endpoints of these upstream systems.
• The collection is version maintained like any other code in the organization.
• The meaningful commits are being made and pushed, as and when the need comes up.
5. I crafted an environment file and wrote the scripts which places a variable having the stamp
• This stamp tells, it is a test data created by automation run for this version of regression cycle and from this team.
• The stamp is max of 10 characters and it is dynamic based on that run to avoid duplicate data.
• I said it is dynamic and not the random!
• This dynamic characters has meaning and an intent.
• Note, when I say dynamic it is created for that iteration of Test Data
6. These created test data are used by team eventually not touching the test data of other teams.

This let us move further with our tests and automation in the pipeline.   Note that, I have multiple upstream systems to which the AUT speaks.  Not disturbing them is a challenge!

I maintain the inventory of test data having versions of test data being crafted sprint-on-sprint.  We pull the test data of different versions from the inventory as and when it is needed for the intent of test and automation.

All that said, the team continues to work in silos with no much collaboration.  Who has to solve this?  Though I can solve this, my role and pay grade does not allow me to do so effectively.  I have solved within my teams and not to the organization level.  It is a engineering and culture problem which has to be addressed at the organization level.

To summarize, 

1. The test data creation and its inventory management having versions of test data is an Engineering culture of an organization.
• It has to be taken in consciously and help the teams move swiftly in building the resilient and usable systems.
2. Test Data and its engineering is a project within an engineering project.
• No wonder if any LLMs offers a business solution solely on Test Data in coming days!
• Test the data offered by such LLMs before consuming it.
3. Practice in the space of test data and testing the test data.
4. There is Data Coverage in the engineering like it has Test Coverage.
• What's your Data Coverage?

Functional Testing Is Must In Performance And Security Testing

 

I'm sharing about how I missed to test for functionality while I was immerse focused on testing for performance of a Stored Procedure.  I was unhappy for a couple of days as I missed something that I practiced for years.  

I'm glad for reinforcing this learning with much more awareness into my testing's MVT and MVQT, now.

Context of Testing

A Stored Procedure was optimized for better execution time.  No change in the functionality.  This part of the system is not touched for a long time (years?).  There was no change in functionality here for long time (years?).  The time taken by SP was of concern.  I was asked to test for the optimization.

The complicated area, here, is the test data to use.  It took me days, for identifying and building the test data to test this optimization by mimicking the production incidents, use cases, and data.

When I got the test data ready, it was the fourth day of my testing this change.

Where Did I Go Blind By Being Focused?

The test data that I prepared is solely for the evaluation of the execution time.  This test data helped to test functionality as well.  But, my focus was on evaluating performance not functionality from this test data.

The change in SP did impact the functionality.  I was supposed to use the large data range to test for functionality of this feature which includes two SPs.  But, the task assigned was to test just one SP which is optimized.  I got blind here!  

Are you asking, what is the impact of this functional problem?

• In the one complete business work flow, this functional problem added the same data into different sets in the subsequent iterations.  Redundant Data -- This is not an expected behavior.

I just spoke performance, traces, data I/O and execution time, because that was a pressing problem.  Why?  That was the objective given to me.  

My testing mission fell short in redefining this objective.  If I had redefined it, I would, have added functionality in the better scale.

If I had redefined it, I would have pulled the other SP into functional testing which is also part of this feature's work flow.  These two SPs are expected to handle the data by eliminating the redundancy.

It was a simple test, but, I did not include/had that in my testing mission that day.

Why Did I Go Blind?

The performance test blinded me for functionality, as I saw the basic functional flow looked functioning.  But, the data count was going wrong when a bigger data range is used in the context.  

See here, how stupid I was in my testing!  I'm testing for a SP that has a change as part of its optimization for execution time.  I never brought the functional testing in.  Why?  I focused on the testing objective.

I just looked into one SP that is optimized.  I did not look the other SP which has to work along with this SP later to complete functional flow of the feature.  Why?  How is that even possible?  I was asking myself this.  I see, this is okay from the perspective of the testing objective I had.  But, not okay from the perspective of a test engineer who is supposed to think the impact and prevent the problems.

My immersed and concentrated focus on performance and its related activities on a SP for four long days did not let me see this.  

What Am I Saying Here?

While I have tested for DBs and ETL systems for years, I did not use my learning here.  What is that learning?

When there is a change in any part of the ETL, SP or DB of a system, testing for the functionality for the business workflow is equally important.  Vary the data dimensions and evaluate the counts.

I was completely hooked into the execution time and the test data while switching between the environments for four days.  The chaos in data between environments is something that misleads easily.  I fell to it this time.

I say to myself, if it is a fix for the performance optimization or a security [or any quality criteria], testing for functionality is equally important and of priority as running the tests for performance or security.

When a DB layer is picked for fixing and optimization, testing for functionality in a equal scale is must.  There is a change in the code or/and infrastructure and it has to be noted with additional attention.

To add on this, this time, I did not go through and analyze the SP.  I took this call from the test team.  This call of me costed and had a major part in letting me not to think of functionality.

My fellow colleague ran a test with varying data size by completing the business workflow and observed the problem, and informed me.  I give the credit to Sandeep.

If I had brought this performance test under the automation, I would not have done this.  Why?  I will evaluate and assert for each data returned for different sizes.  I did not automate here and there was no need for it in this context.

Redefine the testing objective that you have got; it helps when you see the model of a system and test.

Respect all the fix and suspect all the fix.  This helps in a longer run!  

Database: Finding the Tables Having Specified Column Name

 

In today's pair testing session with a mentee, we were testing for Database I/O.  We were on PostgreSQL.  One of the questions a mentee had is,

How can I figure out the tables having this column name?

Running through every tables and exploring if the column being looked for is present or not, is time consuming.  It is not a approach to take as well.

I went through this when I started the ETL testing practice in 2011.

Here is the query that works on PostgreSQL to find table names which has specified column name.

Query:

select table_name, column_name
from Information_Schema.Columns
where table_catalog='database_name' and column_name like '%column_name%'

It is a better approach to know the precise column name and using the condition as -- column_name='EmployeeId'.

This query should work on MySQL and MSSQL Server.  If not working on MSSQL, need to look into the FROM and WHERE clauses if it is vendor specific.

Behind the Every Test Data, There is a ?!

 

Read this blog post to have a perspective about the Test Data and Test Data Management.  The point is, if I'm not aware of a test and what does it tell me to explore, I cannot think of a Test Data.

That said, if I know what I should be evaluating as part of performance, why, when and how, this will help me to come up with a thought for identifying the tests and its test data for the same. 

The ninth question from season two of 100 Days of Skilled Testing is:

What role does data management play in performance testing, and how do you ensure the availability of suitable test data sets?

Testing and "Ensure"

We test and have tests in testing, because, there is no "sure" and "ensure" idea in software.  But, we presume on a rational basis upon, "if, these are this", in a given context when the software processes.

Now, ask yourself, how can we ensure the availability of suitable test data sets?

In my opinion, the Test Data is often misunderstood.  This is the primary problem and should be the first problem, when asked "what are the challenges in creating the test data?".

When you read the concluding lines of this blog post, you will learn why I say this.

Test Data and Immunity

In my opinion and experience in practicing the Test Engineering, I see, the Test Data should be a viral strain and it should have its variants.  When this test data is used to test [experiment, test investigate, and debug], how do the software and its ecosystem respond?

• Does the software and its ecosystem is immune to this test data?
• Does it exhibit any risks and problems?
• If yes, then, do the purpose of my testing and automation is accomplished with this test data?

This puts me back to question, what is the purpose [intent] of my test?  It drives me to derive the test data which helps me to know -- What am I supposed to learn and on priority?  With this, I get an idea for what kind of test data I should be creating knowing its pattern.

If the system is immune to Test Data and not reveling anything new in the context, I classify this pattern of test data as "Immune" to the context.

In my practice and research work in Test Engineering and Software Testing, to start, I categorize Test Data into two areas.

1. Immune
2. Not Immune

Further, I have categories, under these two, where I classify the Test Data deterministically for the context.   Get in touch if you want to learn more about this.  I'm just one ping away!

The tests should help me to evaluate for the immunity and also non-immunity; both are essential and necessity.  

The credit is to me for such classification of Test Data.  It is my research work out of my practice.

Note that, Test Data is not just the input [characters or files] entered or given to a system.  Test Data has its association to tech stacks, infrastructure, ecosystem, business workflows and people.  To craft such Test Data, one has to have the understanding of the system and its internals, and, the problem it solves by knowing how it solves.

Performance Testing and Test Data

1. What is that I'm testing as part of performance?
2. What do I want to evaluate in the name of performance?
3. What part of the system is evaluated for its performance?
• Should I evaluate this in isolation or as a wholeness of the system?
4. What domain knowledge and information I should have when testing for performance?
5. What system's architecture and internal details I should understand and be aware to test for performance?
6. Is this the first delivery?  Or, do we have this system running in the production?
• If it is first delivery,
• How will I create the test data to suit the consumers of this application?
• What are the key workflows of business that we should be evaluating for its performance?
• Do all workflows and sub-systems need the evaluation for performance, and on priority?
• How do I map the fragmentation of users and their data [with its patterns]?
• What are the infrastructure and ecosystem characteristics that should be part of the test data identified?
• Does caching have any effect if the same pattern of data is used?
• If it is a running version in production
• Can I refer to the DB to figure out the pattern for the particular workflow that I'm evaluating?
• How can I match the test data to have the production data's characteristics and attributes?
7. What is the backup strategy for the Test Data?
• How do I version control the Test Data?
• Which version of the Test Data I should be using?
8. What is the threshold I'm targeting with Test Data?
• What should be the size of the data in DB when I make the IO and RW operations?
• What should be the network capability when I make the IO and RW operations?
• What should be the hardware capability when I make the IO and RW operations?
• What should be the geographical traffic and its pattern when I make the IO and RW operations?
• More of such factors will be considered when identifying and deriving the test data.
9. What is the client error yielding Test Data that I should have for the workflow?
10. What is the server error yielding Test Data that I should have for the workflow?
11. What is the redirection yielding Test Data that I should have for the workflow?
12. What is the no-response and no-change Test Data that I should have for the workflow?

And, more.  It is simple; get in touch to discuss and know beyond the listed.

To conclude and stop here, all these questions, do not ensure or assure or make sure that I will have test data for evaluating a characteristic of performance.

• It helps me to know:
• What are the tests I should be doing?
• What kind of preparation I should be having in my practice to create the Test Data for these tests?

The, Test Data should challenge the available Testability and its limits.  If it is not doing, then, we are having a test data no doubt about it; but, it is of shallow. Shallow!?

One has to ask self, "Is this sufficient enough and effective Test Data for the system [and workflow] I'm testing?"

The, Test Data should drive the engineering team to add more layers of Testability into the system.

Test Data is Not [Equal To] a Test

 

Not every release is a critical release!

There are releases which are critical from technical and business interests.  In such critical releases, most of my time is spent on understanding the technicalities of the system and the test data identification on identifying what are the priority tests.  Identify and building the test data takes major chunk of the testing time. This effort has helped me, testing, stakeholders, projects and products immensely.

Test Data != Test

Looks like the word "Test Data Management" has become a buzz and marketing word.  In how the word "Test Data Management" is pushed, to me it looks like the intent is missing for -- how to identify the tests and its test data?

• If I cannot think of tests, how can I think of Test Data?
• How can I think of tests, the right tests in the context, and the priority tests out of it? 
• If I cannot get this, how can I get the Test Data for all these tests?

Before talking of Test Data Management, we need to talk about the intent and goal of the test.  If I have a clarity here, by being aware of the product's technical internals, architecture and its eco system, I can think of test data in a better sense.  For first, the PaaS which provides Test Data Management solution has to push and enforce for learning and mentioning -- What is the intent of the test?

When I know, for what to evaluate and how should I be evaluating it, those test intent will lead me to the Test Data and its dimensions which should be vectoring the tests.  

Test Data is not [equal to] a Test.  Test Data is one of the variance, in fact, it is a multi-dimensioned variance that a test will [need to] have.  While a test has one deterministic objective, the test data will have multi-dimensioned vectors that instruments a test.

Test Data has its role in a test, and, it is a critical role.  Hence, we practicing Test Engineers talk and spend our time on Test Data in equal importance as we give our time to identifying, designing and execution of tests.

Test Data is not a test; but the test data is an expression of the test's intent.  Test Data is one of the byproducts of a test.  Having this clarity is important.

Note this, the test data is not the only expression of the test's intent.  A test will have multiple touch points; these touch points express, advocate and can show the intent of a test.  A test data is also a heuristic as a test is.

 

Test Data Management

The word "management" is underrated and not attempted to understand from context where it is being spoken.  How do this word sound -- "Test Data Leadership"?

In engineering, especially in the software engineering, the word "management" inherently talks primarily about design.  On day-to-day operation, the management designs its strategy and approach to solve the problem and challenges.  

When said "Test Data Management" in software engineering, it is about strategizing and approaching the problem of identifying and categorizing (subsets) the data to test.  A kind of leadership at one layer.  It has its role and critical to the deterministic outcome for a test.

In machine learning, we can see this categorization -- training data and test data.  Then, is training data not a test data?  It is!  The training data is designed [managed] to train and this data as well identifies the problem as the training progresses.

The data which are identified, designed, categorized and collected as Test Data, are well sampled data in the context, for the intent of a test.

To conclude, Test Data Management is one of the correlation in software engineering to solve a problem; it is contextual in how and what tests and testing is executed.  The different teams and organizations has their own way of managing the test data.  How they do it, it is their way of doing it.  We have to look Test Data Management from point of Test Engineering and not as an another buzz word to sell just as a product and business's service.

JSON for Software Test Engineers - 101

 

In the practice of Software Testing & Engineering, I became aware of the below:

• UI (User Interface) is not just GUI (Graphical User Interface)
• UI is an interface through which a user interacts with a system
• UI is one of the touchpoints
• Data we exchange between the two systems is also a UI
• This data can take different formats
• Text
• HTML
• Image
• Video
• Audio
• XML
• JSON
• YAML
• PDF
• custom-defined data format
• and, more

Note: Refer to the MIME types and know what type of media is supported and used in the system you are testing and automating.

Testing Through Data

When the data can be used as a UI, it also means one can do the testing and automation through data using data.  From this perspective, it is important to know the data format.

I see JSON is one of the common data formats we use to store and exchange data between two systems.  Understanding how the data is structured in JSON is important for a software test engineer to think and evaluate the testing and automation through the data.

JSON For Test Engineers - 101

I documented my understanding of JSON and how to interpret it in the gist here. This markdown file has my learning.

App Crash! Testing around and inside the crash

In day and out, I come across testers, programmers, managers, and management having efforts on fixing all the crashes. Yes, all the crashes.  In a way I see, if the app did not crash, I will not know the areas that is not being handled well enough.  My testing focus areas will also have tasks noted in such areas to test and learn as much as possible. I do that task provided I can make/given time for it as it is unplanned task.

The common checks to handle crash!?

I learn, an exception if unhandled at runtime, it leads to crash.  There are multiple exception that app can witness which we never thought of during development.  In my initial days of testing, I was in assumption, if we can have -- null checks, index check, illegal argument check, and state check for an activity, we have handled most of the exceptions.  I learned, I'm wrong! How many checks can I write in the code.  I'm not a programmer by job.  I'm a tester. 

I see these checks are not enough and few more got added to my test strategies eventually -- race conditions, unexpected data, wrong data, environment factors, and many more. The collection of these checks is continuing to grow.  Do I cover all these possible crash inducer collections in my testing?  No, I can't and I won't have that luxury of time as well.  Technically, I will learn and prioritize what to use and when.

How the check looks in code, to me?

I write code for automation, which I need to assist my testing.  Here I did write such checks.  At a point of time, I saw, the automation code was full of checks.  Is that a right way?  Definitely not!  A professional and skilled programmer might not do that.  If a programmer has to have such check in each layers of the app architecture, will that sound good?  Personally as a tester, I will not design my tests that way.  As I'm not a programmer, I'm not aware of the pros and cons of doing so. At least I know that is not a better practice to have checks in each layer of architecture of the app.

By handling the exception in my automation code, I print stacktrace of exception.  But will I learn from it to be a better tester?  That's the question I have asked and continue to ask myself.  The exception fix I'm doing, is it stopping me from learning the problem which I have introduced by the actions I'm doing on the app? Is that exception blocking me from learning the underlying problem in my automation code and app?  If yes, then I have a fundamental problem which I have to work upon is my thought.

Why it crashes?

Why at all the app gets into crash? I learn, if the app gets into a state it was not designed for, it will and should crash. As a tester, I will have to learn this state (and such states) at the earliest when I experience crash and on reading the crash stacktrace. I will be happy and not make a fuss about it, if I see a crash for first.  

I learn what is the priority and impact of the crash?  Should I invest my time and test investigate further to provide as much as details to programmer?  Or should I report it with good enough details and continue my testing?  I will answer this question to me.  All I wish is, I don't want the user of the app to experience the crash.  If there is a crash, as a member from development team my intent is to keep it to minimal number having little or no impact.  I see the crash is a great source of learning about my work in the app.

I used to be fussy about crash years back be it on desktop applications, database, web applications and mobile apps.  Now, I have come to point, I love them and it is absolutely okay for an app to crash and it should crash.  What I do post crash in fixing, that tells the bigger story.  In my work, the crashes have made the app better because the team was serious about those crashes.

What on having the crash?

Should the user lose the data what she or he entered on experiencing the crash?  I personally, don't want this to happen for me.  If it happens, I will be annoyed!  That said, how to handle it?  That's something we will have to sit with the programmers and team, for discussion.  

At what point in app, encountering the crash, should we close the app and start over again. At what point in app, it is okay to note the crash and pull the stacktrace, and let continue the user using the app with data entered intact. At what point in app, I should not show UI in view on entering crash and what to show, then resume over safely from there on.  Personally I feel this is a team effort and not just a programmer's effort in making it happen.  

Few testing strategies hack to uncover crash

Here are few things I do and I ask my fellow testers to do when testing mobile apps:

1. Using the test data which will check the data integrity in app at -- entry point, during processing and post processing.
2. Identifying the states of the app and passing the invalid states in app at -- entry point, during processing and post processing.
3. Identify the input which is not from a tester and user. Classify the input on which I don't have control. For example -- the incoming intent; the app responding to APIs (default values, entered values and processed values); the app receiving the response from APIs; the device state; app's activity life cycle state and data/state exchange, and many more as this.
4. Depending on Android or iOS app, much more strategies can be narrowed down to be specific and work.  At the end, what time I'm left with in the test cycle and what do business want, directs me on what to do.

Debugging and Investigation skills

There are libraries which collects the crash on exception along with other details as -- device info, user info and network info.  I have been with programmers and had difficulty in reproducing the crash and experience it in development environment.  This said, are the logs enough to fix crash?  May be, so we handle the exception and continue the flow of app in runtime.  But did we solve the root problem that caused the crash?  No!  This is where, I feel, the skill of a tester comes in and it is very much needed.

This skill defines me what I'm as a tester and value I bring to the system.

The Test Data Stopped Me from Committing a Blunder

Recently, I took up testing of a feature and it was one of the critical releases from a point of business. We had to create new systems technically and associate with the existing system for a flow to complete the business need.  This involved testing from at least 5 different endpoints other than different client endpoints.  But how to test it with the time I had? 

To test this effectively, I could have made use of existing data from the stage database and mapped it or dumped it to create a copy for the new system.  Then, I thought, let me create a fresh set of data.  The challenge for me in creating test data was:

1. I did not understand existing test data as it was not that straightforward and simple; most data were junk i.e. entered randomly, and created with automation aid to have some data, probably
2. It has to sample for 100000+ entries in production and its mappings to different fulfillment
3. It has to help in testing the newly built system in the unit wise and integrated system in each node wise
4. It has to help in figuring out the problems in each microservices and different client nodes
5. I had time of 2 days to prepare the test data
6. I had to understand the logic of how various associated microservices evaluates the same data for fulfilling the request
7. I had to understand how the data has to be constructed such that when it is fed to the cron job or scheduler, it will just pick it and insert them in the database and then associate the mappings
8. How to make sure this mapping is done right? How shall I test this mapping?
9. I will be getting almost 1 day to test this new feature on one client node after data creation.  How do I make sure the client endpoint is dealing it right along with other client nodes and microservices endpoints?
10. The products get shipped at the day end that is on the one day testing of a client node
11. What is the space I have to correct my error If I did in my testing now?
12. I was said not to test anything and just see if that functional flow is good enough to make one as usual fulfillment, to keep it simple one transaction in a business use case. Should I take this and just do this and nothing else?
13. I said to the team not to make junk data entry and call this as test data; I had stopped the team from entering data for two days in all interfaces
14. Data being created from automation in systems was posing challenges each minute during this time and I had to collaborate with different teams on the same while I had left with little time to complete my testing tasks and delivery from test team

I learn, my testing can be strong here if at all, only if I understand the data and how it gets fulfilled.  I decided to sit and create the test data.  While I created test data, I started referring to the production environment and created the data to simulate them.

I prepared 42 test data and I knew exactly what it is and I had sampled it enough to simulate 100000+ data.  I had taken close to 3 days for designing these data and associating it without the help of any cron job or scheduler.  I figured out how to evaluate these data in each node and microservice endpoints by this time.

This is the advantage of test data creation. I spent most of the testing time here i.e. in preparing the test data. It helped me very much!

With this, I had sets of tests that are "must" and any problem in these will be a blocker. In another set of tests, if there are any problems, then it can be called out by stakeholders if that has to be fixed or not for releasing the product.  I was left with 3 hours before they take a call of shipping the product, that day.  I had to cover my tests by this time.

It was about 2 hours left for the shipping decision, I noticed a behavior that seemed to be a release blocker. On informing it, I started looking at different client nodes to see if it existed there as well.  I could identify this blocker, because, I had designed the test data. If I had not designed the test data, it would have not been possible for me to identify it that day. One of the disasters probably in the product's and business history if this blocker bug had gone out as part of a release.  The functionality seems to work and fulfillment runs. But the feature is not doing what it has to do.  See how tricky it is - it functions but it does not do anything that it is supposed to do eventually. The disguise!

If I had just carried out the functional flow per communication I received from the engineering management, I would have done a blunder as a practicing tester. The business does not do this purposefully; they assume it will be handled and there is no need to test this.  If I have to make the same decision as a tester that is as engineering management and business, I should be convinced enough to say, there is no need to test this at this point in time given the context. I took it up by making time in available time to test each endpoint. 

The functionality flow seem to look good but the outcome was not what the business wants and it is not obvious anywhere in the GUI.  Debugging this behavior, I learned this is a problem. I figured out the root cause of it and shared it with the programmers.  I was given a new deployment to test for the same, later.  The shipment of the product did go on the same day. I wrote the RCA for the same on receiving a fix.

Why did I share this here?

1. Test data is an important part of a test
2. Time what we spend on creating tests and data, usually it is not seen as part of testing and not included in the time given for testing
3. Sampling and creating the test data is a skill to be improvised each day by the tester
4. When said don't test this, as a tester have enough confidence and rational analysis before not testing it; else, as a tester, we will be supporting the growth of the risks in product and risk for using the product
5. The creation of test data helps to know different endpoints of a technical system; if it did not help, then we are not preparing the right test data
6. Blockers, bugs, and ambiguity always exist; the tests and test data should help in making it simple to understand and know the root cause
7. If I had not worked on this test data, I would have not identified the blocker and it would not have been possible for me to test each end node of client and microservices

Now if the use cases are automated in this feature, the automation can assert for same on different end nodes because we know what is the data and what is expected out of it. There is no need for me now to do this as extensively as I did this time.  This is one of the benefits of automation which lets me focus on other stuff in testing.

Note:

• Never exaggerate the bug report saying technical investigation and root cause. Rather, we can provide, this is what we learn from our test investigation.
• Never exaggerate the test data and underestimate the test data. Anything associated with test data is worth investing time to look at it and know it better.