The Test Data Stopped Me from Committing a Blunder

Recently, I took up testing of a feature and it was one of the critical releases from a point of business. We had to create new systems technically and associate with the existing system for a flow to complete the business need.  This involved testing from at least 5 different endpoints other than different client endpoints.  But how to test it with the time I had? 

To test this effectively, I could have made use of existing data from the stage database and mapped it or dumped it to create a copy for the new system.  Then, I thought, let me create a fresh set of data.  The challenge for me in creating test data was:

1. I did not understand existing test data as it was not that straightforward and simple; most data were junk i.e. entered randomly, and created with automation aid to have some data, probably
2. It has to sample for 100000+ entries in production and its mappings to different fulfillment
3. It has to help in testing the newly built system in the unit wise and integrated system in each node wise
4. It has to help in figuring out the problems in each microservices and different client nodes
5. I had time of 2 days to prepare the test data
6. I had to understand the logic of how various associated microservices evaluates the same data for fulfilling the request
7. I had to understand how the data has to be constructed such that when it is fed to the cron job or scheduler, it will just pick it and insert them in the database and then associate the mappings
8. How to make sure this mapping is done right? How shall I test this mapping?
9. I will be getting almost 1 day to test this new feature on one client node after data creation.  How do I make sure the client endpoint is dealing it right along with other client nodes and microservices endpoints?
10. The products get shipped at the day end that is on the one day testing of a client node
11. What is the space I have to correct my error If I did in my testing now?
12. I was said not to test anything and just see if that functional flow is good enough to make one as usual fulfillment, to keep it simple one transaction in a business use case. Should I take this and just do this and nothing else?
13. I said to the team not to make junk data entry and call this as test data; I had stopped the team from entering data for two days in all interfaces
14. Data being created from automation in systems was posing challenges each minute during this time and I had to collaborate with different teams on the same while I had left with little time to complete my testing tasks and delivery from test team

I learn, my testing can be strong here if at all, only if I understand the data and how it gets fulfilled.  I decided to sit and create the test data.  While I created test data, I started referring to the production environment and created the data to simulate them.

I prepared 42 test data and I knew exactly what it is and I had sampled it enough to simulate 100000+ data.  I had taken close to 3 days for designing these data and associating it without the help of any cron job or scheduler.  I figured out how to evaluate these data in each node and microservice endpoints by this time.

This is the advantage of test data creation. I spent most of the testing time here i.e. in preparing the test data. It helped me very much!

With this, I had sets of tests that are "must" and any problem in these will be a blocker. In another set of tests, if there are any problems, then it can be called out by stakeholders if that has to be fixed or not for releasing the product.  I was left with 3 hours before they take a call of shipping the product, that day.  I had to cover my tests by this time.

It was about 2 hours left for the shipping decision, I noticed a behavior that seemed to be a release blocker. On informing it, I started looking at different client nodes to see if it existed there as well.  I could identify this blocker, because, I had designed the test data. If I had not designed the test data, it would have not been possible for me to identify it that day. One of the disasters probably in the product's and business history if this blocker bug had gone out as part of a release.  The functionality seems to work and fulfillment runs. But the feature is not doing what it has to do.  See how tricky it is - it functions but it does not do anything that it is supposed to do eventually. The disguise!

If I had just carried out the functional flow per communication I received from the engineering management, I would have done a blunder as a practicing tester. The business does not do this purposefully; they assume it will be handled and there is no need to test this.  If I have to make the same decision as a tester that is as engineering management and business, I should be convinced enough to say, there is no need to test this at this point in time given the context. I took it up by making time in available time to test each endpoint. 

The functionality flow seem to look good but the outcome was not what the business wants and it is not obvious anywhere in the GUI.  Debugging this behavior, I learned this is a problem. I figured out the root cause of it and shared it with the programmers.  I was given a new deployment to test for the same, later.  The shipment of the product did go on the same day. I wrote the RCA for the same on receiving a fix.

Why did I share this here?

1. Test data is an important part of a test
2. Time what we spend on creating tests and data, usually it is not seen as part of testing and not included in the time given for testing
3. Sampling and creating the test data is a skill to be improvised each day by the tester
4. When said don't test this, as a tester have enough confidence and rational analysis before not testing it; else, as a tester, we will be supporting the growth of the risks in product and risk for using the product
5. The creation of test data helps to know different endpoints of a technical system; if it did not help, then we are not preparing the right test data
6. Blockers, bugs, and ambiguity always exist; the tests and test data should help in making it simple to understand and know the root cause
7. If I had not worked on this test data, I would have not identified the blocker and it would not have been possible for me to test each end node of client and microservices

Now if the use cases are automated in this feature, the automation can assert for same on different end nodes because we know what is the data and what is expected out of it. There is no need for me now to do this as extensively as I did this time.  This is one of the benefits of automation which lets me focus on other stuff in testing.

Note:

• Never exaggerate the bug report saying technical investigation and root cause. Rather, we can provide, this is what we learn from our test investigation.
• Never exaggerate the test data and underestimate the test data. Anything associated with test data is worth investing time to look at it and know it better.