Wednesday, December 20, 2017

The Test Data Stopped Me From Committing a Blunder

Recently, I took up testing of a feature and it was one of the critical release from point of business. We had to create new systems technically and associate with existing system for a flow to complete the business need.  This involved testing from at least 5 different end points other than different client end points.  But how to test it with the time I had? 

To test this effectively, I could have made use of existing data from stage database and mapped it or dumped it to create a copy for new system.  Then, I thought, let me create a fresh set of data.  The challenge for me in creating test data was:
  1. I did not understand existing test data as it was not that straight forward and simple; most data were junk i.e. entered randomly; and with automation aid to have some data, probably
  2. It has to sample for 100000+ entries in production and its mappings to different fulfilment
  3. It has to help in testing the newly built system in unit wise and integrated system in each node wise
  4. It has to help in figuring out the problems in each microservices and different client nodes
  5. I had time of 2 days to prepare the test data
  6. I had to understand the logic of how various associated microservices evaluates the same data for fulfilling the request
  7. I had to understand how the data has to be constructed such that when it is fed to cron job or scheduler, it will just pick it and insert them in the database and then associate the mappings
  8. How to make sure this mapping is done right? How shall I test this mapping?
  9. I will be getting almost 1 day to test this new feature on one client node after data creation.  How do I make sure the client end point is dealing it right along with other client nodes and microservices end points?
  10. The products get shipped at the day end of that -- one day testing of a client node
  11. What is the space I have, to correct my error If I did in my testing now?
  12. I was said not to test anything and just see if that functional flow is good enough to make one as usual fulfilment; should I take this and just do this and nothing else?
  13. I said to team that no junk data entry and term this as test data; I had stopped team from entering data for two days
  14. Data being created from automation in systems was posing challenge each minute in this time and I had to collaborate with different teams on same while I had left with little time to complete tasks

I leant, my testing can be strong here if at all, only if I understand data and how it gets fulfilled.  I decided to sit and create the test data.  While I created test data, I started referring to production environment and created the data to simulate them.

I prepared 42 test data and I knew exactly what it is and I had sampled it enough to simulate 100000+ data.  I had taken close to 3 days for designing these data and associating it without help of any cron job or scheduler.  I figured out how to evaluate these data in each nodes and microservice end points by this time.

This is the advantage of the test data creation. I spent most of the testing time here i.e. in preparing the test data. It helped me very much!

With this, I had set of tests which are "must" and any problem in these, it will be blocker. In other set of tests, if there are any problems, then it can be called out by stakeholders if that has to be fixed or not for releasing the product.  I was left with 3 hours before they take call of shipping the product, that day.  I had to cover my tests by this time.

It was about 2 hours left for shipping decision, I noticed a behaviour which seemed to be release blocker. On informing it, I started looking at different client nodes to see if that exist there as well.  I could identify this blocker, because, I had designed the test data. If I had not designed the test data, it would have not been possible for me to identify it that day. One of the disaster probably in the product's history if that had gone out.  The functionality seem to work and fulfilment runs. But the feature is not doing what it has to do.  See how tricky it is - it functions but it does not do anything what it has to do eventually. The disguise!

If I had just carried out the functional flow per communication I received, I would have done a blunder as practicing tester. The business does not do this purposefully; they assume it will be handled and there is no need to test this.  If I have to make the same decision as a tester i.e. as business, I should be convinced enough to say, there is no need to test this at this point of time given the context. Good that, I took it up by making time to test each end points. 

The functionality flow was good but the outcome was not what business wants and it is not obvious anywhere in the UI.  Debugging this behaviour, I learned this is problem. I figured out the root cause of it and shared it to the programmers.  I was given a new deployment to test for the same, later.  The shipment of product did go on same day.

Why did I share this here?
  1. Test data is important part of a test
  2. Time what we spend on creating test is part of the time which we take for testing
  3. Sampling and creating the test data is a skill to be improvised each day by tester
  4. When said don't test this, as a tester have enough confidence and rational analysis before not testing it; else, as a tester we will be supporting the growth of product's risk
  5. Creation of test data helps to know different end points of a technical system; if did not help, then we are not preparing the right test data
  6. Blockers, bugs and ambiguity always exist; the tests and test data should help in making it simple to understand and know the root cause
  7. If I had not worked on this test data, I would have not identified the blocker and it would not have been possible for me to test each end nodes of client and microservices

Now if this is automated on this feature, the automation can assert for same on different end nodes because we know what is the data and what is expected out of it. There is no need of me now to do this as extensively as I did this time.  This is one of the benefit of automation which lets me to focus on other stuffs in testing.

  • To a practicing tester, it is good to report the problem clearly. There is no need to solve it because we will not know exactly what it is and why it is as we will not have source code with us. But as practicing gets stronger in a project or with a product, few things can become clear technically and this will lead a tester to tell what is the root cause upon technical investigation. This takes practice, yet we can go wrong. So never exaggerate the bug report saying technical investigation and root cause. Rather, we can provide, this is what we learn from our test investigation.
  • Never exaggerate the test data and at same never under estimate the test data. Anything associated with test data is worth investing time to look at it and know it better.

No comments:

Post a Comment

Please, do write your comment on the read information. Thank you.