The Test Data Stopped Me From Committing a Blunder

Recently, I took up testing of a feature and it was one of the critical release from point of business. We had to create new systems technically and associate with existing system for a flow to complete the business need.  This involved testing from at least 5 different end points other than different client end points.  But how to test it with the time I had? 

To test this effectively, I could have made use of existing data from stage database and mapped it or dumped it to create a copy for new system.  Then, I thought, let me create a fresh set of data.  The challenge for me in creating test data was:

1. I did not understand existing test data as it was not that straight forward and simple; most data were junk i.e. entered randomly; and with automation aid to have some data, probably
2. It has to sample for 100000+ entries in production and its mappings to different fulfilment
3. It has to help in testing the newly built system in unit wise and integrated system in each node wise
4. It has to help in figuring out the problems in each microservices and different client nodes
5. I had time of 2 days to prepare the test data
6. I had to understand the logic of how various associated microservices evaluates the same data for fulfilling the request
7. I had to understand how the data has to be constructed such that when it is fed to cron job or scheduler, it will just pick it and insert them in the database and then associate the mappings
8. How to make sure this mapping is done right? How shall I test this mapping?
9. I will be getting almost 1 day to test this new feature on one client node after data creation.  How do I make sure the client end point is dealing it right along with other client nodes and microservices end points?
10. The products get shipped at the day end of that -- one day testing of a client node
11. What is the space I have, to correct my error If I did in my testing now?
12. I was said not to test anything and just see if that functional flow is good enough to make one as usual fulfilment; should I take this and just do this and nothing else?
13. I said to team that no junk data entry and term this as test data; I had stopped team from entering data for two days
14. Data being created from automation in systems was posing challenge each minute in this time and I had to collaborate with different teams on same while I had left with little time to complete tasks

I leant, my testing can be strong here if at all, only if I understand data and how it gets fulfilled.  I decided to sit and create the test data.  While I created test data, I started referring to production environment and created the data to simulate them.

I prepared 42 test data and I knew exactly what it is and I had sampled it enough to simulate 100000+ data.  I had taken close to 3 days for designing these data and associating it without help of any cron job or scheduler.  I figured out how to evaluate these data in each nodes and microservice end points by this time.

This is the advantage of the test data creation. I spent most of the testing time here i.e. in preparing the test data. It helped me very much!

With this, I had set of tests which are "must" and any problem in these, it will be blocker. In other set of tests, if there are any problems, then it can be called out by stakeholders if that has to be fixed or not for releasing the product.  I was left with 3 hours before they take call of shipping the product, that day.  I had to cover my tests by this time.

It was about 2 hours left for shipping decision, I noticed a behaviour which seemed to be release blocker. On informing it, I started looking at different client nodes to see if that exist there as well.  I could identify this blocker, because, I had designed the test data. If I had not designed the test data, it would have not been possible for me to identify it that day. One of the disaster probably in the product's history if that had gone out.  The functionality seem to work and fulfilment runs. But the feature is not doing what it has to do.  See how tricky it is - it functions but it does not do anything what it has to do eventually. The disguise!

If I had just carried out the functional flow per communication I received, I would have done a blunder as practicing tester. The business does not do this purposefully; they assume it will be handled and there is no need to test this.  If I have to make the same decision as a tester i.e. as business, I should be convinced enough to say, there is no need to test this at this point of time given the context. Good that, I took it up by making time to test each end points. 

The functionality flow was good but the outcome was not what business wants and it is not obvious anywhere in the UI.  Debugging this behaviour, I learned this is problem. I figured out the root cause of it and shared it to the programmers.  I was given a new deployment to test for the same, later.  The shipment of product did go on same day.

Why did I share this here?

1. Test data is important part of a test
2. Time what we spend on creating test is part of the time which we take for testing
3. Sampling and creating the test data is a skill to be improvised each day by tester
4. When said don't test this, as a tester have enough confidence and rational analysis before not testing it; else, as a tester we will be supporting the growth of product's risk
5. Creation of test data helps to know different end points of a technical system; if did not help, then we are not preparing the right test data
6. Blockers, bugs and ambiguity always exist; the tests and test data should help in making it simple to understand and know the root cause
7. If I had not worked on this test data, I would have not identified the blocker and it would not have been possible for me to test each end nodes of client and microservices

Now if this is automated on this feature, the automation can assert for same on different end nodes because we know what is the data and what is expected out of it. There is no need of me now to do this as extensively as I did this time.  This is one of the benefit of automation which lets me to focus on other stuffs in testing.

Note:

• To a practicing tester, it is good to report the problem clearly. There is no need to solve it because we will not know exactly what it is and why it is as we will not have source code with us. But as practicing gets stronger in a project or with a product, few things can become clear technically and this will lead a tester to tell what is the root cause upon technical investigation. This takes practice, yet we can go wrong. So never exaggerate the bug report saying technical investigation and root cause. Rather, we can provide, this is what we learn from our test investigation.
• Never exaggerate the test data and at same never under estimate the test data. Anything associated with test data is worth investing time to look at it and know it better.

CI & CD - Exist to solve the engineering problems for first than to solve for customers

I read a tweet post of Michael Bolton. What made me to share my opinion as a blog post is, I hear CI and CD associated with Software Testing quite often; especially in the software testing conferences.

Here is what Michael Bolton asks on Twitter:

Your CI/CD system is probably pretty good at helping you discover problems in the build process. How does it support the discovery of problems in the product that would matter to customer, the team, or the business? Note - there CAN be really good answers to this. #DevOps #testing

The CI and CD is not associated just with Software Testing, but it is integral part of Software Engineering and Development.  In 2006, I have seen CI and CD. In 2017, I see CI and CD. The change is, how we are building software today, when compared to 10 years ago.  The CI and CD has taken upper hand for how we architecture, design and work in building software, before putting it to release line for market.

I want to say this -- "We seeing or associating CI and CD with testing is okay but it is not all about the testing. It is about the engineering and testing is one part of engineering." I see this emphasis is missed in the conferences when they quote CI and CD and tag a word testing with it, especially with automation context.  Note that, when I use word "automation" here, it is not the automation part of testing deliveries, it is the automation part of the engineering pipelines as well.  Yes, the *pipelines* of technology and engineering stack.

We have multiple pipelines in the engineering and from different engineering teams which may span over different geo locations and work culture.  With this, I say, for first CI and CD has to deliver and serve the engineering teams and also the teams which work with engineering team, before serving the market customer or business. If it is not serving here, then that CI and CD has a fundamental problem that needs to be addressed.

Before thinking of Testing as a focusable space in engineering and shippable product to customer, the testing can be seen in strong form and it is "Unit Testing" while product is being developed. Plus the integration of different pipelines outcome while the Unit Testing is executed or being executed or about to be executed, as the code get pushed. This summation of different pipelines integration constitutes a picture of product which gets shipped to customer, one day very soon.  Note that, this product is not yet available to customer in market, while it may be available to different stakeholders within the engineering team and teams associated to it. We need it to vision our work, vision our product and vision my testing for the product.

The CI and CD do not remove any problems nor add problems.  If not handled CI and CD, the problem can get disguised and may ship to market.  For example, did you see a force update of software you use?  The CI and CD, waits and look forward the human's push so it can do its job and give back to Build Engineer or any other engineer.

Testers, do not get panicked about how the word CI and CD is used with Software Testing and automation. The CI and CD is beyond this and it starts with the integration of different engineering technology stack pipelines outcome as one output and the unit tests.  

I don't know how true is this, I heard from my friends years back that Amazon builds for every 11 seconds. Could be and I'm not surprised with it. You see the CI and CD here? The compiled build or deployment does not mean, shipped.  It is the gateway for unit test run and integration of different pipelines.

The coverage and testability offered by CI and CD can vary based on how it is used. It is just like this -- if you noticed a tailor marking the textile to stitch a shirt; the measure of the marking again; cut it on the marks; tie it and give a number; the other tailor picks it and thread it; then pass it to other tailor to add and do necessary stuff before it is put to shirt hanger or cover for handing it to showroom or customer.  What the tailors did it here is the CI and CD. In large manufacturing, these tailors can sit in different places and can work in different time.  Where is the testing here? You answer it! How this CI and CD of tailors help customer to wear a comfortable shirt? You answer it!

Now, just think how many of these tests are technical tests facing tailoring skills and those tests which are facing the customer in the market on having a wearable shirt.

For the teams within engineering, when we have practices that benefits engineering to build better product, then CI and CD is of help.  Because with such engineering practices, the problems in integration of product and building it, can be identified earlier.  This can be identified if we have such Unit Tests written or built; else, it won't.

The CI and CD does not mean no bugs; but helps in identifying sort of bug(s) earlier in the engineering cycle.  These tests are more towards the technology and tech stack facing than facing the customer in the market.  But that does not mean we cannot have tests for customer facing in CI & CD.  This customer facing tests comes when we have the build which can be used by different types of users, stakeholders and testers in business team.  Example of tech facing tests, a MVP (Model View Presenter) not having defined logic container/controller association to pass the information to the viewer; this can be figured out if there is one such unit tests upon integration.  This bug may not be visible in UI if the contract of client and server partially satisfies; it may be seen when looked below the UI while doing the customer facing tests.

For me, be it tech facing or customer facing, test is a test. When tests can reveal product information earlier, why not to tap on it, if that can be achieved using the capabilities of the CI & CD ?

Providing the build having technology facing tests to business teams or SMEs, may not get that powerful feedback always to engineering team in initial stages of development. I have witnessed such outcomes when said about UI is not right, alignment is not right, keep it bold, keep it in lower font size., etc.  Though these are useful feedback, when it is being received is the point here.

To end this post, the CI and CD is first to address the internal engineering problems and assist in having effective engineering, before solving problems to the customer or business.  CI & CD is C[I+D] before seeing a product and problems it can have.