Problem Not From SDK; But, I Said, That's the problem. I'm Wrong!

Today's software is not like the software and hardware what I saw in 2006 to 2010.  I see the software which we are building today, it integrates third party software vendor's SDKs.  As well, today's software what we are building are not desktop application like in the 2000s; but the applications on the desktop and mobile which can connect everywhere else and serving the user.

Recently, I was testing a product which is in its initial state of MVP. It integrates four SDKs from different software vendors.  Among these, one SDK showed a behaviour which was unusual and the engineering team which I was part of, raised the request for support from that SDK's vendor.  While the behaviour what I observed in the product was uncertain and over a period of week, it took a stable behaviour.  When asked, I was said it is due to SDK.

Here is the miss, what I did for first. I took those words as I had got used to this behaviour for two weeks with no technical solution. I did not question enough that and debug around as the time what I had for testing was also crunched. I and other fellow tester owned the ownership of testing for the release. Stretching late nights and weekends, it all drove me to take words when said it is technical limitations and nothing can be done.  Here is the second miss what I made -- I took these words as the behaviour due to SDK and previous experience was consistent and synching very well.

The story got different in production.  One of the user reported the same behaviour and I was confident in my reply and said, it is as a known behaviour.  My other programmer friends had same opinion.  My programmer friends here are highly skilled and they know their job very well.  But we all were fooled by the software as the pressure which eventually starting building upon us. That was also due to the experience of us talking with that SDKs team.

I took up that behaviour for investigation again today after hearing from production and from a techie friend who said, "it should not happen for so long".  Yes, it should not happen for so long as I expect it should be gone after the synch is done, is what I think as well.

I started questioning myself and doubting what I have learned.  Isolated all the environments for my observations and started watching my actions, the requests, the responses, the pay load, the race conditions, the served and unserved data streams, the logs, and the breakpoints & values in code eventually as I did all these moves. 

I was wrong! Yes, I was wrong in what I said confidently to every stakeholders right form CEO, CTO to all other business stakeholders. Also the engineering team had the opinion what I had.  It was time to go back and tell them, "I'm wrong in what I communicated in behaviour of the client software and rational behind it".

I gave the reason why it happens, when it will happen, when it will not happen, the impact of the behaviour, work around for same in production, is it dependent on anything else and awaiting, and why it was a missed from our end technically.  It was not a problem from third party's SDK. It was a UI that got triggered each time when the activity got triggered. 

Though I have to investigate much lot for other problems in using that SDK, this behaviour is not due that SDK. It did work well in this case here.

What I want to say here?

• No matter how confident in the code and in tests, we will be fooled and are fooled each time
• There is no harm in doubting ourselves for second time on -- what we have listened; what we have ignored; what we have learned; what we have not carried technically while we observe the behaviour from the tests consistently
• Being non-technical in our observations and work helps a lot
• Knowing the benefits of not thinking technically each time
• Giving time to ourself so we sit back and investigate the behaviour which everyone claims it is due "that"

• If it is due to that, you will happen to learn about it so you can test it better
• If it is not due to that, then you will happen to learn it is not so and can figure out the cause of it; help team in fixing it

In this case I see no impact to user in terms of data and performance of product, but the experience is definitely annoyed.

So I learn again, that I should build the tests technically -- which will put the product into test for the experience of each feature; each UI; each actions of user on a UI; each interactions with backend and its outcome to the client; and evaluate the outcome of the same.

How will I do this? I can do this sitting in person and also with help of automation.  But the point is what is the scale where I have to be with my tests samplings to experience it and investigate further. This takes time to learn and it won't happen in crunch time of releases.  Having said this, it is not bad to go and talk with stakeholders and buy the time quoting the priority and impact of the behaviour. We testers will have to initiate and assist the stakeholders in learning when there is a need for it.

PID Cat - Reading the logs and test investigating the Android apps

Reading the logs and helping self to design better tests is one of the key skill for a tester, is my opinion.  I insist and encourage the testers to practice it. I assist in learning the log and making use of it.  If you are not sure whether the product you are testing has a log or not, it is time to know it.

It can be server logs, proxy logs, client logs, hardware logs, deployment logs, etc.  The log contents and type of logs will vary from product to product.  What goes into log will also vary and it depends on what is the log level set to print into log file.

It happens that the logging will be turned off or will be set to minimal saying it consumes the disk space and IO of the box where it exists. On the other hands, the DevOps and programmers usually monitor the logs in production.  The logs are one of the most useful utility to know whats happening in the production.  While they use it in production environment, can't we testers use the same in the staging and pre-production environments? Won't that help us to test better? Yes, it will help!

Understanding how the logging feature is written for the product is one of the key task usually missed by a practicing tester. The way of interpreting the observations and further test design will be influenced upon learning how to use the logs.  

In this post, I will share one tool which helps in context of testing the Android mobile apps. The logs and Android devices; if you say, "logcat", yes; it helps to print the logs in different logging levels. What if I want to pick for particular Android app from the debug stream of logcat log? It is not an easy task with logcat's output stream. 

I have been using the PID Cat for years now and I find it very useful. It is written by @JakeWharton  in Python.  It prints the logs in colored text and we have choice to change the color and choose what we want on our box in the program written. Upon that I can just fetch log for a particular Android application.  Also, we can contribute further to this tool. The readability experience of log on using the PID Cat, is better.  Read about the logcat color script from Jeff Sharkey, it is here.

How to get PID Cat?

On Mac OS X, the PID Cat can be installed via brew.  If brew is not configured on the Mac OS, configure it. On successful configuration of brew, using the below command the pidcat can be configured.

brew install pidcat

On Ubuntu, the PID Cat be installed by using below commands in terminal

sudo apt-get update

sudo apt-get install pidcat

On Windows, install the Python and configure the path. I have tested it with Python 2.7 and 3.x during my tests. It appears to work for me in my test environment.  

Below are the steps to use PID Cat on Windows machine.

1. Install Python and configure the path
2. Make sure the path of Python is set
3. Make sure the ADB is installed and the Path is configured for it
4. PID Cat is a Python program and it is available in above mentioned Git repository
5. Get this Python program file and save it locally on the drive
6. Go the drive where the pidcat.py is saved and enter the command in terminal -- python pidcat.py com.package.name
7. This should start printing the logs and on Windows 10 OS, I see colored log text

If using the Cygwin on Windows OS, install the Python package and make sure it is installed. Now go to root and access the directory 'cygdrive'.  From here choose the physical partition drive of Windows OS where the pidcat.py file is stored. Then use the below command to fetch log of specific app.

python pidcat.py com.package.name

Note that, for all the above said, the ADB has to be configured and set in path. If not, it will create trouble. The general log of Android device can also be collected via PID Cat, by using command -- python pidcat.py .

Android device logs in colored text in Windows OS via Cygwin

Android device logs in colored text in Windows OS terminal

Now having setup the PID Cat, while it is up and running, let us test and debug the apps much better.