Testing for Web Performance: Why I want to share ?

I was practicing with a fellow tester and he expressed that he wants to practice performance testing. I too have that wish i.e. to practice performance engineering and testing. I try to find the opportunity each time to practice this and look out for fellow testers and mentors who can assist me here. However I do not wait for mentor now though I want one. Instead I will start my practice and I learn from my practice as I practice.

May be, as me there could be other testers who wants to practice the performance testing. I wish to share my learning and practice here, in the simplest and short way.  

I have heard Rahul Verma saying, "replace the word 'performance' with the word 'capability', when you hear a word performance". I feel, it makes more appropriate and meaningful to look it as a 'capability' when said 'performance'.

With the label 'Web Performance', I will share what I'm learning, practicing and thinking for web performance in last few years. When said, 'web', it is not just the website; it is the web that connects different interconnected systems. I will be sharing my learning on one aspect of web which is connected and communicating via HTTP protocol.

My learning from Agile Testing Days Asia 2017

I had got an opportunity to attend Agile Testing Days Asia 2017 (ATDA-2017). I thank my friend Jyothi for sharing me this opportunity.  Before I happen to say much, I request the reader to go through this post.  I will share what I felt on being part of this conference and what did I take back to my desk so I can practice my testing better than before.

I'm a person who adapts to an environment of project and do what is needed for delivering the best. Whether I'm in a project which claims to work on Agile principles, manifesto and its various methodologies or in a project that claim to be on other methodology, principles and process, I work to see what best I can do there in context.

With that, I have no idea if Agile is successful always or not. I have an idea -- the teams working together to accomplish will get it when each assist each others to deliver.

I have worked with teams which claims to be on Agile. Still being on Agile, I have worked beyond office hours, in weekends, in pressure, and yet seeing the release delayed beyond timeline. There are all problems here as exists in projects which claim to be non-Agile i.e. before Agile got adopted to Software Development. Its all in understanding, solving and going forward is what I see irrespective of principles and methodology.

I don't say Agile is wrong or effective or not effective. I learn, it is we people who adapt it, will do it something else be Agile or non-Agile. Now, I will stop talking on process or Agile here as it not my area of interest for now.

ATDA-2017 conference was organized by STeP-IN Forum and I should be thankful to them. It is not an easy job to organize a conference. Getting people together by taking care of their presence, giving space for networking with the fellow testers of different organizations and helping to carry the thoughts back to work place is not an easy effort.  I thank you STeP-IN Forum, your people and the committee. Respects!

Coming back to conference, I attended Conference Day 1 and Conference Day 2.  The audience were from around India, Bangladesh, Canada, Romania, Spain and USA.


The day 1 had below talks

1. Evoke the Soul of Agile - Keys to Lasting Transformation, by Selena Delesie, Leadership & Innovation Coach, Speak & Soul Igniter, Delesie Soljtions Inc.
2. Leveraging Global Talent for Effective Agility, by Todd Little, Agile Leadership Consultant & Innovation Software Executive
3. Agile Testing of Microservices, by Praveen Kumar, Manager-Agile Practice and Manoj Kumar Nagaraj, Director, Capgemini
4. The Agile QTOPIA, by Rahul Verma, CTO & Founder, Test Mile
5. How to avoid Internet of Insecure Things, by Gaurav Maheshwari, Engg. Manager - Software and Prashant Jain, Project Manaager, TVS
6. Building Inner Trust - there is no agile without confidence, by Ginna Encache, Founder & President, CHOICE
7. Rev up to Agility, Apply Lean Six Sigma in Testing, by Ramesh BR, Sr. Director, OpenText
8. Agile Games -- to build a jeep

The day 2 had below talks

1. Divide and Conquer: Easier Continuous Delivery using Micro-Services, by Carlos Sanchez, Engineer at CloudBees, Member - Apache Software Foundation, Startup Technology Advisor
2. What New Approaches to Software Can Teach the Enterprises, by Puneet Gupta, Global CTO, Brillio
3. Agile and Startups - What can go wrong - A case study, by Vipin Jain, Director QA, Astegic Infosoft
4. All sprints are guilty unless proven innocent, by Deepak Chopra, VP, Genpact
5. Testing in a Responsive Enterprise, by Abhishek Johri, Agile Coach, TEK Systems and I remember other co-presenter was by name Mahesh, consultant.
6. Agility for The Last Mile, by Diwakar Menon, Founder, Last Mile Consultants
7. There is no such thing called Agile Testing - Debunking Rituals and Ceremonies, by Srinivas Kulkarni, VP, JP Morgan
8. The Big Fight - Cirque (Dangal) -- Agile - to be or not to be. Debaters -- Raj Netravati (moderator), Selena Delesie, Rahul Verma, Pradeep Soundararajan, Jayapradeep Jiothis, and Jayaprakash Prabhakar

Here is my experience of Day 1

• I understand, Agile practice is a culture. It is very much essential to have culture that helps an organization and team grow and deliver. In doing so, the process come in; how to make use of process and tailor it to accomplish, is the key. This was iterated probably in one or other way by every presenters giving an example of their work and process brought in to the project or work place. I did not see any thing about testing. Rather it was all about the process.
• Todd Little shared about domain knowledge importance in programming and testing from one of his project relating Petroleum Industry. Further he went ahead and shared how collaboration is important when working with teams situated in different geographical locations.
• A contrasting presentation was from Rahul Verma and to me is evident as I see him presenting in similar lines. I was not of surprise, however the audience had tickling and laughs as he expressed his thoughts. It was around the process, culture and mindset.
• About security in IoTs, it was more about the practice and guidelines. I expected to see testing here. It did not happen for me. I repeat it is for me. I'm not sure about others. And this was the only presentation which did not have the word 'Agile' in presenters talk.
• Gina Encache shared about importance of people relationship in project and working place. It was her first talk and glad that she made it in India.
• Ramesh, gave the case study of his work which had to with process and illustrated how it benefited team in saving time and deliver. Nice! He showed his team members photo to the audience and that is remarkable! But again I was disappointed as I did not see testing here. It is about process. 
• Game session was interesting. It had three teams and each were given stuffs to build a jeep. I enjoyed watching it.  The collaboration, communication, team efforts, and one goal to accomplish helped all to stick together. Interestingly, here team did not follow any process however it was time boxed activity and team members spoke what they have to do and in what order. This was an example probably which the conference should have highlighted having each team to talk on second day of conference. May be the organizers did not think about it. I enjoyed this activity.
• Atul Khatri presented his stand up comedy. He did say, that we will recollect his jokes after two or three days and it gives tickle and brings the laugh.
• Followed this, it was a announcement of winning team who designed the jeep in given design constraint and time. 

Here is my experience of Day 2

• It looked as continued journey for me from Day 1.
• I did not get how to detail out testing the Micro-Services. It was more about the infrastructure and concepts. For one who hasn't heard about micro-services, this session will help to get a idea at a conceptual layer.
• Vipin Jain, had a case study and it was around a process and the impact if not understood how to use process.
• After this, all talks were on process and how to go with process so one can be informed well before a huge cost in the business.
• Testing in responsive enterprise, I was waiting for this talk. Eventually, this too was on process discussing on latest trend in software development and its methodologies.
• Following this, there was a debate about being Agile or not to be.  Each person in debate seat had their views on Agile, tools, and Continuous Integration.  But, none of them here spoke about testing! That was too sad. I expected at least they bring up point asking where is testing in all what we have heard for two days around Agile. It did not happen. See, how one gets into process trap while all are talking around the process.
• Shrini, I have seen him presenting and sharing his thoughts. It was no surprise for me. I did expect something from Shrini which would have spoken about testing execution. The only presentation which said word "testing" quite often along with the words "process" and "Agile".

My observation

• The conference was Agile Testing Days, but it did not have testing in it i.e. in two days of conference.
• Most of the speakers were of management level and was on process thoughts.
• Whereas most of the audience were in execution level i.e. Testers who do testing. See the gap of presenter and audience. 
• Process, process, and process.  The same is heard in office and same was heard in conference. Where is the testing? Or, where is the automation? How to do it? An example and demonstration of one such problem went missing in the conference.
• Testers want to know and seek help in doing their task better. At least I look for that than hearing about the process. I did not get that. 
• The game activity in the conference is clear example for collaboration and deliver as a team in knowing what is the goal. In the game session all were agile and did what their role expected do to in the team for building a jeep by assisting each others.
• I do not remember if there was much discussion on The Four Testing Quadrants and how to work on it, if Agile Testing was being discussed in Agile Testing Conference. As well I remember, just one liner mention in Shrini's presentation on the four quadrant of testing.

My humble appeal to STeP-IN committee

• I wish to share my honest opinion here as a practicing tester and seek your help to practice better. Please do not feel bad on me.
• I see the most of the audience in conferences were to be practicing hands-on testers in desk. It is they who execute and inform the stakeholders (managers and management) about the risk. When we testers better our testing skills, eventually we will learn how to deal with process and principles that comes with any name. Thereby we actually assist the culture of the project and company as well to get better. That said, I request to have more of testing in conferences than process and management.
• If looked into presenter and audience, you will see the gap clearly.  Please do encourage and pick the hands-on testers. Give more slots to these testers while we give few slots to management people. Ask for testers to come up and speak about their testing, automation, problem solving and innovations. This is what we need, importantly, to march forward in the testing practice as a start.
• Having management people talk all the way is good until we testers get what we want to get. If not, it will be boring, at least for me.
• To the panel which selects the paper or topic for presentation that come to them, please do keep the practicing testers in mind. See what is that we testers need and get that. The orientation of problem for tester and management is of different kind. You see, management sends their testers to conferences and not most of their management to conferences. I'm not saying to eliminate management talks but that should not fill the space and eliminate testing all together.
• I wish to see audience feel happy for coming to conference and say it was very useful and recommend it for themselves and others.

I took below to my desk from this conference

• To focus on my testing. Learn how to make use of process to its best for delivering by adapting it to the context on tailoring.
• Convey the same to fellow testers on desk and work along with them to deliver useful testing which is of value and creates demand for itself.
• I said myself, testing is agile in itself when understood the purpose of testing. So is the automation when it is to assist the testing. If this is consistently learned by me on each day, it will help me to adapt and tailor for any process to do better.

When not to test? When is the no need of testing?

I saw a tweet interacting with me and it was from my colleague Pradeep Lingan (PL). He had shared a question -- When to start test once the build arrives?

When to start test once the build arrives? @testingGarage @moolyatesting

— Pradeep Lingan (@Pradeeplingan1) March 15, 2017

I was in silence for few second on reading this question. In silence, I asked myself, "Why should I test? Why at all I should start testing in whatever timeline of product development?" Now, I have two questions, the one from PL and the other which I ask for that question.

The question asked by PL is not always easy to answer. But it is a very logical and technical question which should be answered. If not answered, stakeholders involved and those who are expecting 'valuable' (if expecting valuable) from testing will happen to ignore testing.

I wanted to ask PL what is the context in which he is asking this question. You see, it is not a simple question. It can be better understood when known the context which raised that question.

As a practicing tester, I understand, when there is contextual question, it is also possible to derive questions which is context free. When I say context free questions, it is the question which is generic and can be used along with contextual questions.  Is context free is also a context? I will leave this to you for brainstorming.

I will not ask PL for context here. I will answer my thoughts using context free learning approach here. From there, I will let PL to brainstorm with context for the question he asked.


Why we test? When is "the need" for testing?

I learn, I test and purpose of the Software Testing is,

• To provide information on product, so authorized stakeholder make sound informed decision on product's development, scope of development, pause, abort and release
• Information on testing the product can be -- about it's quality criteria and risks around it; and about the risk of carrying out and not carrying out the testing in context
• Information obtained from testing can be used for -- to make decision about the product; about the decision which stakeholders going to make and risks out of it; if it is worth to pay for tester, testing team and seek the information which is being received; are we investing too much for information that's not worth; prioritization; should we carry out much more testing and better testing?  etc.

Software Testing is not about fixing. Software Testing is to assist the fixing in better, rational and empirical way. The information obtained out of testing to be used for bettering the product. That means if carried out testing, it does not mean, it improves product. When information out of testing is used to better the product by fixing, it helps the product to improve.

The above points should have helped now to start understanding the Software Testing and its service to development of product and stakeholders. If I and you mutually agreed for above points, then you are looking for information which can ruin the existence of product or purpose of building the product or something that will incur loss to everyone in the business.

When not to test? When is "the no need" for testing?

On reading why we test and when testing is needed, it is quite straight simple to know when not to test and when is there no need for testing.  This can be used as a start point to decide whether to start testing or stop testing.

In below cases, it is useful not to test

• When a tester and testing team has no clarity in knowing -- what is useful testing and not useful testing for stakeholder
• When we don't know about risk(s) on which the testing has to focus in product and provide information
• When we have no clarity on what is expected out of testing
• When a tester or stakeholder feel the outcome of testing being carried out is not going to help in making the decision
• There is no area or a question in the product for which testing help to see an answer
• When no one is interested and curious in outcome of the testing

In below cases, there is no need for testing

• When the stakeholder learns or believe and say there is no risk in product, and say to stop testing or denies testing for product
• When the outcome of testing will not change the decision of stakeholder
• When no information is expected out of testing
• When a tester or stakeholder feel the outcome of testing being carried out is not going to help in making decision
• There is no area or question in the product for which testing helps to see an answer
• When no one is interested and curious in outcome of the testing

It is business and business is also on money and time. As a tester it is my responsibility as well to help make decision when testing is not needed and when not to test. For doing this, I need to build the credibility with stakeholders.

However, as a tester, I continue to uncover risks if that really helps stakeholder and if I'm sensing it from my testing. While this is being done, it is also good to note, time of tester is also important as it is value and money. If I think my investment on time is needed here, I will do it, provided I see a good cause and benefit there.

Did that help now in context free way to decide when to start testing once the build is available or yet to be available? Further, build the contextual reasoning to figure out when to start, when to stop and no need for testing when there is a situation.

Disclaimers for the blog posts under this label - "Testing and Conferences"

I want to start writing on what I feel, and what I took back from attending a conference.  I attend conferences that is oriented majorly towards Software Testing and also on programming.  I ask to myself, "Why I'm not sharing to organizers about the experience I had attending the conference as a audience or presenter?"  Never I shared though I filled or wrote the feedback form if given during the end of conference or end of the day in a conference.

I'm not sure if I will be hated, or (and) opposed, or (and) seen as unusual for what I share. But when I share, I do with honesty and seeking the good for me and for all practicing software testers from each one of us; that includes me as well.  With this, I want to keep this post as disclaimer or pointers where one has to read if she or he has different thoughts for what I share in the successive blog posts under this label.

I want to convey the below points in list of disclaimers

• Opinions or experience I share under this blog label are my own. It has nothing do with others. I will be solely responsible for what I say here.
• I do not mean to downgrade or oppose or pull down the fame and legacy of any forum or association or group or practitioners. Instead, I wish to be a honest tester who shares what I feel in respect for the efforts the group has put in making a conference for software tester.
• I do not intended to speak about any speakers or presenters in conference. It is a skill to present to the audience and I admire it; I want to be better and grow here. If one is coming up on to the stage and presenting, she or he has to be encouraged and I'm one such encourager with empathy.
• I never talk on a person i.e. individual nor on their experience or work. If I did so, I will be too bad for doing it.  Rather, I will share what I felt as overall when I want to carry back from conference to my desk to implement in my practice.
• My sharing will be to the conference organizers, panel who sets up theme or topics presented to audience. If the conference audience observe my thoughts are relevant, I would not be surprised. I speak from my tester perspective who is hands-on.
• Organizing a conference or a talk is not a simple job. I know the efforts it takes. The posts never point on that efforts. But, I will share what was beneficial; what would have been beneficial to me or testers similar to me and to people who wants to see value out of testing.

Note: I keep updating these disclaimer pointers as I learn.

At any time if you think, I'm doing wrong or have misunderstood, please do feel free for conveying it to me. My email id is ravisuriya1 at gmail dot com.  I'm human and I'm fallible, yet I want to be better each time; if you happen to assist me to do it, I will be thankful to you.

Native, Web and Hybrid - How related and different these apps are?

During discussion on mobile apps with a tester, one of the question was - "What is native app? What is the difference between hybrid app and native app?" Like one uses the mobile apps in and out almost all days on mobile devices, yet there is confusion in knowing what these apps are and the differences among them.  In this post, I do not share about advantages and disadvantages for the types of mobile apps, rather I share what I understand for what they are.  And, this should help to start learning on advantages and disadvantages if any.

Further knowing what is the technology stack being used for the native mobile app, web app and hybrid mobile app, a tester can design the tests better.  This helps to figure out the testability and from here one can get the idea which are good to automate and assess, and those which requires human skills to assess further and test.

Mobile web app and mobile device

I will walk through briefly on native mobile app and hybrid mobile app in this post. Before I do that, let me tell bit about web app with Responsive Web Design (RWD) for mobile devices.  Do you know any website that you use often? Open that site on your mobile device browser now and experience how it is rendered. Now, why not changing the orientation of your device and view it in portrait and landscape. What was your experience? Were you able to use it on mobile device? Did it render in same width and height as in desktop browser or it gut customized to your mobile device's screen?

Along with RWD, HTML5 and latest web technologies of JS with CSS, their exists frameworks to build web app for mobile which uses the nativity of mobile devices. They are evolving and exists, today.  One such example is Ionic framework.

While mobile OS extends the support of web view to display the web contents within the activity of an app, the above said web technologies and framework supports to build mobile web app.  With this the mobile web app can make use of device's hardware and sensor, as a native mobile app does.

This mobile web app is not available as an app in the Google's Play Store and Apple's App Store. However it can look like a mobile app in an experience but the implementation way differs. When a website can be rendered to mobile device's screen size, is it possible now to get a feel like not a browser based website though address bar is shown?  Frameworks available today to build the mobile web app will help to do so.

Some examples for the mobile web app are GMail, Flipkart, Amazon, and Facebook.



Native app and mobile device

Mobile app which is installed on mobile device and that can make use of hardware and sensor available on mobile device, is typically a native app.  This app is built for a particular mobile device's OS and platform.  With mobile device's few apps come pre-installed and user can still look for an app what she or he needs on Google's Play Store and Apple's Play Store.

A native app can access what it requires on mobile device provided it has the permission to do so. Now make a list of permissions you have seen which app asks you for installing and using it on the mobile device.

Some example for the native mobile app are Contacts, Message, Camera, Calendar, and the one which can be downloaded from the respective store of Google and Apple.

Note: In this context, I'm taking Google and Apple platforms as it is well known to most.



Hybrid app and mobile device

In above said, HTML5 and JS technologies have the major role in mobile web app. Now, the same can be used to build apps which can be installed on Android an iOS?

Technically the hybrid app will have HTML5 web app within a native wrapper which is supported by the mobile OS.  This mobile app is developed with HTML5, JS technologies and CSS, then compiled into native Android or native iOS or any other mobile OS using wrapper support.

Apache Cordova is one such mobile application development framework which  helps to wrap up an app that can run in WebView by having nativity extension to mobile device's (OS) platform. The plugins of Cordova can be used to access the device's hardware support. On knowing the word as Cordova, now if happened to explore further we will hear the words -- Phonegap, Titanium, Ionic, Xamarin etc. Knowing how they support to develop mobile application will help to understand better about the hybrid mobile app.

If the hybrid mobile app is also a kind of web app, then the HTML, CSS and JS files are stored in the app which will be installed on the device? Or, will it get from the server on launch of the web on a desktop browser?  This depends on how the hybrid app will be designed. If it is packaged into the .apk or .ipa file, then web APIs can be used send and received data.  If it is received from server then a layer around the WebView will have to built and web APIs will be still used.

The apps built in this sort are called hybrid because

• they are neither native apps as the rendering is on via WebViews instead of platform's UI framework
• they are neither web apps because they are packaged as an app for distribution across platforms using the platform nativity via its APIs

Have you used, GMail app on your phone by downloading it from store? Are you using Instagram, Facebook, Youtube, Maps apps on your mobile device on downloading it from store? Then that is an example of the hybrid app which combines the strength of native app and mobile web app together.

So whats the point

To end this post, if asked what is best i.e. whether native app or hybrid app, it depends on context and what is trying to be solved with negotiation.  Both helps to accomplish the needs of app developing organization, programmer, tester and user.  While each have they their own strength and weakness, it is the choice to be made by people who builds for whom.  In this post, I have not shared the advantages and disadvantages of it.

Now starts the fun of exploration around testing and automation.  How to test the apps built in hybrid way? What are the level of isolation I can identify so that I can layer it out layer by layer and test? Where to start if it is getting packaged into nativity of the platform on mobile OS?

If you are doing it or figured it out, pull me to pair with you and test together. Let us do it!

The Testability and a Tester

In one of the recent discussion with fellow testers, the topic was "testability".  While fellow testers asked, "what is testability?", I had a question which I shared in the discussion -- "What is codability?".


Codability and Testability

I see both of these are inter related.  If it is codable then it is testable to a degree.  What is that degree? That's the question of interest.  But, what is that one will accomplish by knowing the order of degree, here?  This forms the base in having the curiosity or wish to know about testability.

I have a product's feature which I have to implement. The feature has cases which I need to handle in run time for different situations which a user can encounter. Well, I can code for situations which I can think and which is of priority from point view of using the product. Then for other situations which I'm not seeing or not thought off, will it be a hit for my product?  What is the complexity level of the code I'm writing for this feature and what are the risks that are subtle in code I have written? The written code will always have the risk that is tied along with it.

Now, can I tell that testability is influenced by multiple factors and not just on the basis of the test identified? Well, after knowing this, it is essential to understand -- "testability is the easiness at which I can test the system in a given situation.".


Factors influencing testability

The few factors which will potentially influence the order of  easiness -- in learning the test challenge, identifying the test, approaching the test and executing it, are as below:

• The testing skills of a tester
• The programming skills of a tester
• The technology skills of a tester
• The experience of a tester in testing such systems
• What is know by tester about the product -- purpose of product; people who are building the product; people who will use or who are using the product; the features and functionality of product; technology of the product; code written for the product; how the product is built with limitations from business team; information available about the product; process, practice and culture followed by teams in building product; availability of the system and its present state
• Time available for system development
• Time available for testing
• The level or order to which the testability aid is built and can be provided in system to test, and the complexity or simplicity level of same
• Measures taken to improvise and include testability aiding stuffs in system
• Existing tools or utilities for testing the product or code, or to build one such utilities
• The freedom to tester and testing to control and guide the testing activity
• Information expected out of testing and priority of it
• Relationship with fellow testers, fellow programmers, business teams and any other who are involved and interested in the system being programmed and tested
• The audience interested in test report or outcome of testing and their expectation

Testability and Test Coverage

The test coverage accomplished by a tester in given context is relational to the skills set of tester and the testability factors of the system and environment.

While I have understood this, I tried to learn the scale of testability keeping the test coverage as base. Here is what I have observed for now.

By keeping the test coverage as a base scale might mislead me to learn what I can accomplish in coverage. Because, the testability might be available and easy one for me to accomplish the 10th level of coverage with my skills and other influencing factors of testability. But to cover and accomplish the first three levels can be not the easy one given my skills and other influencing factors of testability. The vice versa is also possible.  Here, I can take longer time to learn and know how could I have cut short and still accomplish the same coverage given the testability level to me in a context, if I had learned about the complexity level of of it. While I reach to a intended level of coverage, I would have done the moves which does not favor business timeline, learned from them and repeated few stuffs.  This is not wrong but I have consumed time which I could have invested in testing to accomplish much more coverage.  The same is represented in below image.

Keeping the test coverage in base and assessing the testability on vertical scale marking the breadth and width of coverage spread out in parallel which  I want accomplish or what I can accomplish, here is what I observe for now:

Understanding the test coverage I want to reach by my testing and automation, so I provide the information expected is a challenge each time.  The reason for simple to quote for first is the 'testability' for that coverage mark.  Once I get an idea of what is the skills that I need to build and use in such situations, it will help me to make strategic decisions in testing execution and its management.  The is very important as per me from point of view testing strategic base.  This is helping me and allows me to test the perceived level of testability itself for marked coverage boundary or milestones.

This base of strategic thinking in learning and identifying the testability can be used in timelines of a project. Not just during the execution of testing, also in pre-execution and post-execution, it can be used. It gives an idea of how the tester has to be equipped in changing needs of testing and engineering. For me this is working in context where I'm exposed and in few cases I had to add few vectors along with test coverage such as technology and programming factors specific. I'm experimenting this in varied projects having different technology and skill challenges.


Visibility out of testability

With this, for now I see,

• Testability is related with codability, programmer, tester, environment, process, people, situation, priorities
• Codability gives the first hint on testability and skills needed to test the same.
• Testability influences the Test Coverage
• Testability influences the time taken by a tester to test the system in a given context
• Like codability requires the skills of a programmer, testability requires skills of a tester

Note: Automation when leveraged with testability, wonders can be done via automation in assisting the testing.

A day ends in seeing crash and hearing "No crash! It works."

It is a case where I involved actively in the second day to complete my investigation, after learning my negligence and assumption.  The happening, in the case, stopped the testing for a day. 

A day with no testing and programming, it is a cost which has not turned into any benefit to business! I heard the same from business team.

Happening in the case

One of my fellow tester had got a project assignment and expectation was to carry out the performance test. The latest release and deployment was available for the testing and it was used. While the fellow tester started the task from tester's desk, the business team started using the product from their place.

The tester noticed the product crashing after few actions. Clueless why it is, just the message was available that said, "Something went wrong!"  Looking at this, tester had rolled back the latest installation and did a fresh installation. Yet, the same behavior and message.  This made the tester to use different test setup and noticed the same behavior and message.

By this time, I was said about this behavior.

Observing the happening

Fellow tester walked through me the context of task, it's priority and expected time to finish the task. Following with that walk through, tester gave me the happening in the test environment and the observations recorded for the crash.

With this, the tester said, the business team is not facing this behavior. But here it is crashing after few actions on launching it.

Hearing this, we wanted to make sure the version of the product and hardware setup was similar or close enough to be fair.  It looked everything is symmetrical in terms of product version and setup.

I asked for the log and noticed the parsing failing at the client end. Running through the stack trace in the log it was evident for me that there is a problem in processing the data. Which data, that was the question.  The stack trace said, the data that was wrong. But the claim from the tester was, the same data is being used by testing team and business team. Then, why the crash experience just for the tester.

Product getting crashed is a common sight. The interesting aspect in the crash is knowing the root. I said this to tester and asked to observe data transmitted and update me on the same.

How negligent here, I'm!

I knew, from so the so far investigation, we are very close in knowing the root cause of the problem. But then, from here, on updating what to do next in investigation and analyze, I moved into my practice. This is the mistake I did. I should have joined the tester in completing it. 

But that next part of investigation did not happen for the whole day and it never came to me back. I heard business team is using product without any problem. At least, I should have inquired with the engineering team but which I did not.

Next day's fresh listening

I noticed the discussions between business and engineering team. The calculation was on the time that went out seeing no testing and no clue of the crash.  After few minutes, I heard from tester, it works if I choose different network. It was tried in morning and since it worked, the other network line is used.

That shook me very strongly and my eyes just became very straight in sight seeing the engineering team. I asked, "How? That should not be a problem at all from network information I see here. It should be something else, not the network."  

I could not convince myself that changing the network will solve the crash. Just the product did not exhibit the behavior for a reason. What was the reason? I started to converse with the product again.

Going back to the undone

I requested for the data from the data monitoring between the client and server. This was one part of the investigation which was suppose to continue.  My bad, I assumed, this was done and engineering team saw no problem here. I did not think of asking it and look at it, on the previous day, because I assumed.

This time, I was very keen to look and requested to setup environment for monitoring the data. Used different networks and noticed data sent, received and parsed.

What's the problem?

From last day's investigation it was evident to me, the client could parse the data and assign it to the object. The consequence was Null Pointer Exception and product getting crashed.

Now with data, I started analyzing and every bit of data and line by line information in the log.  The problem is, the product is unable to handle the data at this transition point if that is not from the server. But why? Isn't that a expectation? I left the question with the engineering team for their discussion.

The request from client did not reach the server at all. The web monitoring and filtering system (WMFS) in the network it did not let the request to reach server. The response was from the WMFS to client.

The other mystery to learn was, how is it possible to work on one network and not on another network line, while the service line is same.  In one network, out of the first three request, all got response from WMFS.  While on another network, the second and third response was from the server while just the first request got response from WMFS.  This showed, though first request fails, the second and third requests are crucial.

Beyond the corners of the problem

I see, there is no harm in having WMFS in place. Customer who procures the product can have WMFS in place. Product cannot say, not to use WMFS to its users.

The product (client module) should be able to parse the response and have identity mechanism with server token to parse data, which it receives be it from server and other sources.

Learning

For engineering team, it is an alert to handle data in all possibly identified corners. To me, it was the learning which said, not to be negligent and hand over the investigation if you do not request for the update after few minutes.

To business team, the learning is to follow up on investigation which started and see a closure.

Android: Test investigating for RAM consumption - Bit more on Garbage Collection

From this post on knowing about the heap, we moved to this post which gave brief thought on the GC. Continuing the understanding of heap, I will share what I have gathered for garbage collection's algorithms.  It is upto to choice of people to select and implement algorithm(s) and technique(s) in the JVM they build.  To remind, Android OS has Dalvik (till 4.x) and ART (from 5.x) JVMs. For understanding and carrying out the test investigation for RAM consumption, being aware of the GC algorithm and techniques is a plus point.

Roots and Garbage Collection

The GC algorithm will be doing these two things.

1. It must find the dead objects and live objects
2. It must reclaim the heap space used by the dead objects and make it available for application/program

Finding the dead objects is accomplished by identifying set of root and enumerating it to determine the reachability from the roots. An object is reachable if there is a path of references from the roots by which the program/application can access the object. The reachable objects are called 'live' objects and those which are not reachable is called 'dead' object i.e. the garbage.  Any objects referred by a live objects is also reachable and it is 'live' object.

What may be the GC algorithm chosen in the JVM, the JVM will stop the running application when it has to run the GC. When this happens, every thread except for the threads needed for the GC will stop their task for a while. The interrupted threads will resume only after the GC is completed.

So there is a need for fine tuning the algorithm and techniques of GC used in the JVM, such that the interrupted time will be as small as possible. Dalvik and ART are no exception from this.

Source of the "Roots" in JVM

The executing program/application will have access to root, always. The root set in a JVM will be based on the implementation. Few of the source of roots can be as below

• object references in local variables
• operand stack of any stack frame
• object references in any class variables
• strings stored in the heap (from loaded classes) -- for example, class name, method name, field name

Approach to identify live objects among dead objects

Reference counting and Tracing are the two common approach for identifying the live objects (reachable) from the dead objects (not reachable) in the heap.

The reference counting GC differentiates the live objects from the dead objects by keeping a count for each object in the heap.  This count keep the track of numbers of references to that object. The tracing GC will trace out the graph of references starting from the root nodes.  The objects that are encountered during the trace, will be marked and it is known be live object. The one which are not marked in the tracing are the dead objects and needed to removed from the heap.

Further learning and exploring

It will be not in scope to write more about the different algorithms and techniques with this blog post. However, use the below to learn more about the GC algorithm, types and techniques. It will help in knowing how the ART better.

1. Generational Collectors -- Weak generational hypothesis; Young generation; Old generation; Permanent generation; 
2. Serial GC
3. Parallel GC
4. Parallel Compacting GC
5. Concurrent Mark and Sweep GC
6. Garbage First GC
7. Reference counting collectors
8. Tracking collectors
9. Compacting collectors
10. Copying Collectors
11. Adaptive Collectors
12. Remembered sets and popular objects
13. Reference objects -- reference object and its referent
14. Reachability state changes and use of Cache, Canonicalizing mapping, and Pre-Mortem clean up

Android: Test investigating for RAM consumption - Garbage Collection

Getting the brief information on heap this post, I will move on to Garbage Collection in JVM. I have tried to understand Garbage Collection and here is the brief information on the same.

Garbage Collection

When an object is no longer referenced by a program, the heap space it occupies it can be recycled so the space is made available for subsequent new objects.

The JVM has to to keep tracking which objects are being referenced by the program which is executed, and finalize and free unreferenced objects at run time.  This activity will need more CPU time than what it requires when program explicitly free unnecessarily occupied memory.  How to optimize the CPU usage by the program is key in designing of the product so product can still do better when CPU is loaded in a given context.

Heap Fragmentation and Garbage Collection

The heap fragmentation occurs gradually as the program execution happens. The new objects are allocated and the unreferenced objects are freed. The freed portions of heap memory are left in between portions occupied by live objects. In this case, the request to allocate new objects will be filled by extending the size of the heap even though there is sufficient unused total space in the existing heap.  And this is possible if there is sufficient contiguous free heap space available into which the newly created object can fit.

The swapping is required to service the growing heap and this can degrade the performance of the program which is in execution.  Now, I will leave it to you to consider what and how will the impact be from heap fragmentation in desktop and mobile device, running on lower RAM. Have you ever noticed message saying "Out Of Memory" or "running out of memory"?

With the garbage collected dead objects, the problem is, it can impact the performance of the application/program. The JVM has to keep track of the objects being referenced by the executing program/application and finalize the unreferenced objects (dead objects) in run time. All this happens on the fly while one uses the program/application.  This activity can consume the CPU scheduling time. What if the program/application itself freed the memory? Won't it consume the CPU, then? Both consumes but how much is the question. It depends on the code of the application/program and the hardware as well. I'm not sure if one can handle the CPU scheduling in garbage collection process to free the dead objects. If it is possible to handle the CPU scheduling in this context, programmers should have the constraints and limitations; else by this time most of the app would have been better in GC environment if the CPU scheduling was in full control to the written code of program/app.



Graphical Representation of GC and Heap Fragmentation

Figure-1: Garbage collection contextual elements

Figure-2: Heap Fragmentation from garbage collection