Performance Testing - The Unusual Ignorance in Practice & Culture

 

I'm continuing to share my experiences and learning for100 Days of Skilled Testing series.  I want to keep it short and as a mini blog posts.  If you see, the detailed insights and conversations needed, let us get in touch.

The ninth question from season two of  100 Days of Skilled Testing is

What are some common mistakes you see people making while doing performance testing?  How do they avoid it?

Mistakes or Ignorance?

It is mistake when I do an action though I'm aware that it is not right in the context.

I do not want to label what I share in this blog post as mistake.  But, I call it as ignorance despite having or not having the awareness, and the experience.

The ignorance said here is not just tied to the SDLC.  It is also tied to the organization's practice and culture that can create problems.

To this blog post's context, I categorize the ignorance in these categories -- Practitioner and Organization.

1. Practitioner's ignorance
• Not understanding the performance, performance engineering, and performance testing
• When said performance testing, taking it as - "It is load testing"
• No awareness on what is performance and performance engineering
• Going to the tools immediately to solve the problem while not knowing what is the performance problem statement
• Be it web, API, mobile or anything,
• Going to one tool or tools and running tests
• No much thinking on how to design the tests in the performance testing being done
• Ignoring Math and Statistics, and its importance in Performance analysis
• No idea on the system's architecture, and how it works
• Why it is the way it is?
• The idea of end-to-end is extended and used in testing for performance and having hard time to understand and interpret the performance data
• How many end-to-end your tests have identified?
• Can we test for performance to all these identified and unidentified end-to-end?
• Relying on the resource/content in internet and applying or using it in one's context without understanding it
• No idea on the tech stack and how to utilize the testability offered by it in evaluating the performance
• Not using or asking for testability
• Getting hung to most spoken and discussed 2 or 3 tools on the internet
• Applying tools and calling out it as performance testing
• No attempting to understand the infrastructure and resources
• How it impacts and influences the performance evaluation and its data
• Idea on Saturation of resources
• Thinking it as a problem
• Thinking it as not a problem
• Not working to identify where will be the next bottleneck when solving a current bottleneck
• What to measure?
• How to measure?
• When to measure?
• What to look when measuring?
• Not understanding the OS, Hardware resources, Tech Stacks, Libraries, Frameworks, Programming Language, CPU & Cores, Network, Orchestration, and more
• Not knowing the tool and what it offers
• I learn the tool everyday; today, it is not the same to me compared to yesterday
• I discover something new that I was not aware of what it offered and exist
• I learn the new ways of using the tool in different approaches
• No story in the report with information/image that is self-describable to most who reads it
• And, more; but the above said resonates with most of us
2. Organization's ignorance
• At the org level, for first and to start, it is ignorance in Performance Engineering
• Ignoring the practice of performance engineering in what is built and deployed
• Thinking and advocating, increasing the hardware resources will increase and better the performance
• In fact, it will deteriorate over a period of time no matter how much the resources are scaled up and added
• Ignoring the performance evaluation and its presence in CI-CD pipeline
• The performance tests on CI-CD pipeline should not take beyond few minutes
• What is that "few minutes"?
• Not prioritizing the importance of having the requirements for Performance Engineering

Recently, I was asked a question - How to evaluate the login performance of a mobile app using a tool "x"?

In another case, I see, a controller having all HTTP requests made when using web browser. Running these requests and trying to learn the numbers using a tool.

I do not say this is wrong way of doing.  That is a start.

But, we should NOT stay here thinking this is a performance engineering and that is how to run tests for learning a performance aspect[s].

To end, the performance is not just - how [why, when, what, where] fast or slow?  If that is your definition, you are not wrong!  That is a start and good for start; but, do not stick on to it alone and call performance.   It is capability.  It is about getting what I want in the way I have been promised and I expect; this is contextual, subjective and relative.  The capability leads to an experience.  What is that experience experienced?

Sometimes, serving the requests by what you call as slow, is a performance.  What is slow, here?

The words fast and slow are subjective, contextual and relative.  It is one small part of performance engineering.

That said, let me know, what have you been ignoring and unaware in practice of Performance Engineering & Testing?

Deep Link and its Testing via Automation

 

I get these question consistently from my fellow testers and community.

1. How to automate the mobile apps and web applications using Deep Links?
2. How to automate the business flows using Deep Links?
3. How to achieve end-to-end business flows testing on using Deep Links?
4. How to automate scenarios in mobile apps using Deep Links?
5. What is the best approach to automate the mobile apps using Deep Links?
6. What is the best practice to automate using the Deep Links?

And, more questions on same pitch.

No Deep Dive into - What is Deep Link?

A hyperlink in HTML is a kind of deep link within a website or to another website.

Deep Link is known with different names for web, Android app and iOS app.  All these names have the same understanding and intent at some point.

The Deep Links are URIs that takes me directly to a specific part (activity or fragment) of the app that I'm using or testing.  The Deep Link will have an intent which tells where I will be taken on using it.

When we converse on diving deep technically into testing and automation of Deep Link, will share more insights into its internals.

Deep Link and Challenges

This question is discussed with me often:

How to do end-to-end testing using the Deep Link?

Automation of a mobile app using Deep Link poses a challenge which is not experienced in web application.  

One such challenge is, say you have not installed the mobile app.  [This is solvable!]

• On using a Deep Link, I should be taken to Apple Store or Play Store based on the app.
• I have to install the app.
• Post this, in the traditional automation, I should start traversing the business work flows via GUI.
• Is this adding to the flakiness aspect of automation via GUI?

When we talk so much about flakiness and how to avoid (not prevent), should we exercise business workflows when automating using Deep Link?  What you are thinking?  Let me know!

Scoping of Automation Using Deep Link

Back to the fundamentals.

• We have to automate, no escape from it.  Let us automate what must be automated!
• Let us not fall into trap of "Automate everything!"
• For today, I'm in this mindset and attitude,
• What we automate depends on the objective or goal that we want to accomplish.
• Each test should have precise and deterministic goal.
• A test via automation is not an exemption to it.
• A test defined in automation should be precise, deterministic and have a single objective - Single Responsibility Principle.

What is the objective of my testing via automation for the Deep Link?  This define the scope and extent of my automation.  This will minimize the number of checks that I do using Deep Link.

The purpose of Deep Link is to take me to specific part of the mobile app.

• Should I start the end-to-end or exercising the workflow to be included in the Deep Link tests?
• If included, am I not complicating the testing via automation?

Automation using Deep Link

I ask this question to myself and to my team.

What is the goal of testing via automation using Deep Link?

This question helps me to pick minimal and necessity flow actions.   It has lead and leads me to define minimal tests for Deep Link based on what we want to learn from automation of same.

To me, the purpose of Deep Link is not end-to-end testing.  It's purpose is,

Am I taken to the intended state and data when used the Deep Link?

I have kept the test intent to this.

With this, I have come with tests that has minimal must evaluation and assertion to learn if the app is responding or not to the Deep Link.  This is what the business wants when the Deep Links are created.

The app usage and workflow function is not a problem statement of Deep Link in a general context.

Deep Link is not for end-to-end.  It is to take to you from a point to another point, that's it.

Are you automating using Deep Link?

App Crash! Testing around and inside the crash

In day and out, I come across testers, programmers, managers, and management having efforts on fixing all the crashes. Yes, all the crashes.  In a way I see, if the app did not crash, I will not know the areas that is not being handled well enough.  My testing focus areas will also have tasks noted in such areas to test and learn as much as possible. I do that task provided I can make/given time for it as it is unplanned task.

The common checks to handle crash!?

I learn, an exception if unhandled at runtime, it leads to crash.  There are multiple exception that app can witness which we never thought of during development.  In my initial days of testing, I was in assumption, if we can have -- null checks, index check, illegal argument check, and state check for an activity, we have handled most of the exceptions.  I learned, I'm wrong! How many checks can I write in the code.  I'm not a programmer by job.  I'm a tester. 

I see these checks are not enough and few more got added to my test strategies eventually -- race conditions, unexpected data, wrong data, environment factors, and many more. The collection of these checks is continuing to grow.  Do I cover all these possible crash inducer collections in my testing?  No, I can't and I won't have that luxury of time as well.  Technically, I will learn and prioritize what to use and when.

How the check looks in code, to me?

I write code for automation, which I need to assist my testing.  Here I did write such checks.  At a point of time, I saw, the automation code was full of checks.  Is that a right way?  Definitely not!  A professional and skilled programmer might not do that.  If a programmer has to have such check in each layers of the app architecture, will that sound good?  Personally as a tester, I will not design my tests that way.  As I'm not a programmer, I'm not aware of the pros and cons of doing so. At least I know that is not a better practice to have checks in each layer of architecture of the app.

By handling the exception in my automation code, I print stacktrace of exception.  But will I learn from it to be a better tester?  That's the question I have asked and continue to ask myself.  The exception fix I'm doing, is it stopping me from learning the problem which I have introduced by the actions I'm doing on the app? Is that exception blocking me from learning the underlying problem in my automation code and app?  If yes, then I have a fundamental problem which I have to work upon is my thought.

Why it crashes?

Why at all the app gets into crash? I learn, if the app gets into a state it was not designed for, it will and should crash. As a tester, I will have to learn this state (and such states) at the earliest when I experience crash and on reading the crash stacktrace. I will be happy and not make a fuss about it, if I see a crash for first.  

I learn what is the priority and impact of the crash?  Should I invest my time and test investigate further to provide as much as details to programmer?  Or should I report it with good enough details and continue my testing?  I will answer this question to me.  All I wish is, I don't want the user of the app to experience the crash.  If there is a crash, as a member from development team my intent is to keep it to minimal number having little or no impact.  I see the crash is a great source of learning about my work in the app.

I used to be fussy about crash years back be it on desktop applications, database, web applications and mobile apps.  Now, I have come to point, I love them and it is absolutely okay for an app to crash and it should crash.  What I do post crash in fixing, that tells the bigger story.  In my work, the crashes have made the app better because the team was serious about those crashes.

What on having the crash?

Should the user lose the data what she or he entered on experiencing the crash?  I personally, don't want this to happen for me.  If it happens, I will be annoyed!  That said, how to handle it?  That's something we will have to sit with the programmers and team, for discussion.  

At what point in app, encountering the crash, should we close the app and start over again. At what point in app, it is okay to note the crash and pull the stacktrace, and let continue the user using the app with data entered intact. At what point in app, I should not show UI in view on entering crash and what to show, then resume over safely from there on.  Personally I feel this is a team effort and not just a programmer's effort in making it happen.  

Few testing strategies hack to uncover crash

Here are few things I do and I ask my fellow testers to do when testing mobile apps:

1. Using the test data which will check the data integrity in app at -- entry point, during processing and post processing.
2. Identifying the states of the app and passing the invalid states in app at -- entry point, during processing and post processing.
3. Identify the input which is not from a tester and user. Classify the input on which I don't have control. For example -- the incoming intent; the app responding to APIs (default values, entered values and processed values); the app receiving the response from APIs; the device state; app's activity life cycle state and data/state exchange, and many more as this.
4. Depending on Android or iOS app, much more strategies can be narrowed down to be specific and work.  At the end, what time I'm left with in the test cycle and what do business want, directs me on what to do.

Debugging and Investigation skills

There are libraries which collects the crash on exception along with other details as -- device info, user info and network info.  I have been with programmers and had difficulty in reproducing the crash and experience it in development environment.  This said, are the logs enough to fix crash?  May be, so we handle the exception and continue the flow of app in runtime.  But did we solve the root problem that caused the crash?  No!  This is where, I feel, the skill of a tester comes in and it is very much needed.

This skill defines me what I'm as a tester and value I bring to the system.

Native, Web and Hybrid - How related and different these apps are?

During discussion on mobile apps with a tester, one of the question was - "What is native app? What is the difference between hybrid app and native app?" Like one uses the mobile apps in and out almost all days on mobile devices, yet there is confusion in knowing what these apps are and the differences among them.  In this post, I do not share about advantages and disadvantages for the types of mobile apps, rather I share what I understand for what they are.  And, this should help to start learning on advantages and disadvantages if any.

Further knowing what is the technology stack being used for the native mobile app, web app and hybrid mobile app, a tester can design the tests better.  This helps to figure out the testability and from here one can get the idea which are good to automate and assess, and those which requires human skills to assess further and test.

Mobile web app and mobile device

I will walk through briefly on native mobile app and hybrid mobile app in this post. Before I do that, let me tell bit about web app with Responsive Web Design (RWD) for mobile devices.  Do you know any website that you use often? Open that site on your mobile device browser now and experience how it is rendered. Now, why not changing the orientation of your device and view it in portrait and landscape. What was your experience? Were you able to use it on mobile device? Did it render in same width and height as in desktop browser or it gut customized to your mobile device's screen?

Along with RWD, HTML5 and latest web technologies of JS with CSS, their exists frameworks to build web app for mobile which uses the nativity of mobile devices. They are evolving and exists, today.  One such example is Ionic framework.

While mobile OS extends the support of web view to display the web contents within the activity of an app, the above said web technologies and framework supports to build mobile web app.  With this the mobile web app can make use of device's hardware and sensor, as a native mobile app does.

This mobile web app is not available as an app in the Google's Play Store and Apple's App Store. However it can look like a mobile app in an experience but the implementation way differs. When a website can be rendered to mobile device's screen size, is it possible now to get a feel like not a browser based website though address bar is shown?  Frameworks available today to build the mobile web app will help to do so.

Some examples for the mobile web app are GMail, Flipkart, Amazon, and Facebook.



Native app and mobile device

Mobile app which is installed on mobile device and that can make use of hardware and sensor available on mobile device, is typically a native app.  This app is built for a particular mobile device's OS and platform.  With mobile device's few apps come pre-installed and user can still look for an app what she or he needs on Google's Play Store and Apple's Play Store.

A native app can access what it requires on mobile device provided it has the permission to do so. Now make a list of permissions you have seen which app asks you for installing and using it on the mobile device.

Some example for the native mobile app are Contacts, Message, Camera, Calendar, and the one which can be downloaded from the respective store of Google and Apple.

Note: In this context, I'm taking Google and Apple platforms as it is well known to most.



Hybrid app and mobile device

In above said, HTML5 and JS technologies have the major role in mobile web app. Now, the same can be used to build apps which can be installed on Android an iOS?

Technically the hybrid app will have HTML5 web app within a native wrapper which is supported by the mobile OS.  This mobile app is developed with HTML5, JS technologies and CSS, then compiled into native Android or native iOS or any other mobile OS using wrapper support.

Apache Cordova is one such mobile application development framework which  helps to wrap up an app that can run in WebView by having nativity extension to mobile device's (OS) platform. The plugins of Cordova can be used to access the device's hardware support. On knowing the word as Cordova, now if happened to explore further we will hear the words -- Phonegap, Titanium, Ionic, Xamarin etc. Knowing how they support to develop mobile application will help to understand better about the hybrid mobile app.

If the hybrid mobile app is also a kind of web app, then the HTML, CSS and JS files are stored in the app which will be installed on the device? Or, will it get from the server on launch of the web on a desktop browser?  This depends on how the hybrid app will be designed. If it is packaged into the .apk or .ipa file, then web APIs can be used send and received data.  If it is received from server then a layer around the WebView will have to built and web APIs will be still used.

The apps built in this sort are called hybrid because

• they are neither native apps as the rendering is on via WebViews instead of platform's UI framework
• they are neither web apps because they are packaged as an app for distribution across platforms using the platform nativity via its APIs

Have you used, GMail app on your phone by downloading it from store? Are you using Instagram, Facebook, Youtube, Maps apps on your mobile device on downloading it from store? Then that is an example of the hybrid app which combines the strength of native app and mobile web app together.

So whats the point

To end this post, if asked what is best i.e. whether native app or hybrid app, it depends on context and what is trying to be solved with negotiation.  Both helps to accomplish the needs of app developing organization, programmer, tester and user.  While each have they their own strength and weakness, it is the choice to be made by people who builds for whom.  In this post, I have not shared the advantages and disadvantages of it.

Now starts the fun of exploration around testing and automation.  How to test the apps built in hybrid way? What are the level of isolation I can identify so that I can layer it out layer by layer and test? Where to start if it is getting packaged into nativity of the platform on mobile OS?

If you are doing it or figured it out, pull me to pair with you and test together. Let us do it!

Android: Test investigating for RAM consumption - Bit more on Garbage Collection

From this post on knowing about the heap, we moved to this post which gave brief thought on the GC. Continuing the understanding of heap, I will share what I have gathered for garbage collection's algorithms.  It is upto to choice of people to select and implement algorithm(s) and technique(s) in the JVM they build.  To remind, Android OS has Dalvik (till 4.x) and ART (from 5.x) JVMs. For understanding and carrying out the test investigation for RAM consumption, being aware of the GC algorithm and techniques is a plus point.

Roots and Garbage Collection

The GC algorithm will be doing these two things.

1. It must find the dead objects and live objects
2. It must reclaim the heap space used by the dead objects and make it available for application/program

Finding the dead objects is accomplished by identifying set of root and enumerating it to determine the reachability from the roots. An object is reachable if there is a path of references from the roots by which the program/application can access the object. The reachable objects are called 'live' objects and those which are not reachable is called 'dead' object i.e. the garbage.  Any objects referred by a live objects is also reachable and it is 'live' object.

What may be the GC algorithm chosen in the JVM, the JVM will stop the running application when it has to run the GC. When this happens, every thread except for the threads needed for the GC will stop their task for a while. The interrupted threads will resume only after the GC is completed.

So there is a need for fine tuning the algorithm and techniques of GC used in the JVM, such that the interrupted time will be as small as possible. Dalvik and ART are no exception from this.

Source of the "Roots" in JVM

The executing program/application will have access to root, always. The root set in a JVM will be based on the implementation. Few of the source of roots can be as below

• object references in local variables
• operand stack of any stack frame
• object references in any class variables
• strings stored in the heap (from loaded classes) -- for example, class name, method name, field name

Approach to identify live objects among dead objects

Reference counting and Tracing are the two common approach for identifying the live objects (reachable) from the dead objects (not reachable) in the heap.

The reference counting GC differentiates the live objects from the dead objects by keeping a count for each object in the heap.  This count keep the track of numbers of references to that object. The tracing GC will trace out the graph of references starting from the root nodes.  The objects that are encountered during the trace, will be marked and it is known be live object. The one which are not marked in the tracing are the dead objects and needed to removed from the heap.

Further learning and exploring

It will be not in scope to write more about the different algorithms and techniques with this blog post. However, use the below to learn more about the GC algorithm, types and techniques. It will help in knowing how the ART better.

1. Generational Collectors -- Weak generational hypothesis; Young generation; Old generation; Permanent generation; 
2. Serial GC
3. Parallel GC
4. Parallel Compacting GC
5. Concurrent Mark and Sweep GC
6. Garbage First GC
7. Reference counting collectors
8. Tracking collectors
9. Compacting collectors
10. Copying Collectors
11. Adaptive Collectors
12. Remembered sets and popular objects
13. Reference objects -- reference object and its referent
14. Reachability state changes and use of Cache, Canonicalizing mapping, and Pre-Mortem clean up

Android: Test investigating for RAM consumption - Garbage Collection

Getting the brief information on heap this post, I will move on to Garbage Collection in JVM. I have tried to understand Garbage Collection and here is the brief information on the same.

Garbage Collection

When an object is no longer referenced by a program, the heap space it occupies it can be recycled so the space is made available for subsequent new objects.

The JVM has to to keep tracking which objects are being referenced by the program which is executed, and finalize and free unreferenced objects at run time.  This activity will need more CPU time than what it requires when program explicitly free unnecessarily occupied memory.  How to optimize the CPU usage by the program is key in designing of the product so product can still do better when CPU is loaded in a given context.

Heap Fragmentation and Garbage Collection

The heap fragmentation occurs gradually as the program execution happens. The new objects are allocated and the unreferenced objects are freed. The freed portions of heap memory are left in between portions occupied by live objects. In this case, the request to allocate new objects will be filled by extending the size of the heap even though there is sufficient unused total space in the existing heap.  And this is possible if there is sufficient contiguous free heap space available into which the newly created object can fit.

The swapping is required to service the growing heap and this can degrade the performance of the program which is in execution.  Now, I will leave it to you to consider what and how will the impact be from heap fragmentation in desktop and mobile device, running on lower RAM. Have you ever noticed message saying "Out Of Memory" or "running out of memory"?

With the garbage collected dead objects, the problem is, it can impact the performance of the application/program. The JVM has to keep track of the objects being referenced by the executing program/application and finalize the unreferenced objects (dead objects) in run time. All this happens on the fly while one uses the program/application.  This activity can consume the CPU scheduling time. What if the program/application itself freed the memory? Won't it consume the CPU, then? Both consumes but how much is the question. It depends on the code of the application/program and the hardware as well. I'm not sure if one can handle the CPU scheduling in garbage collection process to free the dead objects. If it is possible to handle the CPU scheduling in this context, programmers should have the constraints and limitations; else by this time most of the app would have been better in GC environment if the CPU scheduling was in full control to the written code of program/app.



Graphical Representation of GC and Heap Fragmentation

Figure-1: Garbage collection contextual elements

Figure-2: Heap Fragmentation from garbage collection

Android: Test investigating for RAM consumption - Heap and Heap Dump

The process of allocating new objects and removing the unused objects to make space for the new allocation is termed as Memory Management.  

Object Memory Management - Heap and Stack

Hearing the two terms Stack and Heap, can bring confusion in the memory management. Though I will not get in detail about each here, I will share briefly for what I have understood.  

The 'stack' memory is used for static memory allocation and 'heap' is used for dynamic allocation. For those allocated on the 'stack', they are stored directly to the memory and the allocation happens when the program is compiled. Whereas for those allocated on 'heap' have their memory allocated at the run time and the size of heap is limited by the virtual memory. Both the 'heap' and 'stack' are stored in RAM.

Shallow Heap and Retained Heap

Moving ahead, in 'heap' we will hear the words -- Shallow Heap and Retained Heap.

• Shallow heap -- is the amount memory consumed by one object. That is, the amount of memory allocated to store the object itself by not taking the reference objects into consideration. The shallow heap size of set of objects is the sum of shallow sizes all objects in the set.
• Retained heap -- is the amount memory consumed by all objects that are still alive by objects at the root of the retained set. The retained heap size of an object is, "shallow heap size" added with "shallow heap size of objects that are accessible directly or indirectly, only from this object".
• Retained set -- is the set of objects which will be removed by garbage collection when an object or multiple objects is garbage collected.

In other way, the shallow heap of an object is the size of object in the heap. The retained size of the same object is the amount of heap memory that is freed when this object is garbage collected.

From these two resources (1 and 2), I have understood the details of Shallow Heap and Retained Heap. I feel it is simple and well explained; I have been referring it since years.

Heap dump

It is a snapshot of the memory at a Java process at a given point of time. The dump will consist of Java objects and classes. The reference of memory for the objects and classes will be available in the dump taken. If the Garbage Collection occurs at time of obtaining the dump, then the obtained heap dump will have details and information about the remaining (existing & used, existing & unused) objects and classes.

The heap dump do not tell who created the object and where it was created. Based on type of heap dump, the information available in the dump will be as below.

1. Classes -- Classloader, name, super class, static fields
2. Objects -- Class, primitive value and references, fields
3. Thread Stacks and variables -- The information about call stack of threads at time of obtaining heap dump. The information about local variables and objects.
4. GC Roots -- The mentioning of objects which are reachable by JVM

Benefits of sharing the debug APK with tester

On knowing -- the myths around debug APK; what is debugging? and how tester can debug and be of help to programmer?, I tell about the benefits.  The programmer, product owner and team will benefit by sharing the debug APK to tester for testing. However, make sure you are in synchronous and mutual agreement with the tester if there is a 'concern' in sharing so.

"benefits" to programmer in sharing Debug APK to tester

Test investigating using log

With the debug APK the code written to print system messages, debug messages, and stack trace will print the information into the log.  While testing, tester can learn the logs to monitor -- what's happening within the device; what's happening to the app and what's happening in the environment where the device is. Won't this help to debug and investigate the problem or incident? The additional information shared by the tester apart from the incident won't it help programmer to the trigger or cause of the incident?

Test investigating with variables and breakpoints

The source code of app is available to tester and tester is using the debug app on her/his device or emulator. During this time tester can watch the variables, execution sequence and flows, knowing the stack details around the break points and analyze the code. This will help the tester to be stronger and get skilled each time. As a return, it comes back in testing to your product/app. Getting to frame level debugging and testing, wow! Is not that a value add?  If the tester need assistance to start here, giving few hours initially is a worth investment!

Tracking and testing for consumption and allocation

Yes, testing for allocations and use of HEAP. Further testing for network consumption, battery consumption, CPU consumption and UI performance. Is this not a value add in building high performance mobile app?

"limitation" with benefits in context of tester

As the benefit, there are also limitations in what a tester can do in context.  One such context is tester not having the source code of the app but has the debug APK.

Here, the tester might not be able to look into variables, stack values and test using the breakpoints. But, the tester can decompile the APK and work with compiled code with breakpoints and watch the execution flows. This is time consuming task and the task which can end abruptly if insertion of breakpoint and flow is paused incorrectly. Reporting the problem out of this observation is not suggested unless it is very sure that there is a problem.  But this not stop from testing for consumption and allocation by the app on the device.

As this, there can arise several context which yields the limitation. Figuring out how better the testability can be visualized and making use of it in testing, is the skill of tester.  Everyone in the team will have role to help tester to acquire and hone such skills.

"why" the delay in giving a debug APK to your tester?

If you are a programmer and product owner, then on knowing these benefits, yet if you are delaying to share a debug APK with tester, then there is a concern which needs to be addressed by project management.  This is what I learn.

If you are a tester and have come to know the value you can add from your testing, it will be a unfair to be quite and not asking for the debug APK.

"closing" thought with iOS and WP debug app

As with Android APK, the iOS and WP apps also have debug apps.  The benefit of it is quite similar. As well, there are additional benefits when testing in the respective programming environment with it's devices with the debug app.

Explore the benefits. Encourage the better and useful testing for the context of product.

Debugging and the debug APK

As a textbook or reference book reader, debugging is "the process of identifying and fixing of problem that stopped the product to work as intended."

"debugging" - as how I see

As a practicing tester, I see, "debugging is a empirical activity which involves investigation, reasoning and analysis on the buggy/risky/enhancement incident to learn the details about it, so that a fix solution will be found for it." 

There are different techniques in debugging and the person doing the debugging may carry out multiple activities in parallel.  Discussing the different techniques, approaches and activities in debugging is not with scope of this writing. Hence I'm not sharing much about it, in this post.

I learn, "having the debugging attribute in the product is testability strengthening factor."

"no!" to debug APK while USB Debugging is enabled?

In my testing practice, I test for mobile apps as well. I test Android, iOS and WP apps.  In the context of this blog post writing, I will stick on to Android app and Android APK.

It is very common for a programmer to have "USB Debugging" option enabled on her/his Android device by turning on 'Developer' permission. So the programmer will be able to install the Android app while it is being developed. Further the programmer will use this installed app for testing and debugging, typically.

When tester enable 'Developer' permission and choose "USB Debugging" option to assist in debugging the problems in app, won't that help in developing app? Won't it reduce the time and efforts of the programmer in analyzing the problem when the tester write the detailed Test Investigation report from the debugging? To a tester, who aspire to be skilled mobile app tester, it is very usual to have this setup on her/his test or personal device as the programmer will have.

If so, then have you ever said "no!" to share the debug app to your tester who has same interest in the product as the programmer and product owner?  Why? I will leave it to you for introspection.

"few" myths around sharing the debug APK

There are myths which gives fear in mind when asked for the debug APK. I have tired to demystify few of them here and help to be confident in sharing the debug APK with your tester on mutual understanding and agreement in common interest.

"types" of debugging in developing Android APK

I understand, the IDE used to program the Android APK will use the Java debug by default. That means, the code compiled in debug mode will generate the APK which works with tools based out of Java to see the internals of app.

Along with this, if the IDE supports different configurations of debugging, it can be used. I have not used apart from 'Java' type so far. Hence I cannot share anything further on this.

By the way, when I say "developing", I mean programming along with testing.


"differences" between the debug APK and non-debug APK

When said non-debug APK, the assumption is that it will be a release APK. The debug APK is for the testing purpose. In practice, the release APK will have debug attribute to false. Beside this, the certificates used to sign the APK will be different when it is debug APK and the release APK. These certificates will be bound to date for expiry.

Both debug and non-debug APKs can be ZIP aligned and signed.  Making sure the log writing code is available in the app is essential to print system messages, debug messages and stack trace.

next Post

In the next blog post, I will share the benefits what the programmer and interested stakeholder can get by sharing the debug APK with the tester.

Few myths around sharing the debug APK

The debug APK is a useful build which helps to learn the internal aspects of the app. If security is a concern with the debug APK, then it has to be handled in building the security layers for the app and it's code. Saying the debug APK is the gateway to security problems is not thoughtful, technically and logically.

Here are few myths and fear that is observed when asked for the Android debug APK.

Myth-1: Given the release build/app it cannot be turned to debug build/app

One can use the release build/app and turn it to the debug build though the programmer do not share one. However, there is no assurance that this converted debug build will be functional as the release build/app. If tester converts it to debug build and reports the behavior which are not valid then it is additional effort to programmer and project to triage and then say it is not valid. This happens!

Myth-2: One will have access to the code of debug app

One will have access to code when one installs the app on her/his device. The question is in what format and how the access will be available. Apps having the obfuscation with help of ProGaurd and DexGaurd will obfuscate the compiled code of app. To deobfuscate it, one will require the mapping file key. Further one can go and ahead and work to deobfuscate with different approaches and techniques with no mapping file available as well.  It is upto what level the security is strengthened for app.  Anyways the code is available (in a format) when one installs the release APK/app. 

Myth-3: One can tamper the debug app and use it illegally

From above said myth's opinion, I will continue for this topic.  While building the release APK, the programmer will sign the APK with certificate. The release build certificate and debug build certificate are not same. The person who wants to use tamper the app, still she/he can decompile, edit the app, deobfuscate, recompile, sign the app and build it. When signing on recompile, if the person do not have valid certificate still she/he can go ahead with self created test certificate.  But, the question is will this certificate work well for the app which has multiple legacy SDKs included and expecting a certificate and key. In my test practice, I have seen, the app failing to function for its purpose though it launches in such context.

Myth-4: One can misuse and learn the sensitive area and information from debug app

With out the actual source code, one can reach to stage where she/he can see the register and stack values by using just the debug app's compiled code. For getting into this level, one has to successfully decompile and build it back to SMALI data structure and then start the debugging to look into the stack registers and flow sequence. If the obfuscation is applied at this level, then it a round more of strengthening. Yet it does not mean one cannot cross it but the time needed is unknown.  This is possible with non-debug app i.e. release APK as well.

Myth-5: One can know what happens in the debug app while using it

The very purpose of the debug app is this i.e. to know whats happening when using the app. The question is -- who is using the debug app? Without the actual source code of the app, the information learning will become narrow as the testing progresses with debug app.  

Myth-6: Giving the debug is like giving the permission to code of app

Debug APK means not permission to code of app for editing and building.  If there are other checks on the decompile, recompile and signing to stop app from functioning after this, the APK will not be usable as the release APK.

Myth-7: One can gain the control over the app with debug app

Unless the decompiled app is edited in code and built again to do illegal and incorrect activities, it is unfair to think as this. If this is the possibility with the Android app which is being built, then it has to be handled in how the structuring happens for security layers of the APK. This is nothing to do with debug APK.

If the debug app is pointing to the staging or test environment and access is restricted with IP white listing and similar other approaches, it will restrict the access from public use. The other way is to reduce the validity time for debug signing certificate. Keeping this validity that is reasonable for the test cycle is another approach. After this validity the debug APK will not get installed on devices unless it is decompiled and resigned again. 

Fragmentations: Heuristic and COP FLUNG GUN -- Part 3

Back in this post, I have listed very few perspectives of 'Communication' factor. But it is beyond what I see or what I have seen in view of programming and testing. There is a point which is usually not considered in this Communication factor. It is device fragmentation.  It is a buzz word and it lies there most times as it is not attempted in Software Testing to understand what actually it is and what is the outcome of it. Is is just the device fragmentation the problem source?

Interestingly the behaviors noticed on a device will not be because of fragmentation in first place. But the buzz words makes it to feel it is because of device fragmentation. Going back a bit, is device fragmented or the OS is fragmented? I see both and every other factors of COP FLUNG GUN will have the influence of fragmentation and anything that comes in near future will undergo the influence of fragmentation. It indicates, this will never come down unless the user needs from technology come down. Wait, what is fragmentation and defragmentation? It is very much essential to understand because the understanding of COP FLUNG GUN factor gets better each time if this is understood each time.

In an Operating System, everything is file which in turn will be instructions and data. If this is a file system, then processing of data which will be represented as a file and will be broken into block of files. This fragmented files might be sequentially placed or may not. If not sequential then problem starts -- where disk read, disk write and wait time, all increases in Operating Context.  To improve the condition here in Operating Context due to fragmentation we initiate activity called Disk Defragmentation. Defragementation is a process of grouping the same logical content file so that they are connected in close groups and as a result the file operations by an Operating System becomes easiest and space on disk/memory and processor can be used better for other needs. If observed, the Disk Defragmentation in Operating System actually helps to overcome the impact of fragmentation in Operating Context environment. Then why cant we defragment on the mobile device?

It is like this, I tear the novel and throw its paper pieces across the streets of a city, randomly. Say, I can recognize these piece of papers on the city roads. This is fragmentation in a perspective. Now how will I defragment it by collecting and place sequentially and keep it connected logically and content wise? There are lot other factors in the city's environment -- sun, dust, heat, rain, wind, cleanliness of city etc which will disturb me to collect them together and make a novel again and not just the street which I use to commute.  The same happens in the mobile device its mobile operating system. 

The mobile device's hardware is different from each other where the same app has to run on it. Likewise the Operating System gets customized for each of this device. To relate, the novel cover which was torn into different shapes is the mobile device and the pages of torn novel are customized mobile OS.  I have to make points on each of this torn devices and OS. How? As these torn pieces miss my context and in the influence of environmental and operating context factors, it is more likely that  my points might fail or show more problems now which I cannot guess nor anticipate easily.

Few Fragmentation Factors in Mobile Device and OS

1. Hardware Specifications -- Input; Processing and Storage; Output; Collaborating Unit and Capability
2. Software Specifications -- Customized OS and its versions; Other software & its versions; Software updates - device software; component software; Anything that is bound to be updated or receive the updates
3. User's Context -- Usage pattern; Environmental factors; User's choice and Locale Settings; Rooting and Jail Breaking
4. Environmental Factors -- Infrastructure; Service Provider; Network
5. Regulations -- Policies of devices; Polices of user location; Polices of software
6. Application installed -- It's internal way of functioning on above identified factors
7. Other unidentified factors -- which always exists as unknown until identified

Ecosystem Fragmentation Not just Device Fragmentation

Now it is obvious that fragmentation is not just the device. Not just the device fragmentation is all. It is also including the user who uses the device and app as well along with unidentified factors. How to defragment these fragmentations? This is the challenge to the mobile technology which keeps changing consistently with the next version of app or mobile software or hardware as it gets updated or rolled out. As a result, does it get outdated quickly leaving the little to applications installed and to the previous model (and versions) of hardware and software? Fragmented!


Fragmentation and COP FLUNG GUN

While I learn fragmentation is everywhere here, I see it is also observed in this heuristic -- COP FLUNG GUN.  Identifying it in first hand is not possible as it is highly contextual while few identity remains generic in most cases.

Further in this blog post series I will learn and mention few generic and as well the context specific fragmentation factors in this heuristic. Interesting will be how the fragmenation of  Communication is while it is not mentioned in previous post. Did I tell myself about the fragmentation of files on mobile OS and devices while I was saying how it is from external factors? That has a role too in contribution to the external factors.

Now, let me allow myself to defragment what I have got here.  It is fragmented and goes unrecognized!



Note: I keep the technical stuffs in simple relativity and examples so I can learn it anytime and share it to others who wishes to know about it. Being technical and non-technical is expressing in simple and to anyone is what I understand for now.

Heuristic and COP FLUNG GUN -- Part 2

Continuing from the previous post, I'm seeing what are the different factors in Communication to test and to automate.  In a brainstorm session, I got the below areas.  The below mentioned are apart from the call receiving and dialing, receiving and sending of the message -- text, multimedia, voice or whatever the message supports to send and receive.

I learn and see, these are the activities whose events bring in communication. When I say communication, I also see the communication as disturbance because it disturbs the ongoing activities of another active app(s) in the device.  For example, the disturbance to other app which is active in device and communicating.  I don't claim these are the only factors of Communication. These are the few which I was able to identify in a brainstorm session. As this, one can keep identifying and include it in test coverage and automation coverage areas.

• Phone Call
  • Dialing
  • Receiving
  • Hold
  • Ending
  • Call Swapping
  • Conference Call
  • Unattended
    • Missed Call
    • Changing the system (device) state and ignoring call
      • Flight Mode
      • Mute off sound
      • Unmute 
      • Turn off/on vibration
      • Turn off/on indicators
  • Network interrupted call
    • Inconsistent signal transmission
  • Telephony Events and Notifications
• Message (text, multimedia, voice)
  • Incoming
  • Outgoing
  • Notifications
  • Failure
  • Drafted
• Bluetooth
  • Scanning
    • Manual Scan
    • Auto Scan
      • Scheduled Scan
      • Regular Scan
  • Receiving
  • Sending
• Internet (WiFi and Network Carrier)
  • Switch of network signal
    • EDGE / 2G
    • 3G
    • 4G
  • Intermittent signal
  • Restricted Signal
  • Network/Signal Jammer
  • Upload Scenarios
  • Download Scenarios
  • Browsing scenarios
  • Other Apps Activities
    • Regular Interval Synchronization
    • Anti Virus and Enterprise App Synchronization
      • VPN
      • Restricted Network Channels
  • Client Server Scenarios
    • Restricted
    • Redirection
    • Timeout
    • Denial of Service
    • Payload
      • Processing by device and its resources
      • Processing time for app to keep communication alive
        • For example, transactions of amount via mobile banking apps
    • Platform Organization
      • Cloud
      • Distributed
      • One Store
• System Calls
  • Alarm
  • Reminder
    • Meeting
    • ToDo
    • Tasks
  • Updates
    • Software Update
    • Scheduled update
    • Vendor Update
• System Monitoring
  • Device Intelligence
    • Low Battery Interruption
    • Resource Shortage Interruption
    • Compatibility Interruption
      • OS and Device
      • OS, Device and App
      • OS, Device, App and Sensors
      • OS, Device, App, Sensors and related hardware and software
    • and many more upon building the heuristic list
  • Network Surveillance Interruption
  • Power and Discharge Interruption
  • Device not responding for any tap or interruption
    • During call
    • During transaction's request or response
• Sensor Interruption
  • Gyroscope
  • Accelerometer
  • Compass
  • Motion Sensor
  • Light Intensity Sensor
• Gaming Interruption
  • With association of above sensor
• Hardware Support
  • Device support on different geographical network
• Transfer of Files
  • On different USB modes
  • On different transfer modes
• Manifest Permissions
  • Manifest not providing required communication permission
  • Revoked permission by device
    • aborted communication channels due to restrictions on launch of app or/and device
    • Geographical restrictions
• Platform Library Integration
• 3rd Party Library Integration
  • ODM's data or component being replaced by OEM's components
    • For example, default Android libraries being replaced by 3rd party vendor's SDK on particular device model
    • iOS platform can be approached on down scaling of device model i.e. installing latest apps on previous Apple devices
• Compatibility with different version of pure native libraries of Android and iOS platform and the OEM's device
• Device's Interpretation of Communication and Conveying to User

As this, I keep identifying the areas to interpret with tests which potentially can influence the Communication which is

• existing and communicating
• about to communicate
• to start the thread of communication
• the terminated communication thread

I will continue to identify them by several means based on the context of testing. In general, knowing and learning the Platform, OEM and ODM along with its associated system helps very much.  Now, looking closely into the above mentioned, I see Communication crossing over Network, Platform and other factors of COP FLUNG GUN. Further, understanding the architecture of the product and how technology is used to build the app, I will start isolating and narrow down much into Communication from and along with other factors of COP FLUNG GUN.

On the closing note, it is up to me for what do I bring in communication -- a phone call or a message. And both of these need Network and Platform assistance isn't it?  To keep it simple and structured, I believe, Network and Platform are seen as separate and individual factors here.  How I come up with tests and figure out the areas to automate using these factors which overlaps will be based on my Test Strategy for the context.

There are simple and yet handy tools which can be used to create events which will interrupt, disturbs and starts the new communication while a communication is happening. To end this here, I learn, communication is no more a phone call and text events in mobile devices; it is also a disturbance, noise and interruption to its very own system. And, the communication also exist within the device like the communication exist between the two or more devices.

Heuristic and COP FLUNG GUN -- Part 1

I have Heuristics given by testers from different geographical locations. I use them to identify to the test or what I perceive from outcome of the test using it. And, I often forget to learn this -- heuristic will not solve problem; it is tester who recognizes there is an unusual here or the usual or the consistency here with help of heuristic.

If this is very much right, then the Oracle never identifies the problem or unusual for itself. It is tester who identifies the problem with the help of an Oracle in the test. The heuristic and oracle helps in recognizing the information (which can be a problem or potential problem or not a problem); it won't identify or solve anything for itself.

For example, the tests done to diagnose if one is HIV positive or not, will not cure him. It may help doctor and test seeker to know the information.  The HIV test is one of the heuristic and the outcome of the test to which it is compared against and evaluated, is an oracle, here. But, it does not solve problem for the doctor and test seeker. It just helped to know the information. Solving or fixing the problem is in hands of test seeker and doctor using this information. That is, the same HIV test each time, the same result probably each time saying negative or positive or may be unable to diagnose.

On learning this, I understand, till to an extent, a heuristic can drive me. If I'm over dependent on the same perspective of heuristic, probably I will remain with the same ideas which I used earlier. With this, I learn, heuristic should help me in exploring further to identify more such heuristics. The more heuristic and oracles, I have, the better I can be prepared to approach and test.

For example, SFDPOT heuristic does not do anything. It is just an acronym for words -- Structure, Functionality, Data, Platform, Operation and Time.  It is me who does the action using the initiative given by word "Structure" in the context of testing.  Likewise, COP FLUNG GUN is useful as SFDPOT is. But both (and any) heursitic does nothing by itself. It is powerful enough when I identify and frame the tests to the context using it.

Are there any similarity between SFDPOT and COP FLUNG GUN ?  Has this thought come into me while I use the list of heuristics every day? I see the similarities in both and one is more specific to a context of mobile testing.

I will take an example of what I see in my practice. When the COP FLUNG GUN is used to assist the testing, I'm curious for the tests which get identified and frames to the context under each factors -- Communication, Orientation, Platform, etc.  Now, I see,

1. COP FLUNG GUN gave me the help in identifying the factors of mobile handset, software of the mobile handset, operations of mobile, and the mobile apps.
2. But it did not provide list of component in each factor and it cannot provide too
3. Thought it provides, it might not be valid for all cases, hence it is heuristic
4. Heuristic is effective when it is specific for the context of problem learning and solving, right?

What components I see in the Communication, now?  The obvious two stuffs that I happen to hear and see is -- Phone Call and Messages.  Are these the only two components in the Communication?

What the Communication is at all?