Test Charter and the Four Components of a Charter

A tweet from Ministry of Testing was -- Do you use Test Charters?

Speaking out of my so far practice, I heard the word 'charter' 10 years back when I was referring to SBTM in James Bach website.  Till then I had not heard the word -- charter.  The two words which confused me then was -- Mission and Charter.  

I looked into the sample SBTM reports which James Bach had shared.  I observed the word 'mission' for each session.  That is each session is chartered with a mission to accomplish.  Meanwhile, I strongly believe the spark that took off to testing globe wide referring such testing resources had its birth in Bengaluru then.  Most will say nah and disagree to it.  But that's the truth and Weekend Testing was one of its outcome.

Coming back to the subject of charter, I say, it was not that easy then for me to write one.  I got better each day and I try to get better today as well.  While I worked in the role of Test Coach in Moolya Software Testing, I pair tested with freshers and lateral hires.  I insisted the testers (whom I was assisting to practice testing) to write charter on their own.  I could see the difficulty they were going through and it was a mirror of me showing what I went through some time back.  In beginning all had difficulty to practice in session as they could not concentrate for 30 minutes (forget 45 or 60 min or 90 min) and to take make test notes in parallel.  Yet, there was no practice without a session and a charter to it.

To keep it simple and make lives of the tester easier, I broke the charter into four components. They are:

1. Intent  -- What is the intent of my testing in this session?

2. Target -- What I should be testing?

3. Resources -- What I should be referring while I test?

4. Information -- What information I have to learn from the tests and how to report it.

For example, say the charter of a session is -- Browse through the different doctors displayed based on specialty by swiping the doctor info card in the app. Validate if the subsequent result pages of doctors displayed is still relevant to the specialty chosen, while a doctor can have multiple specialty.  Refer to the HTTP request and response coming in for each result page and validate data displayed on client end. Report with logs and test data when you find or feel there is a problem.

The same charter can be written as this -- Test for the specialty displayed for doctor in search result is consistent in each result pages returned by server. Make sure the test data of doctor have multiple specialty. Report with logs and test data when you find or feel there is a problem. Make note of the confusion if you have any in analyzing the search result.

Also it can be written as this -- Test for the doctor card shown in search results based on specialty chosen.  Report with logs and test data when you find or feel there is a problem.  Make not of the confusion if you have any in analyzing the search result.

If I had to break this charter into four components, then:

1. Intent -- Search for doctors based on a specialty

2. Target -- Doctors info card displayed in the app that is search result pages

3. Resources -- Specialty of a doctor; HTTP request and response; Data displayed on client app

4. Information -- The search results returned and displayed are consistent with chosen specialty? Report if any problem and confusion.

All the above written three charters mean the same.  But how it is written and details provided, varies.  What style to pick and details to provide it depends on multiple reasons when I work with team.  When I work as a only tester testing, I keep the charter pretty self explanatory with details to the reader of session notes.

In my opinion, a tester gets better in writing charter when these four components is clear and can be identified. 

Backward Compatibility - What and who drives the compatibility?

A tweet of Ministry of Testing read as this -- How do you decide how backwards compatible your product should be?

I learn, there is no general rule and as well no specific rule that applies when it comes to backwards compatibility, forward compatibility and the compatibility.  It is highly specific to what product we are dealing with and how the system around it are developed, or being developed or will be developed.  Also how it is deployed.

While compatibility is one of the main feature of product feature, it also signifies -- the design, the deployment style & patterns, communication and the exchange of data.

Further to be precise, compatibility is context specific task.  I have not come across where it is generic task where it is applied and applicable to any where.

If asked what software can come into compatibility spectrum, then:

>> Database System

>> Server

>> Client libraries like SDKs

>> Server Configurations

>> Operating System

>> Hardware Configurations

>> Client application versions being used by user and number of users

>> Server configuration versions being used in different client applications

>> Browser versions, it was a major compatibility concern a decade back

>> Application system version and its dependencies compatibility version

Few analytics that influences the decision on compatibility

>> Analytics showing the user on specific versions of libraries

>> Analytics showing the user on specific versions of OS

>> Analytics showing the user on specific version of hardware configurations

>> Analytics showing the user using specific feature which involves server configurations

>> Analytics showing the conversion funnel on specific setup of system

Few emotion that influences the decision on compatibility

>> Experience of user using a product on specific setup and configuration

>> Customer Support analytics on problems, queries and user satisfaction on specific setup and configuration

>> History of previous compatibility and migration

>> Competitor's presence

>> Team's opinion

Business aspects

>> Negotiation extent

>> Bearable Cost and Non-bearable cost

>> Business Value

>> Social Value

>> Strategic Value

>> All above converting to business profits

>> Survey carried out

>> Timeline

>> Budget and Resources

>> Talents -- people

Environment

>> That simulates the backward version of the system

>> That simulates the provisioned version of the system

>> Outcome of the assessment in such environments

If seen, where do tester fit here to decide the backward compatibility of the product?  In my experience, the decision of backward compatibility is most times strategic from aspect of the technicality.  In the enterprise ecosystem, the same strategic will weigh more on the business decisions and then the technical decisions.

The executing tester most time will not be involved here in such decisions.  Could be a person leading the testing team or the engineering efforts can be involved in such meetings.  In start-up environment I have worked so far, the decision meeting most times did not have the testers involved.  However, on getting the heads up for backward compatibility need, I use to assess the risks in system and its sub-systems.  Then, I shared and had a meeting scheduled if there was a need of different stakeholders and teams to clarify and resolve any risks mentioned.

In simple, the business don't want the service to be broken for a user.  The user should get the service that business provides on using the software application.  Further, the conversion what the business expects has to be what the business expects.  If not, then there is a problem to user and as well to business.

Communication, has explicit and implicit messages! Have you got them?

Besides all the technical work the testing team and a tester do, there are times the tester and her/his testing will take a hit.  Here is one such hit and how I solved it.  When working in an organization which is building a product, usually there will be multiple teams involved.  Likewise, multiple people on influencing and making the decision about the product, development and shipment.  Any one miscommunication here and the slipping of time, the testing team gets its time squeezed and cut, most times isn't it?  I have experienced it.

When I looked into what was the problem here and to solve it, I learnt these:

1. When decision makers communicate, they can have two types of communication being conveyed for teams. 
2. One is explicit communication - which is verbal and written.
3. The other is implicit communication - which is neither verbal nor written; but it is expected that it has to be understood by teams.

If the testing team don't catch the implicit communication, what could be the impact in the time given to testing?  It depends on magnitude of the problem!  Most times, the explicit communication made in a context as well goes misinterpreted by the teams.  If you are not part of that teams which misinterpreted and did it right each time, probably you had a better and standing leader in your team who solved the communication problem.

A simple heuristic here for the testing team and others who involved in this context -- How one can misinterpret what I said? Did I misinterpret in what is said?  One can be not that sharp in communication and interpretation, say.  But that number of team members with varied experiences level, are same in that skill?  Could be yes, then we have to help team; if not, then where is the problem?  Here the question is about solving and proceeding further so team can be of help to each others.

I lead the projects and testing delivery while I was working in Moolya Software Testing Pvt. Ltd. Having lead 55+ projects for different customers and its deliveries, in this role, I had to communicate with external teams (programmers and testers), internal teams (i.e. team within Moolya), sales team, recruit team, human resource for skill development programmes and management team (at customer place and in Moolya).  Especially working in the Startup projects will give the lessons very well, at least I have seen it for me.  Note here, if I had one misinterpretation with customer's communication and passing it to my teams in Moolya and to management, how impact that would be? Is it a big cost?  I did misinterpret in initial stages; but then when I observed it, I made sure, I will have to minimize it and keep it to zero if possible.  I worked on it and assisted the teams I was leading to practice it.  For example, say a feature and release date said by customer explicitly and I misinterpreted it. Further customer too assumed I got the message.  Were there any implicit messages here, which customer assumed that I have got it?  So do I assumed that I got it?  Once I found that I was struck with problem here in a project, I fixed it in me for first.  Note, my Moolya is always close to the tester in me!

Here is how I worked on it and continue to work on it. Despite this practice, at times I do still fail and I know I'm human and so are others.  All I check is what is the cost and value of what I did here!

Whenever I'm certain that I fully and completely understand the other person and teams without the benefit of clarifying questions, I ask myself this question -- If I weren’t absolutely, positively certain that I fully and completely understand, what would I ask?  This question helps as a trigger heuristic to know more about my understanding.  But will I do this each time?  No, I pick it in contexts which hints me something is missing. For example, when there is confusion arising in me or in teams; in the time of where major decision leads are being considered in decision being made.  If none of the team have confusion or questions, I still revisit on this and ask teams for saying what is been understood.  How I ask team members is another way of doing it. I will not get into that details, you see that's another challenge and problem to solve. Let me keep one problem here and focus on its solving.  If you said, who will do this when there is no time to do task which I have been assigned, fair enough.  In my role, I will have to do that until I get confidence -- this team or these team members can handle themselves in the situation by questioning and asking for clarification.  My role is not just to lead and deliver the projects.  I'm responsible and accountable for my team members skill development too - which is implicit expectation of an organization though it is not communicated in offer letter or in promotion letter or in promoted role.  If you don't agree or you say I should not be doing this, fine!  One day, when it comes and hits to you or to your team and organization, probably that is the whiteboard day.

I keep in mind that it’s in situations of absolute certainty in which I'm sure I understand and I say it to myself -- I'm most likely to misinterpret. I make it my responsibility to clarify my own terminology to ensure that the other person or team members understands me. As I provide clarification, I ask these questions to myself:

1. What assumptions might I be making about their meaning?
2. What assumptions might they be making about my meaning?
3. How confident am I that I’ve exposed the most damaging misinterpretations?

I see sometimes, people getting annoyed!  But my testing career experience have shown me, the decisions most times goes not conveyed well enough explicitly.  Later it is said, isn't that mean same or no change or there is a change, i.e. it is implicit.  Later, when taken implicitly and proceeded, I was asked why did I do that.  Now should I say -- that isn't implicit as last time?  Then answer is NO.  I decided that I should be avoiding this mistakes and I started to practice -- interpreting and getting clarified it, though few feels annoyed.  I see value here and cost of being catastrophic happening is high if went on assumptions for the implicit meanings that is not literally communicated.

The best solution for me is simply to try to heighten my awareness of the potential for misinterpretation.  I probably can’t catch all communication problems.  But if I do the best I can, and don’t allow myself to feel too rushed or too intimidated to ask for clarification, I should find that I'm in sync with the other person(s) or team(s). If I'm not, it’s better to find out early on, rather later when the consequences could be catastrophic.

I make sure, I ask questions and get it clarified.  I do insist on communicating verbally and in written about the explicit and implicit part of communication and clarification.  I hope this will be helpful to testers and others who work in a team and with teams to deliver the shipment on time.

To end this post, see this is not part of any technology stack, automation, testing, and roles.  It is about how we solve the problem that blocks the core problems which we are solving.  If you happened to see a value in it, do talk a word about it with your team member.  A change in team member or in you, can help your organization, your team, your product, your customer, your promotion and your salary.  

Software Testing as I see - A decade back and today, here is a start to the story of reality

This blog post shares insights of a tester who have crossed a decade, doing testing and practicing the testing.  If you are also a decade young, or bit more younger, or a tester who got into testing in recent times, you should be reading this.  If you are not a tester but you work with tester or the testers report to you or there are testers in your organization, then you should be reading this blog post.

It was around 2006, when I got into testing.  Among the standard questions, these question was common in the interview then -- explain the SDLC and explain the V model.  Forget the word Agile, it was not even relevant then.  By 2012 i.e. the word Agile started to appear as a question in the interview.

A decade back the internet was not strong as today.  Today, we have information written and web can host it.  Then, there was no community visibility as today, though it existed.  Internet was not available to all then, you won't believe this.  Today we carry internet in our pockets and palm.  

Most who hired a tester, would have asked to write test case and ask how test would be, in interview. The question around QTP, Load Runner and Cross Browser Testing.  Further a case asking the difference between sanity and smoke.  If went ahead, it was how would you test a pen, that was the question.  Then the tester was said -- you don't have to code; you don't have to learn programming; you have to test, that's big enough task.  Today, the story has changed -- we hear you to have code and not test, for a tester.  Don't you agree?  You have not been said this today?  Or did someone from your friend did not share this with you, that what happened to her or him?

If stood above all this and looked, there was one missing element.  The Software Industry then did not know how to assist a Software Tester to build her/his career.  So the management and peers, did not have much idea on how to assist a tester in her/his team.  That have come in generations passed on and as my seniors went on and got it, I too received it.

If there was one exposed and relevant tester who had vision for today's need, probably the shaping of me would have been so different.  I have tried my best and trying my best, when I learned this isn't what I should be doing as a tester.  

I know in 2019, still there are management and team, which do not know how to help tester in their organization and team.  It is no change, I see there.  But there are few companies which have shown the changes, but it is invisible when seen a picture as a whole.  All the buzzword -- Agile, Sprint, Backlogs, Automation, x Testing, CI & CD, DevOps, xDD etc., have come and striving to remain.  Yet there is no much change in life of a Tester in the Industry.   I had the same thought then as well and today as well.  Then, why didn't I solve it and just saying i.e. writing here?

I have done my best to the team and to organization with which I have worked.  I have to align to vision and goals of the organization as well, while I say the testing can be this today.  Change in the organization and industry, is not so easy.  But I see, it is a ripple, someone did there and when it ripples, the industry i.e. different organization want to ripple the same.  Note that the culture and practice who did it initially, had it so much into it; but the one followed the ripple just imitated it and imitating it, without knowing, without understanding and by not practicing like the one who did and found the value out of it.  This created the difference and misunderstanding, in my opinion.  And this is continuing today as well.  May be it will continue.  This is not a worth problem to solve in my opinion.  Because the delusion of ripple cannot be changed to who see it and want to do same.  Then, what's the point in reading further in this writing, right? Wait!

I see testers asking what should I do to programmers and managers despite they are testing for 'n' years.  Do a chef in hotel and mother in home ask how should I cook and can I do it this way?  Won't we wait for their cooking and eat?  Then, why the testers ask their managers/programmers should I do this or not -- while manager has no idea of testing as the tester has?  Learning from manager/programmer by asking is good; but not how to test and what to do after certain point of time.  I understand, we should inform the manager and stakeholders on what we are doing and why, but not asking what to do.  If asked that probably the manager should get additional (part of salary) salary of tester as well.  If the manager is not getting that additional salary, that manager is being cheated miserably.  This problem did exist a decade back, two decades back, exist today too, it will be there for in future as well, if a tester don't understand testing and practice testing.  If I'm a manager, I will in-turn ask "what you want to do, you do, just let me know in brief before you do that; if there is anything I want to say on that, I will.".  Further, I will ask, "how can I help you?" to testers.  As a manager, I will be wanting to build testers who decide what they should be in their job as a practitioners; I do not want to be instructing as steps in the test cases, step-1 do this, step-2 do this, and this is expected result and asking what is your result? It is our result, when I assist them to grow! You can read it again, the last 5 sentences in this paragraph and it is worth, I see.  

On the upper view like a view from the eagle eyes (see it is so clear), the testing management most times was in hands of people who never tested and it is continuing to be so.  Could be this is one of the primary reason why the testing takes a big hit most times and the testing practice goes to below par each time.  What could be done here?  As a tester, I feel, I should be assisting the testers who are interested and determined to be a value adding tester.  I don't want to change management mindset or testers mindset.  I want to assist a tester by pairing up in practice, that's all.  If I can do that with just fewest testers, that's enough, I feel.  Those testers will do much better when they see their practice and what it has to be.  I just want to pair up and practice with testers in their mindset and not in my mindset; this is easier than working to change the industry and management.  I see the tester as a velocity; while the industry and organization is the acceleration.  To see a better picture of acceleration, velocity has to be clear in its deeds.  Hey, I have a fundamental problem topic i.e. tester not getting the thoughts of business and management. I will write it under a different series of post, soon.  This fundamental problem has created much bigger problems than management and industry not understanding the testing. You know why, because tester did not understand manager, business, organization and industry.

In the next continued post of this series, I will share about the testing practice assistance that was received a decade back and today.  Till then, if you disagree or agree or neither of this, do let me know, if you wish.

App Crash! Testing around and inside the crash

In day and out, I come across testers, programmers, managers, and management having efforts on fixing all the crashes. Yes, all the crashes.  In a way I see, if the app did not crash, I will not know the areas that is not being handled well enough.  My testing focus areas will also have tasks noted in such areas to test and learn as much as possible. I do that task provided I can make/given time for it as it is unplanned task.

The common checks to handle crash!?

I learn, an exception if unhandled at runtime, it leads to crash.  There are multiple exception that app can witness which we never thought of during development.  In my initial days of testing, I was in assumption, if we can have -- null checks, index check, illegal argument check, and state check for an activity, we have handled most of the exceptions.  I learned, I'm wrong! How many checks can I write in the code.  I'm not a programmer by job.  I'm a tester. 

I see these checks are not enough and few more got added to my test strategies eventually -- race conditions, unexpected data, wrong data, environment factors, and many more. The collection of these checks is continuing to grow.  Do I cover all these possible crash inducer collections in my testing?  No, I can't and I won't have that luxury of time as well.  Technically, I will learn and prioritize what to use and when.

How the check looks in code, to me?

I write code for automation, which I need to assist my testing.  Here I did write such checks.  At a point of time, I saw, the automation code was full of checks.  Is that a right way?  Definitely not!  A professional and skilled programmer might not do that.  If a programmer has to have such check in each layers of the app architecture, will that sound good?  Personally as a tester, I will not design my tests that way.  As I'm not a programmer, I'm not aware of the pros and cons of doing so. At least I know that is not a better practice to have checks in each layer of architecture of the app.

By handling the exception in my automation code, I print stacktrace of exception.  But will I learn from it to be a better tester?  That's the question I have asked and continue to ask myself.  The exception fix I'm doing, is it stopping me from learning the problem which I have introduced by the actions I'm doing on the app? Is that exception blocking me from learning the underlying problem in my automation code and app?  If yes, then I have a fundamental problem which I have to work upon is my thought.

Why it crashes?

Why at all the app gets into crash? I learn, if the app gets into a state it was not designed for, it will and should crash. As a tester, I will have to learn this state (and such states) at the earliest when I experience crash and on reading the crash stacktrace. I will be happy and not make a fuss about it, if I see a crash for first.  

I learn what is the priority and impact of the crash?  Should I invest my time and test investigate further to provide as much as details to programmer?  Or should I report it with good enough details and continue my testing?  I will answer this question to me.  All I wish is, I don't want the user of the app to experience the crash.  If there is a crash, as a member from development team my intent is to keep it to minimal number having little or no impact.  I see the crash is a great source of learning about my work in the app.

I used to be fussy about crash years back be it on desktop applications, database, web applications and mobile apps.  Now, I have come to point, I love them and it is absolutely okay for an app to crash and it should crash.  What I do post crash in fixing, that tells the bigger story.  In my work, the crashes have made the app better because the team was serious about those crashes.

What on having the crash?

Should the user lose the data what she or he entered on experiencing the crash?  I personally, don't want this to happen for me.  If it happens, I will be annoyed!  That said, how to handle it?  That's something we will have to sit with the programmers and team, for discussion.  

At what point in app, encountering the crash, should we close the app and start over again. At what point in app, it is okay to note the crash and pull the stacktrace, and let continue the user using the app with data entered intact. At what point in app, I should not show UI in view on entering crash and what to show, then resume over safely from there on.  Personally I feel this is a team effort and not just a programmer's effort in making it happen.  

Few testing strategies hack to uncover crash

Here are few things I do and I ask my fellow testers to do when testing mobile apps:

1. Using the test data which will check the data integrity in app at -- entry point, during processing and post processing.
2. Identifying the states of the app and passing the invalid states in app at -- entry point, during processing and post processing.
3. Identify the input which is not from a tester and user. Classify the input on which I don't have control. For example -- the incoming intent; the app responding to APIs (default values, entered values and processed values); the app receiving the response from APIs; the device state; app's activity life cycle state and data/state exchange, and many more as this.
4. Depending on Android or iOS app, much more strategies can be narrowed down to be specific and work.  At the end, what time I'm left with in the test cycle and what do business want, directs me on what to do.

Debugging and Investigation skills

There are libraries which collects the crash on exception along with other details as -- device info, user info and network info.  I have been with programmers and had difficulty in reproducing the crash and experience it in development environment.  This said, are the logs enough to fix crash?  May be, so we handle the exception and continue the flow of app in runtime.  But did we solve the root problem that caused the crash?  No!  This is where, I feel, the skill of a tester comes in and it is very much needed.

This skill defines me what I'm as a tester and value I bring to the system.