Functional Testing Is Must In Performance And Security Testing

 

I'm sharing about how I missed to test for functionality while I was immerse focused on testing for performance of a Stored Procedure.  I was unhappy for a couple of days as I missed something that I practiced for years.  

I'm glad for reinforcing this learning with much more awareness into my testing's MVT and MVQT, now.

Context of Testing

A Stored Procedure was optimized for better execution time.  No change in the functionality.  This part of the system is not touched for a long time (years?).  There was no change in functionality here for long time (years?).  The time taken by SP was of concern.  I was asked to test for the optimization.

The complicated area, here, is the test data to use.  It took me days, for identifying and building the test data to test this optimization by mimicking the production incidents, use cases, and data.

When I got the test data ready, it was the fourth day of my testing this change.

Where Did I Go Blind By Being Focused?

The test data that I prepared is solely for the evaluation of the execution time.  This test data helped to test functionality as well.  But, my focus was on evaluating performance not functionality from this test data.

The change in SP did impact the functionality.  I was supposed to use the large data range to test for functionality of this feature which includes two SPs.  But, the task assigned was to test just one SP which is optimized.  I got blind here!  

Are you asking, what is the impact of this functional problem?

• In the one complete business work flow, this functional problem added the same data into different sets in the subsequent iterations.  Redundant Data -- This is not an expected behavior.

I just spoke performance, traces, data I/O and execution time, because that was a pressing problem.  Why?  That was the objective given to me.  

My testing mission fell short in redefining this objective.  If I had redefined it, I would, have added functionality in the better scale.

If I had redefined it, I would have pulled the other SP into functional testing which is also part of this feature's work flow.  These two SPs are expected to handle the data by eliminating the redundancy.

It was a simple test, but, I did not include/had that in my testing mission that day.

Why Did I Go Blind?

The performance test blinded me for functionality, as I saw the basic functional flow looked functioning.  But, the data count was going wrong when a bigger data range is used in the context.  

See here, how stupid I was in my testing!  I'm testing for a SP that has a change as part of its optimization for execution time.  I never brought the functional testing in.  Why?  I focused on the testing objective.

I just looked into one SP that is optimized.  I did not look the other SP which has to work along with this SP later to complete functional flow of the feature.  Why?  How is that even possible?  I was asking myself this.  I see, this is okay from the perspective of the testing objective I had.  But, not okay from the perspective of a test engineer who is supposed to think the impact and prevent the problems.

My immersed and concentrated focus on performance and its related activities on a SP for four long days did not let me see this.  

What Am I Saying Here?

While I have tested for DBs and ETL systems for years, I did not use my learning here.  What is that learning?

When there is a change in any part of the ETL, SP or DB of a system, testing for the functionality for the business workflow is equally important.  Vary the data dimensions and evaluate the counts.

I was completely hooked into the execution time and the test data while switching between the environments for four days.  The chaos in data between environments is something that misleads easily.  I fell to it this time.

I say to myself, if it is a fix for the performance optimization or a security [or any quality criteria], testing for functionality is equally important and of priority as running the tests for performance or security.

When a DB layer is picked for fixing and optimization, testing for functionality in a equal scale is must.  There is a change in the code or/and infrastructure and it has to be noted with additional attention.

To add on this, this time, I did not go through and analyze the SP.  I took this call from the test team.  This call of me costed and had a major part in letting me not to think of functionality.

My fellow colleague ran a test with varying data size by completing the business workflow and observed the problem, and informed me.  I give the credit to Sandeep.

If I had brought this performance test under the automation, I would not have done this.  Why?  I will evaluate and assert for each data returned for different sizes.  I did not automate here and there was no need for it in this context.

Redefine the testing objective that you have got; it helps when you see the model of a system and test.

Respect all the fix and suspect all the fix.  This helps in a longer run!  

Performance Testing - The Unusual Ignorance in Practice & Culture

 

I'm continuing to share my experiences and learning for100 Days of Skilled Testing series.  I want to keep it short and as a mini blog posts.  If you see, the detailed insights and conversations needed, let us get in touch.

The ninth question from season two of  100 Days of Skilled Testing is

What are some common mistakes you see people making while doing performance testing?  How do they avoid it?

Mistakes or Ignorance?

It is mistake when I do an action though I'm aware that it is not right in the context.

I do not want to label what I share in this blog post as mistake.  But, I call it as ignorance despite having or not having the awareness, and the experience.

The ignorance said here is not just tied to the SDLC.  It is also tied to the organization's practice and culture that can create problems.

To this blog post's context, I categorize the ignorance in these categories -- Practitioner and Organization.

1. Practitioner's ignorance
• Not understanding the performance, performance engineering, and performance testing
• When said performance testing, taking it as - "It is load testing"
• No awareness on what is performance and performance engineering
• Going to the tools immediately to solve the problem while not knowing what is the performance problem statement
• Be it web, API, mobile or anything,
• Going to one tool or tools and running tests
• No much thinking on how to design the tests in the performance testing being done
• Ignoring Math and Statistics, and its importance in Performance analysis
• No idea on the system's architecture, and how it works
• Why it is the way it is?
• The idea of end-to-end is extended and used in testing for performance and having hard time to understand and interpret the performance data
• How many end-to-end your tests have identified?
• Can we test for performance to all these identified and unidentified end-to-end?
• Relying on the resource/content in internet and applying or using it in one's context without understanding it
• No idea on the tech stack and how to utilize the testability offered by it in evaluating the performance
• Not using or asking for testability
• Getting hung to most spoken and discussed 2 or 3 tools on the internet
• Applying tools and calling out it as performance testing
• No attempting to understand the infrastructure and resources
• How it impacts and influences the performance evaluation and its data
• Idea on Saturation of resources
• Thinking it as a problem
• Thinking it as not a problem
• Not working to identify where will be the next bottleneck when solving a current bottleneck
• What to measure?
• How to measure?
• When to measure?
• What to look when measuring?
• Not understanding the OS, Hardware resources, Tech Stacks, Libraries, Frameworks, Programming Language, CPU & Cores, Network, Orchestration, and more
• Not knowing the tool and what it offers
• I learn the tool everyday; today, it is not the same to me compared to yesterday
• I discover something new that I was not aware of what it offered and exist
• I learn the new ways of using the tool in different approaches
• No story in the report with information/image that is self-describable to most who reads it
• And, more; but the above said resonates with most of us
2. Organization's ignorance
• At the org level, for first and to start, it is ignorance in Performance Engineering
• Ignoring the practice of performance engineering in what is built and deployed
• Thinking and advocating, increasing the hardware resources will increase and better the performance
• In fact, it will deteriorate over a period of time no matter how much the resources are scaled up and added
• Ignoring the performance evaluation and its presence in CI-CD pipeline
• The performance tests on CI-CD pipeline should not take beyond few minutes
• What is that "few minutes"?
• Not prioritizing the importance of having the requirements for Performance Engineering

Recently, I was asked a question - How to evaluate the login performance of a mobile app using a tool "x"?

In another case, I see, a controller having all HTTP requests made when using web browser. Running these requests and trying to learn the numbers using a tool.

I do not say this is wrong way of doing.  That is a start.

But, we should NOT stay here thinking this is a performance engineering and that is how to run tests for learning a performance aspect[s].

To end, the performance is not just - how [why, when, what, where] fast or slow?  If that is your definition, you are not wrong!  That is a start and good for start; but, do not stick on to it alone and call performance.   It is capability.  It is about getting what I want in the way I have been promised and I expect; this is contextual, subjective and relative.  The capability leads to an experience.  What is that experience experienced?

Sometimes, serving the requests by what you call as slow, is a performance.  What is slow, here?

The words fast and slow are subjective, contextual and relative.  It is one small part of performance engineering.

That said, let me know, what have you been ignoring and unaware in practice of Performance Engineering & Testing?

Deep Link and its Testing via Automation

 

I get these question consistently from my fellow testers and community.

1. How to automate the mobile apps and web applications using Deep Links?
2. How to automate the business flows using Deep Links?
3. How to achieve end-to-end business flows testing on using Deep Links?
4. How to automate scenarios in mobile apps using Deep Links?
5. What is the best approach to automate the mobile apps using Deep Links?
6. What is the best practice to automate using the Deep Links?

And, more questions on same pitch.

No Deep Dive into - What is Deep Link?

A hyperlink in HTML is a kind of deep link within a website or to another website.

Deep Link is known with different names for web, Android app and iOS app.  All these names have the same understanding and intent at some point.

The Deep Links are URIs that takes me directly to a specific part (activity or fragment) of the app that I'm using or testing.  The Deep Link will have an intent which tells where I will be taken on using it.

When we converse on diving deep technically into testing and automation of Deep Link, will share more insights into its internals.

Deep Link and Challenges

This question is discussed with me often:

How to do end-to-end testing using the Deep Link?

Automation of a mobile app using Deep Link poses a challenge which is not experienced in web application.  

One such challenge is, say you have not installed the mobile app.  [This is solvable!]

• On using a Deep Link, I should be taken to Apple Store or Play Store based on the app.
• I have to install the app.
• Post this, in the traditional automation, I should start traversing the business work flows via GUI.
• Is this adding to the flakiness aspect of automation via GUI?

When we talk so much about flakiness and how to avoid (not prevent), should we exercise business workflows when automating using Deep Link?  What you are thinking?  Let me know!

Scoping of Automation Using Deep Link

Back to the fundamentals.

• We have to automate, no escape from it.  Let us automate what must be automated!
• Let us not fall into trap of "Automate everything!"
• For today, I'm in this mindset and attitude,
• What we automate depends on the objective or goal that we want to accomplish.
• Each test should have precise and deterministic goal.
• A test via automation is not an exemption to it.
• A test defined in automation should be precise, deterministic and have a single objective - Single Responsibility Principle.

What is the objective of my testing via automation for the Deep Link?  This define the scope and extent of my automation.  This will minimize the number of checks that I do using Deep Link.

The purpose of Deep Link is to take me to specific part of the mobile app.

• Should I start the end-to-end or exercising the workflow to be included in the Deep Link tests?
• If included, am I not complicating the testing via automation?

Automation using Deep Link

I ask this question to myself and to my team.

What is the goal of testing via automation using Deep Link?

This question helps me to pick minimal and necessity flow actions.   It has lead and leads me to define minimal tests for Deep Link based on what we want to learn from automation of same.

To me, the purpose of Deep Link is not end-to-end testing.  It's purpose is,

Am I taken to the intended state and data when used the Deep Link?

I have kept the test intent to this.

With this, I have come with tests that has minimal must evaluation and assertion to learn if the app is responding or not to the Deep Link.  This is what the business wants when the Deep Links are created.

The app usage and workflow function is not a problem statement of Deep Link in a general context.

Deep Link is not for end-to-end.  It is to take to you from a point to another point, that's it.

Are you automating using Deep Link?

RAAMA: My Test Discovery Model

 

RAAMA -- I Look at You Everyday!

I have tried to put up one of my Test Discovery models in a conceptual way here with name RAAMA - Refer to, Arrange, Action, Monitor, and Assert.

Maybe this model helps you and your test engineering team as it is helping me.  Use this to your context with addition or subtraction for what you are seeking.

I refer to this RAAMA of me everyday and when I'm testing.  I'm finding the new learning and realization everyday that I was unaware earlier.  My understanding of RAAMA is not same what I had on the previous day.

My understanding of this RAAMA is incomplete and I have made PeACE with it by accepting it.  My understanding is growing and getting better everyday.  I will share a better version of it as I experience it.

Each time I look up to RAAMA and refer to it, I see a new dimension to RAAMA.  The awareness, exposure, and the questions are getting better giving the better realization of what I was ignorant and unaware.  The RAAMA is exposing me to be a better test engineer today than what I was earlier.

RAAMA - I Look at You Everyday!

RAAMA - One of my evolving models for Test Discovery

Note: I have not explained in detail what I mean for each node and its sub-nodes.  I can talk and discuss it with you if you look for it; I'm just one email away to get started.

Behind the Every Test Data, There is a ?!

 

Read this blog post to have a perspective about the Test Data and Test Data Management.  The point is, if I'm not aware of a test and what does it tell me to explore, I cannot think of a Test Data.

That said, if I know what I should be evaluating as part of performance, why, when and how, this will help me to come up with a thought for identifying the tests and its test data for the same. 

The ninth question from season two of 100 Days of Skilled Testing is:

What role does data management play in performance testing, and how do you ensure the availability of suitable test data sets?

Testing and "Ensure"

We test and have tests in testing, because, there is no "sure" and "ensure" idea in software.  But, we presume on a rational basis upon, "if, these are this", in a given context when the software processes.

Now, ask yourself, how can we ensure the availability of suitable test data sets?

In my opinion, the Test Data is often misunderstood.  This is the primary problem and should be the first problem, when asked "what are the challenges in creating the test data?".

When you read the concluding lines of this blog post, you will learn why I say this.

Test Data and Immunity

In my opinion and experience in practicing the Test Engineering, I see, the Test Data should be a viral strain and it should have its variants.  When this test data is used to test [experiment, test investigate, and debug], how do the software and its ecosystem respond?

• Does the software and its ecosystem is immune to this test data?
• Does it exhibit any risks and problems?
• If yes, then, do the purpose of my testing and automation is accomplished with this test data?

This puts me back to question, what is the purpose [intent] of my test?  It drives me to derive the test data which helps me to know -- What am I supposed to learn and on priority?  With this, I get an idea for what kind of test data I should be creating knowing its pattern.

If the system is immune to Test Data and not reveling anything new in the context, I classify this pattern of test data as "Immune" to the context.

In my practice and research work in Test Engineering and Software Testing, to start, I categorize Test Data into two areas.

1. Immune
2. Not Immune

Further, I have categories, under these two, where I classify the Test Data deterministically for the context.   Get in touch if you want to learn more about this.  I'm just one ping away!

The tests should help me to evaluate for the immunity and also non-immunity; both are essential and necessity.  

The credit is to me for such classification of Test Data.  It is my research work out of my practice.

Note that, Test Data is not just the input [characters or files] entered or given to a system.  Test Data has its association to tech stacks, infrastructure, ecosystem, business workflows and people.  To craft such Test Data, one has to have the understanding of the system and its internals, and, the problem it solves by knowing how it solves.

Performance Testing and Test Data

1. What is that I'm testing as part of performance?
2. What do I want to evaluate in the name of performance?
3. What part of the system is evaluated for its performance?
• Should I evaluate this in isolation or as a wholeness of the system?
4. What domain knowledge and information I should have when testing for performance?
5. What system's architecture and internal details I should understand and be aware to test for performance?
6. Is this the first delivery?  Or, do we have this system running in the production?
• If it is first delivery,
• How will I create the test data to suit the consumers of this application?
• What are the key workflows of business that we should be evaluating for its performance?
• Do all workflows and sub-systems need the evaluation for performance, and on priority?
• How do I map the fragmentation of users and their data [with its patterns]?
• What are the infrastructure and ecosystem characteristics that should be part of the test data identified?
• Does caching have any effect if the same pattern of data is used?
• If it is a running version in production
• Can I refer to the DB to figure out the pattern for the particular workflow that I'm evaluating?
• How can I match the test data to have the production data's characteristics and attributes?
7. What is the backup strategy for the Test Data?
• How do I version control the Test Data?
• Which version of the Test Data I should be using?
8. What is the threshold I'm targeting with Test Data?
• What should be the size of the data in DB when I make the IO and RW operations?
• What should be the network capability when I make the IO and RW operations?
• What should be the hardware capability when I make the IO and RW operations?
• What should be the geographical traffic and its pattern when I make the IO and RW operations?
• More of such factors will be considered when identifying and deriving the test data.
9. What is the client error yielding Test Data that I should have for the workflow?
10. What is the server error yielding Test Data that I should have for the workflow?
11. What is the redirection yielding Test Data that I should have for the workflow?
12. What is the no-response and no-change Test Data that I should have for the workflow?

And, more.  It is simple; get in touch to discuss and know beyond the listed.

To conclude and stop here, all these questions, do not ensure or assure or make sure that I will have test data for evaluating a characteristic of performance.

• It helps me to know:
• What are the tests I should be doing?
• What kind of preparation I should be having in my practice to create the Test Data for these tests?

The, Test Data should challenge the available Testability and its limits.  If it is not doing, then, we are having a test data no doubt about it; but, it is of shallow. Shallow!?

One has to ask self, "Is this sufficient enough and effective Test Data for the system [and workflow] I'm testing?"

The, Test Data should drive the engineering team to add more layers of Testability into the system.

Software Engineering - The Unquestioned Understanding of Client in Testing

Client - The Unquestioned Understanding

When asked or said "client" or "client-side", most of us assume or take it as:

• Web page displayed in the browser
• Mobile apps - Android and iOS apps
• Desktop applications

I see this is one of the unquestioned understanding and assumption in the Software Engineering.  While it is not wrong, the client does not always mean -- a web page displayed on browser, mobile apps, desktop applications, a terminal, etc.

The client is one of the subjects which we have not attempted enough to understand in Software Testing & Engineering.

A client is one who consumes the service in form of a response, and then does what it has to do.

Client - The Contextual Entity

The entity which takes the a client place [role] is entirely based on the context.  That web page displayed on a browser is a client per some model.  So is the mobile apps.  A service looking for data from Redis is a client too.

The client can be within the backend system which requests another entity to process its request and awaits for the response.  This client is not always a mobile app, web page on a browser, or a terminal where I'm working with commands.  Do you see that?

Next time when you hear the word client, ask for the context and know who is the client that is being discussed.

Client's Awareness in Performance Tests

By now, you should be breaking your assumption and the myths around the word "client".

In testing for Performance, it is critical to be aware who is client, when and how?  Evaluating this client will not be like evaluating a web page on a browser or the mobile apps.

The fifth question from season two of 100 Days of Skilled Testing, is:

How does client-side performance testing contrast with server-side performance testing, particularly in their objectives and area of emphasis?

Hope now you will question when said client-side performance,

• What client are we talking here?
• Where do this client sit in the system's architecture.

Based on this information,

• How the tests for performance is approached and executed for a client, differs.
• What is collected and observed to evaluate the performance of a client, differs.

This blog post is not to illustrate the different clients and how evaluate it for performance.  When the question talks about a particular client and in a context, I will share the approach, and how to evaluate the same.

To end, have we explored the clientless interface?  How to test this interface?

  

Testability Revisited

 

I read the below question on The Test Chat's Telegram group.

When you start working on a project, what steps do you take to establish the testability of the product?

This question is helpful in learning how we see the Testability of a product.  It is a common perception to see the product with testability and then to test the product using the testability.

But, in reality, the testability is associated more with the tests; the tests which are used to test a product being developed or developed.

So, when we talk about testability, we need to be more aware of the test that we will be designing to test the software.  This test should be quick and easy to execute with the help of the programmed or available testability factors and their attributes.

You can find more blog posts in and around testability here in Testing Garage.  Testability in software engineering and systems is one of my research areas.

Testability

I understand Testability as

• How easy it is easy to test by an engineer
• In a given context and skills of an engineer
• With the being used test approach and strategy

Note: The context can keep including factors as we add more and continue to test

It is not about if one can test the software or not.  It is all about software that is easily tested.  How easy?  That is one of the testability factors in software design and programming.

Test and Testability

Unless I know the test, I will not be certain about the Testability.  Testability does not drive tests.  It aids the execution of the test and it is a heuristic.  If the test is designed well to the context and if the testability is used well in the test's context, the execution of a test can be quick and easy.

The tests

• make use of available Testability
• helps to strengthen the Testability
• add more Testability in different seams/layers of the engineering and product

From here it will be two ways; the tests and testability will complement each other.  Further, it leads to developing and including more specific and deterministic tests and testability types in respective seam/layers.

Testability and Automatability

Testability can be classified further into several categories.  Based on the purpose and what to evaluate we will have to identify Testability in respective categories and need to use it.

As a software engineer one is bound to think testability with software programming and infrastructure.  But, testability in software engineering is not just bound to software programming.

The testability is diverse and available across engineering activities.  It is used in all engineering activities.  Maybe, for a software engineer who is hands-on with programming and testing, they infer Testability most times with programming and infrastructure.

I see, the Testability always exists to an extent.  But, can it be identified and used in the way I approach, design, execute and evaluate my test?  That is the point to explore.

If it is testable to some extent, then we are using some Testability attribute(s).

If there is Testability in a seam/layer, then there is an Automatbility in that seam/layer to an extent.

If it is automatable then there is some attribute of Testability in that seam/layer.  Again, the question comes to knowing and learning -- What am I testing and automating? Why? How? When? Where?

This discovers seams/layers to test and automate.  It leads to identifying the tests.  Then, to identify and build more Testability and Automatability.

A written program feature essentially will have an automatable characteristic and space.  If it is automatable, then it is making use of and extending the testability.

In summary,

• Know the test to know and identify the Testability better
• Know the Testability to automate better
• Know the Automatability to assist your testing better.

Context-Free Questions to Identify Testability

To know and have better Testability, here are a few things that I will want to know:

1. What is the test?
• What am I testing and what am I supposed to test?
• How is this test designed to learn and evaluate the system?
• What are the data, states, and events that I'm experimenting, exploring, and experiencing as I test this system with help of this test?
• How can I make this testing quick and easy?
• What should I use to make my testing quick and easy with this test?
• How should I use it to make my testing quick and easy with this test?
• When and where should I use it to make my testing quick and easy with this test?
2. Why am I testing this?
3. What happens if I test this and do not test this?
4. What is the value loss I will incur if not tested?
5. What is the value loss I will incur if I do not understand and learn the outcome of the test?
6. What changes the dimensions of my tests?
7. How can I learn the product better from this test?
8. What information am I learning from this test?
9. What information, heuristics, and Oracle help me and stakeholders to analyze and decide better?
10. Do I actually know the product from the perspectives of
• tech
• business
• user
• risks
• problems
• protocols
• guidelines
• environment
• money
• benefits
• exploitations
• team developing it
• and, more that I can add to the context of the product and project

To summarize, know the test and know how the test is designed.  It helps to identify better testability at the right layer/seam of the software system and engineering.  If there is no effective testability at that seam/layer, it helps to build one.  That way, the automatability also gets built in that seam/layer if the team collaborates well.

Before Identifying and Listing My Tests

 

I read the below query in TTC's Telegram chat. The discussion had started on this thread and fellow members here were responding.  Further, I read this line and it made me look into it -- "The question was we have to use valid username and password..and perform a negative testcase".

The Default Thinking and Applying Interface

Including me, I see it is subconsciously common for us to approach the problem statement visualization in terms of Graphical User Interface.  When I ask why it is so, maybe it is rooted in our subconscious thinking i.e. with first order and second order or any orders of thinking.

I want to give a try to attempt approaching it by reminding and asking self the below questions:

1. Is it a GUI specific problem?
2. Is it a problem that is tied to the context of GUI?
3. What does this question encapsulate within and open as an interface?
4. What forms do these interfaces take when I stand out of specific interface?
5. Should I stick to one interface to learn and attempt this problem?

Identify the Tests and Framing of Tests

We test to learn

• Does the system do what it is supposed to do and how, why, and when?
• When the system does not do what it is supposed to do and how, why, and when?

Should I call it Negative Tests?  This is not what I share in this post.

To me, these are tests that help me to learn when the system responds and behaves in the other way than I expected.

I can start to identify the straight use cases for inputting an error (a human introduced error) at a given state/data/event; then look for the behavior of the system.  It is good when we can keep identifying and ideating the use cases.  

We get limited with use cases as we continue to think about use cases.  That said, for sure we will identify and frame the tests within identified use cases.  But, we need tests that help to learn when the system fails in doing what is supposed to do.

To supplement it there is another way, which I use.  I do not say this is the only way to supplement.  I use multiple approaches to supplement and identify the tests.  When I do so, I ask the question to the system with the help of these tests and evaluate the response of the system.

Questions to Identify the Priority Tests

I learn and understand the system each time, to identify the better tests.  And, each time I learn something new about the system that I did not know.  

When I'm asked a question in the interview, I ask for details that help me to test better or to demonstrate my deliverable better.  I will watch the questions that I ask!

If I were the candidate who got this question in an interview, I would ask the below questions.  When I learn this is good enough for the initial tests, I will pause with questions.  I move to identify and frame the tests using the responses I got for the questions that I asked.  

These questions will surely help me to be precise and close to the context that better demonstrates my testing skills.  If it is not close, then there is a problem (or a difference) in my presenting and expectations in the interview.  I will have to address it with the help of the interviewer.

Questions:

1. What is the interface where I'm entering the username and password?
• Where is this authentication used?
• On UI (if so which UI), or CLI, or touch interface, or what is its interface type?
  
• At which layer of the system this authentication is used?
2. Where is the format of username and password?
3. What is used as Authorization identity on successful authentication?
• What happens if my authentication is not successful in the UI you want me to test?
• How do I understand that UI is communicating to me that my authentication is not successful?
4. How is this authentication processed?
5. Where the authentication is mapped to authorization and stored for references?
6. What protocol is used to communicate in authentication?
• What protocol and communication order is used to grant and revoke authorization?
7. Who uses this authentication and authorization?
• To know the different means of doing the same
8. Is there any other form of authentication that grants me the authorization?
• Do these different entry points of authentication update my authorization?
• Will I have different authorization data to authenticate? If yes, how the data, states, and events are maintained for my authentication and account?
9. What's the language and Unicode supported by this system?
• Will the languages and Unicode used in the system have any impact when I try to authorize by changing the language and Unicode?  How does the system understand these differences and maintain one state of data with authorization?
10. Are there any computing differences for authentication and authorization on big and small endian machines?  If yes, how and for what context of the system's behavior, processing, and decision?
11. Where and how the authentication and authorization details are processed, stored, and presented back.
• Is there any specific reason for doing it in this particular way?
• How you have strengthened the authentication process to grant the authorization?
• For example, 1FA, 2FA, nFA, what else?
12. Does any other system use your authentication to authenticate and authorize?
13. Do you use SSO for authentication and authorization?
14. What testability layer do I have that I can make use of to support and identify the tests?
• Does this testability layer help me to identify more tests and also classify them?

I can keep generating the questioning like this.  But I will have to pause and start working on what the questions offer me.  

With the help of these questions, I can learn better about the system before attempting to identify the test and frame it.  This also pulls out the risk or problem area if any that looks important and of priority.

I have eased my work to an extent when I know:

• the target surface area to start my work 
• what it takes and brings back, and how

In this context, I would have started this way!

Question on Quora: Debugging of "Login button not working"

 

I read a question in the Quora which said -- "We are manual testers and we are testing a login page, but the login button is not working. How can we debug that thing?". I found the phrase "not working" and that me attentive about it. The words as this make me curious and lead me to debug and learn. 

Also, I see this thought of me can help someone who is wanting to test in such cases. The phrase -- "... but the login button is not working." open up multiple possibilities of seeing what the person is saying, like:

• The login functionality looks functional; but, the login button looks to be not functional?
• How did one know about the broken login button?
• More questions like this can cross the mind of a Test Engineer!

I have tried to put my thoughts in the below mindmap. I hope it will help someone who is looking for a start in similar context lines.

A Test Design being guided by the Test Strategy

In this post, this image looks to have too much structures around it.  But, it is not. It is very simple one as I see it.  It is simple when I happen to understand the Test Strategy and what is the help out of it.  Having said this, why is that simple, because, it is from the fundamentals and nothing else.

If I understand, Test Strategy is set of ideas which directs my test ideas and test execution, then the test strategy has influence on Test Design.  Where the test design consists of these elements necessarily within it and it is as below.

1. Test Model
2. Knowing the Oracle
3. Knowing what is my Test Coverage
4. How do I operate my tests and execution of it.

The below picture says the same in the equating form.

The same applies when I have to figure out how to approach the automation for assisting my testing. I use this and when it requires a change in the context, I will do it accordingly and try to get most out of it.

Update (on 23rd Aug 2016):  I unlearn and learn the equation represented as this is more meaningful:

Test Strategy = { Ideas <--> Test Design + Test Execution}

As the test design and test execution in turn helps the ideas to evolve better as the testing progresses.

Inside Out of Test Design: Test Technique Identification with help of Test Model and Strategy -- Part 2

From the overlapping, I move ahead to learn how the test, technique and design is part of each other, I see, the three more now by adding to conversation.  It is -- Coverage, Action, Information (Potential Problem), and Assessment. This looks to be me as in the below image. 

Figure: Test Design and Identification of Test Technique

I understand this for Test Model -- a representation by which I'm learning the product either associatively or relatively or by both means. Further, I can still refine the model to specific learning i.e. to assist in obtaining the specific type of information from my testing.

Moving ahead, I will learn what information is expected from my testing. Knowing this, I will have to build a Test Model or use a existing one, to understand the product and my tests which I will be executing. The Test Model will help me to learn critical factors associated to product, environment, project and related components.

Now, I will have to choose the test techniques or build one. This has to assist me in building and strengthening tests by involving the heuristic within it also by identifying the heuristic from Test Model and tests.  This heuristic will help me to understand the outcome of test.

When I have figured out (or as I figuring out) the Test Model, Test Technique, and Heuristic for the test, I will have know what is the Test Coverage to be accomplished in context of testing.  This gives the dimensions and influencing description to say why is this my Test Strategy, and why I have made use of this model, technique, and heuristic to arrive at this coverage. As I execute my test and the way in which I record it, this overall constitutes the part of my Test Design.

Looking at the techniques, I have known there classification from the Software Testing books as Black Box, White Box and Grey Box. I'm learning there is much more within each of these types than the books say.

To summarize my so far learning, I see, the Test Techniques is one of the outcome of Test Strategy. The test strategy directs me to pick or build a Test Model. With the help of Test Model(s), I will execute the test by learning how I have to execute it and to what extent using the technique(s) (which is also a heuristic).

Inside Out of Test Design: When each disguises each other - Test, Design, and Techniques -- Part 1

I have been and continuing to be mesmerized about the Design Patterns in programming space. While I'm continuing to look at the Design Pattern, I ask myself is there anything similar in the 'software testing' space? If software programming and software testing are subset of a super set, the engineering, then design pattern and test design are to each other. That is, the both exist to support each other well enough in the context. Isn't it?  This is the question among lot other which runs in me when I think about Test Design.

Further, I go into my thoughts (of today) for letting know myself what I understand for 'test', 'design' and 'techniques', I hear below from me.

• test -- An experiment which is an open ended and whose outcome is unknown.
• design --  The set of ideas which says how to (I) use it or how to go about; and how I see it further to use it by learning. How simple it is to use for me, those are the attributes of design with the learnability in context being prominent.  In other ways, it comprises of pattern as soon as I say, 'design'.  Each design has a pattern isn't it?
• technique --  applying; i.e. a skill being applied to study, observe and evaluate a subject of interest further.

While I'm learning this, do I see a familiarity or the familiarities in these three above said?  May be this familiarity or these familiarities, it disguises each other while they are distinct and co-exist mutually in practicality of Software Testing practice.

Why one cannot see the differences or draw the differences between these other is, a simple fact. Yes, the fact, which is not studied quite often; hence it is a fact.  The unquestioned fact is, all three of these are heuristics - which helps in discovering, learning and solving, while it can fail as well.

A test does not help one to learn, discover and solve? But how a test can fail? Test never fails, it provides the information and I tag the label of true positive or false positive to the outcome of it. Likewise, is the design and technique, it helps in discovering, learning and solving; and as well it can fail too. But how do I see the 'fail' in case of design and technique is -- it did not meet the purpose or serve the purpose. Because out of the design and technique, I expect the discrete value which is of usabale, reusable and valuable. Whereas for a test, I see the information is an outcome from the execution of it (or out of not execution as well; not testing is as well a useful test in a context).  This is where the disguising appears by each others to each other.