A Test Is Not a Metric

 

A test execution by human or automation will provide information to be aware & learn.

If there is a metric, it should be for what I got to be aware of & learned.  Not for the number of tests.  If it is for a test, it is blunder, before taking it as a metric.

One can identify infinite tests; automate adding annotation @test in big numbers. Should this number be a metric?

That is a question to ask when you see a metric on this number.

What is of value to you in this outcome from a test?  That value should have a metric, and not test or the number of tests.

Identifying the metric in a context which serves, is not easy.

Number of pass/fail or green/red is a measurement; not a metric!

Anything measured cannot be a metric.

A metric helps to measure in a way that establishes rational correlation & upholds its necessity in business.

Share this awareness!

Behind the Every Test Data, There is a ?!

 

Read this blog post to have a perspective about the Test Data and Test Data Management.  The point is, if I'm not aware of a test and what does it tell me to explore, I cannot think of a Test Data.

That said, if I know what I should be evaluating as part of performance, why, when and how, this will help me to come up with a thought for identifying the tests and its test data for the same. 

The ninth question from season two of 100 Days of Skilled Testing is:

What role does data management play in performance testing, and how do you ensure the availability of suitable test data sets?

Testing and "Ensure"

We test and have tests in testing, because, there is no "sure" and "ensure" idea in software.  But, we presume on a rational basis upon, "if, these are this", in a given context when the software processes.

Now, ask yourself, how can we ensure the availability of suitable test data sets?

In my opinion, the Test Data is often misunderstood.  This is the primary problem and should be the first problem, when asked "what are the challenges in creating the test data?".

When you read the concluding lines of this blog post, you will learn why I say this.

Test Data and Immunity

In my opinion and experience in practicing the Test Engineering, I see, the Test Data should be a viral strain and it should have its variants.  When this test data is used to test [experiment, test investigate, and debug], how do the software and its ecosystem respond?

• Does the software and its ecosystem is immune to this test data?
• Does it exhibit any risks and problems?
• If yes, then, do the purpose of my testing and automation is accomplished with this test data?

This puts me back to question, what is the purpose [intent] of my test?  It drives me to derive the test data which helps me to know -- What am I supposed to learn and on priority?  With this, I get an idea for what kind of test data I should be creating knowing its pattern.

If the system is immune to Test Data and not reveling anything new in the context, I classify this pattern of test data as "Immune" to the context.

In my practice and research work in Test Engineering and Software Testing, to start, I categorize Test Data into two areas.

1. Immune
2. Not Immune

Further, I have categories, under these two, where I classify the Test Data deterministically for the context.   Get in touch if you want to learn more about this.  I'm just one ping away!

The tests should help me to evaluate for the immunity and also non-immunity; both are essential and necessity.  

The credit is to me for such classification of Test Data.  It is my research work out of my practice.

Note that, Test Data is not just the input [characters or files] entered or given to a system.  Test Data has its association to tech stacks, infrastructure, ecosystem, business workflows and people.  To craft such Test Data, one has to have the understanding of the system and its internals, and, the problem it solves by knowing how it solves.

Performance Testing and Test Data

1. What is that I'm testing as part of performance?
2. What do I want to evaluate in the name of performance?
3. What part of the system is evaluated for its performance?
• Should I evaluate this in isolation or as a wholeness of the system?
4. What domain knowledge and information I should have when testing for performance?
5. What system's architecture and internal details I should understand and be aware to test for performance?
6. Is this the first delivery?  Or, do we have this system running in the production?
• If it is first delivery,
• How will I create the test data to suit the consumers of this application?
• What are the key workflows of business that we should be evaluating for its performance?
• Do all workflows and sub-systems need the evaluation for performance, and on priority?
• How do I map the fragmentation of users and their data [with its patterns]?
• What are the infrastructure and ecosystem characteristics that should be part of the test data identified?
• Does caching have any effect if the same pattern of data is used?
• If it is a running version in production
• Can I refer to the DB to figure out the pattern for the particular workflow that I'm evaluating?
• How can I match the test data to have the production data's characteristics and attributes?
7. What is the backup strategy for the Test Data?
• How do I version control the Test Data?
• Which version of the Test Data I should be using?
8. What is the threshold I'm targeting with Test Data?
• What should be the size of the data in DB when I make the IO and RW operations?
• What should be the network capability when I make the IO and RW operations?
• What should be the hardware capability when I make the IO and RW operations?
• What should be the geographical traffic and its pattern when I make the IO and RW operations?
• More of such factors will be considered when identifying and deriving the test data.
9. What is the client error yielding Test Data that I should have for the workflow?
10. What is the server error yielding Test Data that I should have for the workflow?
11. What is the redirection yielding Test Data that I should have for the workflow?
12. What is the no-response and no-change Test Data that I should have for the workflow?

And, more.  It is simple; get in touch to discuss and know beyond the listed.

To conclude and stop here, all these questions, do not ensure or assure or make sure that I will have test data for evaluating a characteristic of performance.

• It helps me to know:
• What are the tests I should be doing?
• What kind of preparation I should be having in my practice to create the Test Data for these tests?

The, Test Data should challenge the available Testability and its limits.  If it is not doing, then, we are having a test data no doubt about it; but, it is of shallow. Shallow!?

One has to ask self, "Is this sufficient enough and effective Test Data for the system [and workflow] I'm testing?"

The, Test Data should drive the engineering team to add more layers of Testability into the system.

Test Data is Not [Equal To] a Test

 

Not every release is a critical release!

There are releases which are critical from technical and business interests.  In such critical releases, most of my time is spent on understanding the technicalities of the system and the test data identification on identifying what are the priority tests.  Identify and building the test data takes major chunk of the testing time. This effort has helped me, testing, stakeholders, projects and products immensely.

Test Data != Test

Looks like the word "Test Data Management" has become a buzz and marketing word.  In how the word "Test Data Management" is pushed, to me it looks like the intent is missing for -- how to identify the tests and its test data?

• If I cannot think of tests, how can I think of Test Data?
• How can I think of tests, the right tests in the context, and the priority tests out of it? 
• If I cannot get this, how can I get the Test Data for all these tests?

Before talking of Test Data Management, we need to talk about the intent and goal of the test.  If I have a clarity here, by being aware of the product's technical internals, architecture and its eco system, I can think of test data in a better sense.  For first, the PaaS which provides Test Data Management solution has to push and enforce for learning and mentioning -- What is the intent of the test?

When I know, for what to evaluate and how should I be evaluating it, those test intent will lead me to the Test Data and its dimensions which should be vectoring the tests.  

Test Data is not [equal to] a Test.  Test Data is one of the variance, in fact, it is a multi-dimensioned variance that a test will [need to] have.  While a test has one deterministic objective, the test data will have multi-dimensioned vectors that instruments a test.

Test Data has its role in a test, and, it is a critical role.  Hence, we practicing Test Engineers talk and spend our time on Test Data in equal importance as we give our time to identifying, designing and execution of tests.

Test Data is not a test; but the test data is an expression of the test's intent.  Test Data is one of the byproducts of a test.  Having this clarity is important.

Note this, the test data is not the only expression of the test's intent.  A test will have multiple touch points; these touch points express, advocate and can show the intent of a test.  A test data is also a heuristic as a test is.

 

Test Data Management

The word "management" is underrated and not attempted to understand from context where it is being spoken.  How do this word sound -- "Test Data Leadership"?

In engineering, especially in the software engineering, the word "management" inherently talks primarily about design.  On day-to-day operation, the management designs its strategy and approach to solve the problem and challenges.  

When said "Test Data Management" in software engineering, it is about strategizing and approaching the problem of identifying and categorizing (subsets) the data to test.  A kind of leadership at one layer.  It has its role and critical to the deterministic outcome for a test.

In machine learning, we can see this categorization -- training data and test data.  Then, is training data not a test data?  It is!  The training data is designed [managed] to train and this data as well identifies the problem as the training progresses.

The data which are identified, designed, categorized and collected as Test Data, are well sampled data in the context, for the intent of a test.

To conclude, Test Data Management is one of the correlation in software engineering to solve a problem; it is contextual in how and what tests and testing is executed.  The different teams and organizations has their own way of managing the test data.  How they do it, it is their way of doing it.  We have to look Test Data Management from point of Test Engineering and not as an another buzz word to sell just as a product and business's service.

Waterfall or Agile: Testing for Performance - Where to Start?

 

Do you understand the Agile?  I have shared my understanding here; give it a read.

The eighth question from season two of 100 Days of Skilled Testing is:

Can you share some best practices for conducting performance tests within an Agile development environment?

Best Practices and the Agile

The irony is, the Agile says, there is no best practice.  It asks, to tailor and fit the practice to the context so the continuous delivery and value is delivered consistently upholding the Agile's principles.  

Yet, we talk about the best practices in the Agile's context, like the eighth question asked here.

What is the effective way to test in the continuous delivery?

As a test engineer, how can I start thinking and testing for performance from the inception of a feature's thought?  I see, it is not hard to do so.  As you read further in this post, you will have a perspective and awareness to do it.

Performance in Waterfall and Agile

I learn, the performance is an experience.  It does not differ because of the Waterfall or Agile.  If the performance is not a pleasing experience, it will impact stakeholders no matter it is Waterfall or Agile.

But, the question when evaluating for the performance is -- where to start, when to start, how to start, and with what to start?

As of today, I do not see differences in the mindset and skills one has to have for testing of performance in Waterfall and Agile.  Could be the approach differs in certain phases here; otherwise, I see the same in both practices.

I will rephrase the eighth question to this:

What is your practice to evaluate the performance right from the start of product development in your project?

I do not want to wait until to hear -- the development is completed and deployed; now we can start running the performance tests.

What can I do as part of performance tests from the first day of development and first commit?  This is my intent and area to look in strategizing the testing and tests.

The Culture of Engineering

At the start and end of the day, when we developers start and finishes the work,

• How the work is done and why, is defined by the engineering culture practiced by that organization.
• The Performance Engineering of the software products and solution being built will be driven the by the culture practiced.

The Test Engineering and how we test and automate will be driven by the culture of engineering practiced in the organization.

Writing the code not just for building the functionality, but, also for performance is a culture driven factor.  The organization's culture for engineering practice drives it!

Testing for Performance - Where to Start?

I'm sharing my research work that I'm doing and experimenting on performance engineering and performance tests.  I'm seeing the results and value out of it and so are the stakeholders.

Today, we are getting skilled in exploring and testing without the requirement document and SLAs in hand.  Isn't it?  Haven't you?

I use my MVPT to figure out what are the minimum performance tests for the feature.  As part of this, I will explore with help available aids to evaluate the performance.

To start, I will use these questions to figure out the performance tests:

• What is the minimum viable questioning performance tests that you have got to test this feature?
• What is the minimum viable questioning performance tests that you have got to test this workflow?

Unit Tests for Time and Space Complexity

I will work closely with programmers to gather information on below when the code for the feature is committed as part of Unit Tests.

• The execution time taken by the code of that feature - the Big O Notations for space and time complexity
• Usually the Unit Tests focuses on functional tests and clean code practice
• But, when we test team ask and push for performance data, this can come as part of Unit Tests
• An architect or a principal engineer can set an expectation on
• What should be the time and space complexity of a code for a feature?
• Each functions and blocks need to be evaluated on this
• As said earlier, this depends on a engineering practice culture of an organization
• If the culture wants it, it will be there; else just the functional code will be delivered and not the performance code
• If the time and space complexity analysis outcome is not as expected, the code written has to rethought and refactored
• The review process need to put it back
• The comment with data has to be published
• This will be useful to model the performance tests by test engineers who will be working on performance tests
• Doesn't it look a like a effective useful practice as part of Performance Engineering right in the early stage?
• This is very well applicable to projects running on Agile or Waterfall

Do you have this in your project and Unit Tests written?

The time and space complexity questions should not be confined just to the SDETs [test engineer] interview.  A test engineer has to ask for it and apply it in her or his day-to-day work.

Profiling Tests by Test Engineers

We testers do not get into product's code analysis.  We have to build skill to run the profiling on product's code and analyze the resources data.

• Test Engineers can test the feature's code with the help of IDE's profiling (runtime analysis) and collect the performance data by identifying the performance bottlenecks
• This runtime analysis can profile for
• Memory snapshots
• Thread analysis
• Monitoring resources
• CPU and allocation profiling
• And, more
• The problems and risks can be reported upon analysis
• Compare the two different solution's approach performance data

This information will tell and indicate where is the risk and problem when we deploy the code.  In my opinion, this is a useful information in modeling further performance tests.  This information is first-hand information which is very powerful before we start using any other performance testing strategies and tools to aid the tests.

Get Started with Performance Engineering and Tests

These are available in the IDE.  We think of performance testing tools and ask how to test for performance.  To be precise, we test developers (test engineers) should change our mind and shift for first.  If not, as I say, we will be the bottleneck for first to ourselves.  Did you know this way of testing for performance?  Why not you introduce this in your project and organization?

If seen, these test practices can be used right from the day we commit the feature's code. This is a place to start for the performance tests.   This will be a differentiator together with MVPT and guides the MVPT to design effective performance tests in the context.

I do not say these are best practices and there is no best practices.  But this is a useful practice when the organization and stakeholders ask for it.  Let your organization and stakeholders know how well you can test for performance right from first commit of product's code.

To stop and end here,

1. Just do not test for functionality from day one, also test for the performance from the day one.
2. Influence your organization's engineering culture and developers not just for developing functional code, but, also for the performance code

Is Performance a Perception to an Engineer and User?

 

When hearing about performance from customers and engineers on team, I see each are having a perception of it on using a product.  To one it was fast enough, to another it was as usual and for one it was too slow.  Each are expressing their perception.  But, what is the performance?

What is Performance?

I see, understanding and knowing "What is performance?" is important for everyone who is involved in building the software system and product.  In my opinion, this should be the start point.  It is beneficial, when everyone has a shared common understanding to it as a team and business.

Performance is an emotion to a user!  Technically, the performance has multiple facets to it for understanding the capability of a system and its sub-systems.

Facets of the Performance

What facet do you consider and call it out as Performance?  This is another point where most of us do not align.  For example, below are the different facets that we usually hear and read a lot:

• Heap and Memory
• Threads
• Not supporting for concurrency
• Not handled well in concurrency
• Holding up other processes and causing bottleneck cases
• Memory Leaks
• Concurrency
• Memory not reclaimed
• and, more ....
• CPU consumption
• Open connections and its I/O
• Held up in processing requests
• No enough resources to process
• and, more ....
• DB I/O
• Open connections
• Unindexed data
• SPs and Queries holding the transactions
• Incorrect configurations of DB Server and nodes
• and, more ...
• Disk I/O
• Running out of space
• RW I/O not responsive
• Network consumption
• Unmanageable transactions
• Transaction's data, size and time
• Latency
• Latency in which interfaces?
• Ineffective caching, queuing and messaging
• GUI not rendered or painted
• GUI exhibiting jank behavior
• GUI partially rendered
• GUI not in a interactive state
• GUI not responding to an action
• GUI and UI loading multiple times with no room for interaction 
• Terminal yet to return and show the prompt
• Older and deprecated libraries with latest dependencies
• Server, orchestration, and sub-systems configured incorrectly
• Hardware resources and its specifications used in a context
• Display refreshing rate and frames lost with GUI rendered
• Heat dissipation
• From the hardware
• Experienced in the environment
• Time taken for a request to reach the actual end point
• Execution time on receiving a request
• And, more...

Further we have classified it to frontend and backend; both are important and equally needed.  The webpage has got different KPIs and metrics to determine where do its performance stand.  Likewise, for backend.

Which facet of performance need to be evaluated and in which phase?  Why?  The perception will be established when testing, on how we test it, and how one uses the product.

With all these for performance, where to start and what to look at?  This is one of the question with which we are left in Waterfall and Agile.  That is where the eighth question of 100 Days in Skilled Testing comes in -- Can you share some best practices for conducting performance tests within an Agile development environment?

Have you asked these questions to yourself and team?

1. What is performance to you and to your stakeholders?
2. What should I consider in evaluating for the performance of your software system?
3. What is the practice I want to pick to evaluate the performance?
4. Where do I start and how?  

MVQT: The Testing and Tests with a MVP's Perspectives

I was leading multiple teams and its delivery in a testing service company.  Then, I came up with this thought -- Like MVP, I also have the MVT (Minimum Viable Tests) for a MVP.

Further, I expanded this thought in my day-to-day practice on tailoring to different contexts. I'm observing that it is applying well to the different contexts when I tailor it to the contexts.  After experimenting it for 10 years, I'm sharing this as a blog post.

What is a MVP?

I take this from Eric Ries.  It looks simple and precise to me.

The Minimum Viable Product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.

I see this technique [and a concept] can be applied to anything when I'm developing.  As a test engineer, I develop the tests and test code in major as part of my testing.  On applying the idea of MVP to my testing and deliveries, I see the value and result.

Reading this blog post of me to know who is a developer.

Testing, Tests, MVP and MVQT

In software test engineering, I see the MVP as Minimum Viable Questioning Tests.

The Minimum Viable 'Q' Tests (MVQT) for a focused area of a feature [or to a feature]

• Helps me to identify the priority tests that should be executed for first
• Allows me to learn information on priority which matters critically to product and stakeholders
• So that a informed decision can be made.

The Q in MVQT stands for "questioning".  I read it as Minimum Viable Questioning Tests.  I see the "Q" as a placeholder for the Quality Criteria.  That is, MVFT means Minimum Viable Questioning Functional Tests to a feature or a workflow.

The MVQT are key to know:

• Have I identified and designed the priority tests?  How do I know that I have got them?
• Did stakeholders get the information which they wanted to know on priority?
• Did MVQT help me to
• Explore and know what I wanted to know about a feature or a workflow?
• How fast I was here to know and learn this?
• How did I develop my tests incrementally?  Did I?  If not, then, is it a MVQT?
• Did MVQT help me to know
• Am I aligned and in sync with expectations of my stakeholders and customers who are using the software product I'm testing and automating?
• Did the MVQT help me 
• In collecting the critical information in a given context for the scope of testing and automation?
• Do the learning and outcome from this MVQT help to reinforce the validated learning of customer?
• Do MVQT result support the outcome of Unit Testing result?

The tests in MVQT has to be consistently revised and evaluated to keep it as a MVQT.  Note this, not all tests are MVQT.  If the number of MVQT is growing to a part of feature or to a feature, it is time to think about what is MVQT for you.

The "minimum" tests are highly effective and it helps me learn and test better technically and socially.

MVQT and Testing

• Sanity or Smoke Tests
• The set of MVQT which helps me learn can the build be taken further testing
• MVFT - Minimum Viable Questioning Functional Tests
• Apply this to a feature or a workflow or to that part which can be evaluated with minimum tests for its functionality
• To update is this aligning to the validated learning of customer [stakeholders]
• MVPT - Minimum Viable Questioning Performance Tests
• MVUT - Minimum Viable Questioning Usability Tests
• MVAT - Minimum Viable Questioning Accessibility Tests
• MVTxT - Minimum Viable Questioning Tester's Experience Tests
• MVST - Minimum Viable Questioning Security Tests
• MVAF - Minimum Viable Questioning Automation to a Feature
• MVLT - Minimum Viable Questioning Localization Tests
• MVUIT - Minimum Viable Questioning UI Tests

You add more of this to your list and context.

In a way, MVQT should ask and look for the testability, automatability and observability.  If this is not happening, then there is no possibility of saying I have got my MVQT.

More importantly, in the CI-CD ecosystem, MVQT pays a major role.  If I should have my tests in the  CI-CD pipeline, then, the MVQT is the way and it focuses on a targeted area to evaluate it.  Else, it is hard, impractical and not possible to test in CI-CD eco system by delivering continuously.

Ask and Review for MVQT

Ask for MVQT, when you review these:

• test strategy, test framing, test design, test ideas, test cases, test plan, test architecture, test engineering, testing center of excellence, and test code

For example,

• What is the minimum viable questioning performance tests that you have got to test this feature?
• What is the minimum viable questioning performance tests that you have got to test this workflow?
• What is the minimum viable questioning security tests that you have got to test this feature?
• What is the minimum viable questioning GUI tests that you have got to test this feature?
• What is the minimum viable questioning contract tests that you have got to test this end point?

Likewise, What is the minimum viable questioning automation tests that you have got to test this feature?

Ask how these tests qualify as MVQT in this context of testing and automation?

This should help you to see how effective is the test strategy in a given context.

Importantly, the MVQT and its effectiveness is a testability to test your tests.

The Credit is to Me

I'm not sure if the idea what I'm saying here in this blog post is practiced by other test engineers.  I have not seen this being discussed about it in public forum.  I have not come across it in my awareness and to the exposure I have put myself.

Hence, I will take this credit to me.  Giving the credit honestly is not a common sight and practice.  I have not got my due credits for using the ideas, thoughts and work that I have come up with.

So, I make it as a open letter and call out that credit for this idea, thought, concept, and practice will be to me when you listen, use and practice it.

Who is a Developer?

 

Who is a Developer? 

I understand the developer is the one who builds.

• A product owner builds the product by bridging the gap between market, business and consumer.
• A programmers builds the product by programming.
• A tester builds the product by testing it with her/his tests and programming the test code.
• A devops engineer builds the product by building and managing the pipelines and environment.
• A DBA builds the product by writing and managing the database environments.\
• A support engineer or executive builds the product by troubleshooting and listening to the problems being reported.

Who else do you see in developing the software system and its product as a solution?

If you see,

• With the skills and expertise we have and build consistently,
• Each of us will work with our expertise in a particular space of the problem solving as a team and business.

Each of us develop some kind of artefact that are combined to build and ship a usable software.  To me we all are developers with a specific roles.  As a developer we upskill consistently and catchup with current and upcoming changes.

Note: The above said are not the only tasks of these roles.  Each do beyond this in developing & continuous delivery of a usable software.

The Agile & Testing: We're Pulled & Hidden from "Shift Right"

Do I Understand the Agile?

If I work in a project which follows the Waterfall approach, I still can apply the Agile Manifesto and the 12 principles behind it.

But, am I adaptable to the change and uncertainty for better ways of developing software and helps others do it?  This shows the mindset of me and others involved.  Have you heard this line, "every project that claims Agile, it has [a small] waterfall in it?".

Agile Software Development is not wrong!  It has values in what it says and for what it stands.  I have no doubts in it!  It describes how I perform a activity and respond to a situation in the software development.

Frameworks and Methodology

We have picked the frameworks which are under the Agile's umbrella. We pick the methodology that suits our work and have customized it to our contexts.

This is what Agile also say, the team will always need to adapt its use of a framework to fit properly in its context.  

Dr. Alistair Cockburn, says, the Agile methodologies are the conventions that a team choses to follow in a way that follows Agile values and principles.

The Agile is Not a Problem!

We do not collaborate well in solving the problem and say Agile is not working and creating more problems.

In reality, how we respond to the uncertainties and what got prioritized, it will derail us.  We derail from what we should be achieving as team [and as an individual] in a given time frame and context.

The Agile as I Understand

In a nutshell, the Agile is about

• Balancing effectively with collaboration in the context we are.
• Identifying, learning and solving the problem as a team no matter where we are in the project's timeline.
• Being ready to face the uncertainty and responding to the changes.
• See the progress and value in what we do as a team and bring the "continuous" in practice.
• Having the consistency in the excellence of our work and the value we deliver with it.

Read the Agile Manifesto and its 12 principles.  It is the day-to-day life of a team who works to build a software in a continuous delivery having the values to a stakeholder.

Testing and the Agile

Testing is a collaborative activity.

The Agile also talks about collaboration as an essence and necessity in the continuous delivery of the software we develop.  

The Agile talks about short interval consistent deliveries.

• How should I fit myself as a Test Engineer here?
• How should I upskill myself to deliver in short intervals as part of continuous delivery?  

I ask and say to my test teams,

• Think of delivering value consistently in the shorter intervals.
• Rather saying -- releases in short intervals, call it out as a value to a stakeholder in regular short intervals as a continuous delivery
• It sounds positive, affirmative and profound.
• Upskill and help your peers to build the skills. List the skills, let us discuss and start our practice on it.
• This mindset changes the perception in how you approach the testing.
• In these short intervals,
• Let us prevent bugs in the ways we can think of.
• Finding bugs also need resources.
• Let's start investing in building and improvising the resources to prevent bugs.
• How can we prevent and where to begin?
• For what should we test and automate other than functionality in these short intervals?
• What is the priority?
• If I do not test and automate for evaluating any particular quality criteria, does it impact the value to a customer in what we deliver?
• If I do not test and automate on a particular seam and system's boundary interface, does it impact the value to a customer in what we deliver?
• How am I collaborating with stakeholders?
• Who are my stakeholders?

In short, I see the Software Test Engineering practice in the Agile setup as this,

• We [team] exploring technically and analytically;
• Evaluating collaboratively in the software development and engineering, by having the effective staged feedback loops in place to improvise the delivery;
• By responding actively to the changes and uncertainties that comes anytime in the timeline of a delivery and practice;
• Thus, the consistent continuous delivery is delivered with value to the stakeholders.

Note: This is my observations so far having worked with 46+ startups in my software test engineering practice.

We're Pulled and Hidden From - Shift Right

I and you are said to Shift Left and start the testing from the inception of a feature's discussion.  And, further we're said, we have to shift left from right.

Did the Agile manifesto and its 12 principles say explicitly to Shift Left from Right?  Who said that?

What I see, How I see and understand the Right and Left here, matters a lot!  If I see the Right within the delivery loop, I'm setting a trap to myself to be late and lag back.  And, probably this trap gets set in the work, that is, to see the Right in the [continuous] delivery loop.  At certain context in continuous delivery loop, starting from Right is a need and necessity.

I see the Right outside the [continuous] delivery loop, as well.  In this Right, I see:

• What is coming up in the tech stack and technology?
• How this is a need?
• How this [is to be and] will be adapted by tech solutions we deliver?
• How should I upskill myself here so that I fit into the loop of continuous delivery?
• In short, the Right should be observing the trend and its analysis so that I'm aligned being agile.
• I will be skilled to test and automate in this change and uncertainties.

The Agile never said to completely move or step out of the Right.  Or, I have not read it in its manifesto and principles. 

You see and learn, what is your Right and ask am I aligning to it for being agile in the Agile.  After all, the Agile says adapting to the changes and uncertainties, yet delivering the value to the stakeholders consistently in short intervals.

If seen closely, the Agile is also about balancing, isn't it?  If I'm moving towards Left and me standing in the Left, am I balanced?

In short, we are being pulled and hidden from Shift Right.

Shifting Right is important to see what is outside the delivery loop.   Are you seeing your Right?  If yes, what is the trend analysis you see?  Where you need to upskill?

Shift Right can also be to look what you can bring into your Shift Left for better development and delivery.  For example, what you see in the market and business space for your product?  Do I have anything to add now in Left from this learning?  Do you see that?  Are not we missing this when we pulled and hidden from Shift Right?

Left or Right, if I do not respond, adapt and deliver for the changes, I need to ask, -- Am I agile and testing in the Agile?

To conclude, know what is your Shift Right while you are delivering values in your Shift Left.  Strike the balance and ace.

Exploring to know the Web API - 101

 

Here is what I look when I'm exploring to learn an end point of web API for first.  I'm talking of the Web API that uses HTTP protocol.

1. What is the purpose of this end point?
• How it helps and to whom?
2. Who uses this end point?
3. Identify the host of the end point
4. Identify the path of the end point
5. Identify the API's end point
6. Identify the version of the end point
• Know the different version that are available for an end point
• What are active?
• What are inactive?
• Who are the consumers using these different versions of this end point?
• What's the difference between these versions?
7. Identify the HTTP method of the end point
• Know the different HTTP methods this end point supports
8. Identify the resources the end point interact with for CRUD activity
9. Know the HTTP Request headers it needs, uses and good to have
10. Know the HTTP Request payload or data and its format
11. Know the HTTP Response headers and good to have
12. Know the HTTP Response Status Code and what actually it should be returning
13. Know the HTTP Response content and format returned
14. Does this end point enforces any contract with the consumer?
15. Know the intent of your test on this end point
• One test in one request is useful unless the context demands otherwise
16. Is there anything cacheable in the end point's response?
17. How the request payload is?
18. How the response payload is?
19. What is the language in which the end point is written?
20. What is the language in which the resources that end point updates is written?
21. Any framework used to build this end point?

On learning this, I continue to next phase in exploring and knowing the web API.

The Actual Performance Bottleneck is a Test Engineer's Awareness & Practice

 

The bottlenecks are necessity.  We cross bottlenecks everyday; just, we do not observe it.  If you have not read my today's thought on the bottleneck, read it here.  

The seventh question from season two of 100 Days of Skilled Testing is:

What are the common performance bottlenecks, and how do you go about pinpointing and resolving them?

I have a different opinion and thoughts to share on this question.  It may look weird, but, I see this is the reality.

Let me share what happens to a practicing Test Engineer when going through a bottleneck in practicing a subject to upskill.

• The time taken to accomplish tasks and its milestones looks unusually high - the other extreme end.
• Some attempts will not even kick off on entering the bottleneck for several reasons. 
• One of the reason is the fear and what my peers think if I fail to accomplish it.
• I lose the focus.
• It leads to lose of interest, unhealthy confidence and the inconsistency.
• The procrastination kicks in.
• Without no self-motivation, determination and consistency, we give up; clog and remain in the bottleneck.  One will fail to make use of the bottleneck to scale self.
• One do not go back or come out of it.
• We start to blame the bottlenecks.
• But, we fail to understand that the bottlenecks are part of the system.

What makes one to have a hard time with available bottleneck is,

• Ignoring the bottleneck.
• Not being aware of the benefits and impact from the bottleneck's existence.
• Its effects are adverse in a longer run
• Likewise, its benefits are immense in a longer run

If seen, the existence of bottleneck has two extreme ends -- good and not so good.  The bottlenecks cannot be eliminated or eradicated; it can only be managed with awareness and knowing how to adapt and scale through it.

Me, The Actual Performance Bottleneck

This may appear as a critical self evaluation.  But, it is not.    In my experience and practice, I'm learning consistently,

• If there is any difficulty in communicating about the tests and identifying the information from my tests,
• It is do with me for first.
• I will have to communicate it and why it is so.
• If I'm not exposing myself to a subject's area and its practice by putting the subject to test,
• I'm the bottleneck.
• Further, the other bottlenecks that I encounter from all other systems, it easily compounds the problem and its impact.

Did you see the bottleneck is a necessity? It drives me to scale and be operable.  If no bottleneck, may be, I will not expose myself to the awareness to identify and learn the information.

Call out any testing with a name, for first I will be the bottleneck to myself and to my testing, if I'm not aware of it and not practicing in it.

For example, I'm asked and taught to do functional testing at a GUI layer.  Why I'm not asked and taught to do the performance and security tests at a GUI layer?  Or, why I do not think of it?  Should someone say me to think of it, practice it, and, do testing for the same?  If the floor and industry do not push me to do so, I will have to open myself to create that opportunity and awareness.  Did you see that, this is my performance bottleneck?

To test, evaluate and identify [pinpoint] the performance bottleneck in the software system, I will have to practice Performance Engineering & Testing.  Only then, I will be able to identify its existence precisely before pinpointing.  If I do not practice by building and refining the awareness, I cannot explain what I'm pinpointing.

I pinpoint myself [the bottleneck] first, if I identify wisely what is the performance bottleneck.  If I fix my bottlenecks, I will be able to identify bottlenecks in the system that I test.

Let me practice the performance engineering and help my team, and community to practice it.  If not, I will not be in position to name a bottleneck, forget pinpoint it. I will just follow, what others say and think it is right.

Note: I share the above from my personal experience in testing practice and also from reading [& answering] the performance testing questions that are posted in the communities social spaces.  I see fixing this and practicing better here is important for first, than identifying and solving the bottlenecks in software system.

The "Bottleneck" in a Test Engineer's Eyes

 

Preference to Bottle Over Jar! Why?

Have you heard Jar Neck anytime when describing a problem or solution?

• I have heard Bottleneck often and consistently; but, not Jar Neck .  Why? 
• Be it in Software Engineering or day-to-day life problem solving description,
• The Bottleneck is referenced and not a Jar Neck.

Looks like people want Bottle but not the Bottleneck speed and benefits.  Bottle without its neck is a jar?!

Bottleneck exist for better controllability.

• In a bottle, the bottleneck is a solution!  It is not a problem!
• It is to mitigate any risk and problem that arises from the flow of content in the bottle.

Yet we describe, learn and communicate the neck of a bottle as a relativity and analogy to a problem.  

Are you aware of Gateway in the software system?

• The Gateway can be seen as a neck of a bottle which controls the incoming requests and outgoing response.
• Gateway is a necessity.
• We need Gateway to be adaptable in size of its neck based on traffic volume it is handling.  Here, the gateway's neck size should adapt and scale contextually.
• When describing a problem, we are talking about how this bottleneck size which is not adaptable for the context.
• That adaptability has to be built in engineering to scale in any dimensions and magnitude.
• When this is not done, we equate the software system's problem to a bottleneck as a analogy, which is incorrect!  The bottle has got its size and its neck size fixed for a purpose and as a solution.
• The context of a bottle and today's any systems are different.
• It is good to draw similarities from General Systems Thinking and observations.
• But the solution cannot be generic to all systems; it has to be contextual.  The software system has to have its contextual solution.

So, next time when someone in your team or network talk about bottleneck, do share them bottleneck is for better controllability.  Having a contextually resizable and adaptable bottleneck is the need for Software Engineering; not the elimination of bottleneck.

In fact, a software system should have and will have a bottleneck in a point.  And, this bottleneck will be adaptable to the context for having what it should let through and process.

Is the runway of an airport a bottleneck when it is compared to a sky?  Is that a solution or a problem?  Likewise, the ship will have a defined route path and it does not sail without a route path.  Is this a bottleneck to ship and its business?  A elevator can accommodate the defined number of people or kilograms allowed, and not beyond that to move.  Is that a bottleneck?  The esophagus in human body has a size which medical science observes as normal and acceptable; any deviation from that size measurement, the medical science test investigates it as a risk and problem. Why?  Is the circumference size and length of esophagus a bottleneck to human anatomy and physiology system?

The engineering solution will and should have a bottleneck at a point.  Having a adaptable bottleneck to the context is one what tries to accomplish in a software system's scalability and operability.

Please, do not equate solving a "bottleneck" situation with Agile practice.  Does it look like a joke?  I will not be surprised if someone says bottleneck problem is solved if practiced Agile.

The Internal Metrics of High Performance Mobile App

 

Do you have a question -- What are KPIs and Metrics and the differences between them?  Then read this blog post.  It will help in knowing the differences and how each help the other mutually and remain exclusive.

The sixth question from season two of 100 Days of Skilled Testing is:

How do you determine the essential performance metrics and Key Performance Indicators (KPIs) when assessing the performance of a mobile native app?

Why I Focus on Metrics Here?

Here, in this blog post, I will focus on Metrics and not KPIs.  If you have read the above referenced blog post on KPIs and Metrics, you will learn,

• The KPIs are defined objectives by the business.
• Knowing this is important.
• But, to accomplish the KPIs objective, one has to extract the contextually suitable metric from the system and have to evaluate it.

As a test engineer, I evaluate from technical perspective with an orientation of business thoughts.

• So, the identifying and defining the metrics makes more sense to my role.
• Hence, I pick the Metrics.

A KPI example for a mobile app,

• The set percentage [what percentage?] of five start rating in store with positive emotions in a review.

What may be the KPIs, I will have to correlate it with [a] metrics so that we work towards accomplishing the KPIs set.

What is Performance to Mobile Native App?

The Native App

A native mobile app is an app developed for a particular mobile device or platform like Android and iOS.  The native apps developed for one platform cannot be run on another platform.  That is, Android app cannot run on iPhone, and vice versa.

The Android apps are written in Kotlin, today.  The iOS apps are written in Swift.  The native apps can be developed in a way that it can run both in offline and online mode based on its business objectives.

An example of the native app which we can easily understand is,

• WhatsApp for Android and iOS devices

Performance and Native App

For first, my mobile device is not my customers device.  The Android device fragmentation makes it much more challenge with computing power offered for devices by OEMs.

Though, iOS have its fragmentation on OS version and device's computing power, it is not huge when compared with fragmentation of Android devices.  This is a challenge for Android mobile app in all aspects and especially in performance.

The performance will indicate different advantage, risks and problems to a native app on a platform and its devices.  The performance can be classified into different areas for native mobile apps.

It can range from, being deep technical areas to the experience a user expresses in using the app in a given context.  Everything is performance here!

Hence it is not easy when talking about performance in the mobile's app space.  While it is so ambiguous and hard subject, how one can pull the metrics for the performance here?

From a technical perspective, the word "performance" is not specific; it is vague.  If one says, the mobile app is performing well, what does it mean?

• Is it fast?
• Is it consuming low battery power?
• Is it consuming less memory?
• Is it not consuming much network?
• Is it smooth to interact, responsive, and no jank experience?
• Is it intuitive and secure?
• Is it number of crashes?
• Is it having a consistent update and bug fixes?

What you say?

All these leads to a question - What is a high performance app?  You ask this question to yourself and to your tech team, business team and consumers.  Figure out what is a high performance app to them.  This helps!

The Common Metrics - Android and iOS

Below are some of the common metrics for which a Test Engineer extracts data and evaluate it via testing and automation.

1. App Install Time
2. App Launch Time
1. Cold Start
2. Warm Start
3. Hot Start
3. Private Memory Size of App on available Heap
4. Number of Views
5. Garbage Collection - Frequency
6. Data Residual Size and Sharing
7. Network Payload Size
8. Energy Consumption in Workflows
9. Wakelocks and its Impact
10. Frames Skipped
11. Open Threads and Processes
12. Storage & Data Size
13. App Size

And, more.  Each of these are own deep area for analysis and tuning the performance.

I have been practicing this for last 11 years in my testing.  It is one of my research areas in Software Testing & Engineering

These days, instrumentation also offers different metrics for the mobile apps which can correlate with the KPIs set for the native mobile apps.

To conclude, start understanding the internals of Android and iOS.  It opens up you to the performance and practice of high performance apps.

I'm happy to share if you are interested in practicing these subjects and testing.  Let us connect and converse!

What are KPIs and Metrics?

 

I use to have this question - Are KPIs and Metrics the same?  Especially when I started to learn and practice the Performance Testing, this question bounced back to me often.

Do you have this question?

The Use of KPIs and Metrics

When business and stakeholders talk about it so much, there should be a value out of it.  What is that value?  Why it is important to identify and capture the KPIs and Metrics?

The KPIs and Metrics are derived from data we collect.  These data are processed to extract and normalize, so that, it is in a state as expected by the consumer for making a decision.

The stakeholders will make decisions and take actions referring to KPIs and Metrics. For example,

• The number of Daily Active User (DAU) is a KPI and also a metric.
• But, a metric cannot be a KPI
• How many of this DAU, closed the transaction within five seconds using a wallet?
• It is a metric; not a KPI.

Another example,

• KPI
• How many users installed the latest version of the mobile app and have signed in?
• If there no active users on latest version that indicates a kind of risk and problems to the business.
• Reopened Tickets in Customer Care
• This indicates there is something going wrong
• Metric
• What is the average time taken to see a streaming screen for users in 4G data network?
• If this is not captured, there is no data for business to establish a relationship with KPI set.
• Average Reopened Tickets in a customer service
• The distribution and time towards lower number
• The distribution and time towards higher number

KPIs and Metrics are not the same while both have quantitative measurement. Both are different.  Identifying and knowing the difference between them in your context is important.

They go hand-in-hand when setting a direction and action.  So that, the business and stakeholders realign to the goals and objectives defined.

KPIs vs Metrics

To conclude this post, investigate your metrics and question why the chosen KPIs.  It will help you to design your Test Models and identify the tests in given context.

Software Engineering - The Unquestioned Understanding of Client in Testing

Client - The Unquestioned Understanding

When asked or said "client" or "client-side", most of us assume or take it as:

• Web page displayed in the browser
• Mobile apps - Android and iOS apps
• Desktop applications

I see this is one of the unquestioned understanding and assumption in the Software Engineering.  While it is not wrong, the client does not always mean -- a web page displayed on browser, mobile apps, desktop applications, a terminal, etc.

The client is one of the subjects which we have not attempted enough to understand in Software Testing & Engineering.

A client is one who consumes the service in form of a response, and then does what it has to do.

Client - The Contextual Entity

The entity which takes the a client place [role] is entirely based on the context.  That web page displayed on a browser is a client per some model.  So is the mobile apps.  A service looking for data from Redis is a client too.

The client can be within the backend system which requests another entity to process its request and awaits for the response.  This client is not always a mobile app, web page on a browser, or a terminal where I'm working with commands.  Do you see that?

Next time when you hear the word client, ask for the context and know who is the client that is being discussed.

Client's Awareness in Performance Tests

By now, you should be breaking your assumption and the myths around the word "client".

In testing for Performance, it is critical to be aware who is client, when and how?  Evaluating this client will not be like evaluating a web page on a browser or the mobile apps.

The fifth question from season two of 100 Days of Skilled Testing, is:

How does client-side performance testing contrast with server-side performance testing, particularly in their objectives and area of emphasis?

Hope now you will question when said client-side performance,

• What client are we talking here?
• Where do this client sit in the system's architecture.

Based on this information,

• How the tests for performance is approached and executed for a client, differs.
• What is collected and observed to evaluate the performance of a client, differs.

This blog post is not to illustrate the different clients and how evaluate it for performance.  When the question talks about a particular client and in a context, I will share the approach, and how to evaluate the same.

To end, have we explored the clientless interface?  How to test this interface?

  

Performance & Tests: Getting Started and Data Analysis

 

On running tests,

• We will have data (information) as one of the byproduct.
• Analyzing the data of the integrated sub-systems in isolation and correlation,
• It will lead us to a technical analysis on each integrated system.

In the report, we draft this analysis along with actions to be taken.

Note: When said sub-systems do not ignore or skip the client or consumer; the system does not comprise just server.

No Golden Rule

There is no one way to do a testing.  Likewise, there is no one way or the golden rule to test for performance.  It is contextual and depends on what I want to learn.

In fact, in few contexts, we can have a value adding performance test with just one request.  Just, I should be well aware of -- what is that I want to know and learn from this test.

That said, there are multiple interfaces where we can observe, analyze and learn from the performance data collected.

The fourth question from season two of 100 Days of Skilled Testing, is:

What are your favorite hacks to analyze performance testing results and find anomalies?

Well, this question do not mention explicitly if it is for server or client or database or caching or messaging or for what interface of a system.  It is a question; but, to me it looks too generic and at a point it looks vague.  Having said this, that is how the learning journey and curve starts! 

Result vs Report

What is a result?

• Is it an evaluation after a data [information] is put to scrutiny?
• Or, the result is a data that is collected and not yet interpreted?

It depends on individual or team and how it being practiced.

The result is different from a report.

Getting Started and Data Analysis

I should know how the system architecture is designed and orchestrated with its boundaries and interfaces.  This helps a lot.  What kind of architecture is this?  Is it a monolith?  If it is monolith, my approach to test for performance differs.

If I'm asked to start the analysis of data for a system that I'm not aware of,

• I will start by analyzing the below indicators on knowing the architecture and the orchestration of the sub-systems for critical business workflows

1. CPU usage
2. RAM usage
3. Data I/O
4. Network usage
5. The Heat and sound dissipated from the hardware which holds and binds
• CPU, RAM, Data I/O, Network and tech stacks installed and configured

It hints me to look further and test investigate, when I observe:

• Having a steady consumption
• What is steady consumption in this context?
• Having a low consumption
• What is low consumption in this context?
• Having a unusual consumption spike and fall of it
• I follow the pattern to study further
• What is considered as knee, spike and fall, in this context?
• Having a zero consumption
• Having a maximum consumption
• What is maximum consumption in this context?

Having a high consumption doesn't mean a problem.  Likewise, having a low consumption does not mean all is well.  I have to uncover them to learn what it means in the given context.

In each of this, there will be a pattern.  I will learn them.  I will correlate with other sub-systems and learn what they were doing in the said timeline.

Do you recollect this line -- "the architecture should provide the Testability"?

• I wrote about it in one of the blog posts of Performance Engineering.

I refer to the below by traversing with the timeline,

• The logs by asking for it
• Data recorded
• Any APMs that are in place

I correlate all these with above said indicators.

This gives me a start. It is one of easiest start that I can have to get started with analysis.

Well this is to analyze at the server end.  What about the client [consumer] end?  It is simpler and will share in the coming blog posts.

Do you want to know more on this and other strategies that can be used contextually?  Let us get connected and converse.  I'm happy to share and learn on listening to you.  It is fun and awareness!

Prioritizing Performance & Its Requirements - The Two Engineering Tasks

  

How do I gather and prioritize the performance requirements of a student from schools, colleges, universities and society? 

Note that, I said performance.  What do performance mean in schools, colleges, universities and society?

• Any time, you asked this question to self?
• If you are living with children, did this question cross your mind, no matter in what class the children are studying?

This is not the question which can be ignored.  Also, this question is not precise and to the context.  It is the question that is resonated but has no acceptable rational base for whatever context from which it arises.  The same when it comes to performance requirements of a software system.

If you observe closely, the system in which we live, it pushes towards performance for what it thinks as a performance.  Isn't it?

The third question in season two of 100 Days of Skilled Testing is:

How do you gather and prioritize performance requirements from stakeholders and project documentation?

Prioritizing Performance & Its Requirements

If you read attentively, the title of this section says - prioritizing performance and its requirements.  I did not say, prioritize performance requirements.

That said, what is performance for Netflix is not same for the  Aadhar system.  But, both have prioritized the performance and looks to be aggressive in knowing the requirements for same.  Don't you think so?

We're in the timeline of Shift Left.

• How to shift with performance to left? 
• What all in the system should focus on the performance in the Shift Left?

MVP & Performance Engineering Story

When we are going to take MVP as early as possible to market, there is a tradeoff.  What are considered in subsequent priorities which will be compromised on negotiation by engineering and business?

The context matters when prioritizing!  Be it for performance or functionality or security or any quality criteria.

I will interpret the question asked from the point view of a MVP's deployment or publishing.

• Are you asking why I have picked MVP?
• The performance is contextual and it is based out on multiple touchpoints, its boundaries and interfaces of a system.
• I cannot talk on all those in this blog post.
• Neither, I want to talk about the KPIs and metrics.  
• I want to share which you can pick, consume, and apply in your work.

Now, we have prioritized performance for a MVP.  Aren't we?  Prioritized means, it matters, it concerns us and we are okay to compromise on few for it.  Let us jump to Left with MVP in our hands to identify the requirements of business.

As a business, we will have a rough idea on how we are pitching and selling our services and to whom.  As a test engineer, you can sense what is the key transaction [business work flow] in the MVP.  You will know the touchpoints, interfaces and boundaries in the architecture that communicates and work together to keep MVP delivering the value.  Don't you?

Say the business wants the MVP to support and serve 500 requests per second.  I should know about the 500 requests here.

Is it,

• Concurrent Requests?
• Active Concurrent Requests?

This matters!  Both are not the same.  Have you asked this question?  It is a requirement we miss to capture.

Capturing Performance Questions for a MVP

It is about the awareness for first!  How much am I opening up myself to the awareness?  This brings an energy and it is contagious.

How do I bring the performance awareness in my team so that it is engineered into the system we develop?  This is a culture drive to an organization.

Now, I know, the MVP has to serve 500 concurrent active users in a second per business's expectation to meet its reach and target.  If I do not know this, I have to capture this data, for first.  How do you capture?

A Use Case to Ponder

One use case which would trigger the spark of thinking is - How should Disney+ Hotstar's services perform to live stream the India vs Pakistan Cricket World Cup 2023 on 14th Oct 2023?

• How should it capture the performance and its requirement for this day?
• How should this system scale to crores of viewers streaming the live video of the match?
• How should this system scale for the gamification - emoji, chat and other viewers engagement during the crores of viewers making requests from client interface?

Try to play the past 30 minutes of this live video? What did you see? Why? That is part of the performance engineering strategy!

This use case opens up the different topics of Architecture and Performance Engineering. Be aware of it and explore on them.  This is not what we want to talk, now.  We want to talk on a MVP and how to capture its requirements for having better performance experience.

Step Up by 5% Heuristic

On having a test environment which is close to the production context and the test data that looks realistic, I get started.

I framed this "Step Up by 5% Heuristic" after few months on starting the practice of Performance Engineering and Testing.  I failed, and I learned. I'm learning.

I know, the expectation is 500 active concurrent users per second.

I will start to evaluate the integrated systems of the MVP for the 5 percent of 500.

• What is the 5% of 500?
• I will start with 25 concurrent active users requests for the MVP.
• I will observe the emotional experience of when using the MVP during this time.
• I will monitor and record the KPIs, and other needed data.
• Does it fail to serve 5% of concurrent active users?
• If failed, I know what to do now.  It helped me.
• This helps me to draw the requirements better and rationally for the existing system's architecture.
• If it succeeds, it helped me partially in knowing what actually I wanted to know.
• I will raise the active concurrent users to 10%
• That is, increasing it by 5%
• I repeat these tests until the MVP architecture lets me know about the requirement it needs for the performance of serving 500 active concurrent users in  a second
• Read the above sentence, again
• The tests on MVP will let me learn what are its performance requirements for serving active concurrent 500 users in a given architecture, infrastructure, and tech stacks

Beyond by 37% Heuristic

I framed this "Beyond by 37% Heuristic" after I failed in framing the tests for performance.  Talking the rationale of this heuristic is not the purpose this blog post.  Let us catch up if you are curious and interested, we will discuss on this.

Do a salary hike of 30% indicate a high performance?  I don't know!  But 30% hike is something not commonly given to all, is what I see in my career so far.

That said, this 37% has worked for me in the contexts I'm testing.  Did it serve 685 (500 + 185) active concurrent users in a second?  It helps me to draw a requirement analysis of the MVP system for this volume of concurrent active users.

Now, I will step up by another 37% of concurrent active users. That is, 870 (685 + 185) active concurrent users in a second.

• If seen, I have 1.5x traffic now.  Did it serve?
• If yes, how many active concurrent users were served in a second?  
• I will correlate the KPIs of other integrated systems of a MVP.
•  With the captured data and emotions
• This will tell, what should we expect despite what is the expectation from business
• This difference will let us know "the requirements"
• How to gather information on -- what has to be optimized, changed, reorchestrated, eliminated, included, and more.
• We start technically in establishing and framing the Performance SLAs and SLOs between the tech team and business.
• Now the performance & its requirements will appear in the dots that are,
• Being connected
• To be connected
• To be disconnected
• That does not exist

To conclude, shift wherever, take the performance engineering together! Revive its requirements to be healthier!

Note: You should read these blog posts if you have not:

1. Performance Testing: Unspoken KPIs and The Missing Correlation
2. Architecture: The Common Shared Understanding -- Part 1
3. Architecture: Its Aid in Performance Engineering -- Part 2