A Test Is Not a Metric

 

A test execution by human or automation will provide information to be aware & learn.

If there is a metric, it should be for what I got to be aware of & learned.  Not for the number of tests.  If it is for a test, it is blunder, before taking it as a metric.

One can identify infinite tests; automate adding annotation @test in big numbers. Should this number be a metric?

That is a question to ask when you see a metric on this number.

What is of value to you in this outcome from a test?  That value should have a metric, and not test or the number of tests.

Identifying the metric in a context which serves, is not easy.

Number of pass/fail or green/red is a measurement; not a metric!

Anything measured cannot be a metric.

A metric helps to measure in a way that establishes rational correlation & upholds its necessity in business.

Share this awareness!

What are KPIs and Metrics?

 

I use to have this question - Are KPIs and Metrics the same?  Especially when I started to learn and practice the Performance Testing, this question bounced back to me often.

Do you have this question?

The Use of KPIs and Metrics

When business and stakeholders talk about it so much, there should be a value out of it.  What is that value?  Why it is important to identify and capture the KPIs and Metrics?

The KPIs and Metrics are derived from data we collect.  These data are processed to extract and normalize, so that, it is in a state as expected by the consumer for making a decision.

The stakeholders will make decisions and take actions referring to KPIs and Metrics. For example,

• The number of Daily Active User (DAU) is a KPI and also a metric.
• But, a metric cannot be a KPI
• How many of this DAU, closed the transaction within five seconds using a wallet?
• It is a metric; not a KPI.

Another example,

• KPI
• How many users installed the latest version of the mobile app and have signed in?
• If there no active users on latest version that indicates a kind of risk and problems to the business.
• Reopened Tickets in Customer Care
• This indicates there is something going wrong
• Metric
• What is the average time taken to see a streaming screen for users in 4G data network?
• If this is not captured, there is no data for business to establish a relationship with KPI set.
• Average Reopened Tickets in a customer service
• The distribution and time towards lower number
• The distribution and time towards higher number

KPIs and Metrics are not the same while both have quantitative measurement. Both are different.  Identifying and knowing the difference between them in your context is important.

They go hand-in-hand when setting a direction and action.  So that, the business and stakeholders realign to the goals and objectives defined.

KPIs vs Metrics

To conclude this post, investigate your metrics and question why the chosen KPIs.  It will help you to design your Test Models and identify the tests in given context.

Performance Testing: Unspoken KPIs and The Missing Correlation

 

I love Performance Engineering!  In a nutshell, the performance is all about capability in a given capacity,  in a context.  The context is critical element in Test Engineering.  In the common language of a workplace context, the performance it is about the productivity expected and delivered.

This blog post is part of 100 Days of Skilled Testing series.  The first question posted for season 2 is,

What key performance metrics or KPIs (Key Performance Indicators) do you consider when defining performance requirements?  Do you use the same metrics for client and server-side performance testing?  If not, these differ in what way?

KPI Classification

On a high level, irrespective of a boundary and interface of a software system, the Performance KPIs can be classified into two:

1. Service Oriented
• It helps to learn by correlating,
• How well a system is providing the service to the intended users in a given context?
• How a system is not providing the service to the intended users in a given context?
• For example, Response Time, Availability, etc.
2. Efficiency Oriented 
• It helps to learn by correlating,
• How well an application uses its features and resources in a given context?
• How an application is having trouble in making use of its features and resources in a given context?
• For example, Utilization of Resources, Throughput with the resources available in the context, etc.

Performance Engineering & Metrics

You will be aware of the other commonly spoken or written metrics that a performance tool offers or has it in its glossary.  I want to focus on KPIs what we are not aware or not exposed to in the communication or Performance Engineering & Testing Report.

The metrics in Performance Engineering & Testing depends on what in the system, I'm putting to an evaluation.  

For example, we do not consider how a feature is implemented and how many threads it can spawn and use in a given point of time.  Further, this leads to CPU and its logical cores, RAM, Disk I/O, and types of server or machine used.  Also, another important data is how the OS is setup with configurations where the different tiers of a system is deployed or installed.  These metrics are common either when I'm facing a client end or serve end, yet it is missed.

If you notice, these are hardly spoken or heard metrics.  But it plays a crucial role.  A performance report compiled should have this data so that the technical people can correlate with other commonly heard metrics.

Being aware of the KPIs that are commonly seen in the Performance Report is a must.  But, knowing what we are not aware or/and missing besides the known is a necessity!  This is missing in the correlation when evaluating the performance's perspective of a system.

The Missing Correlation

Identify what in your report is a need to correlate.  The statistics or numbers are representation and not a derivation.  It is of no use until I derive out of it with a correlation.

I have to derive the correlation by including and also by eliminating the representations.  For doing so, the missing correlation representation is a must so that KPIs look rational, technical, logical and analytical.

Uncover what is being missed in your correlation so the KPIs representation look senseful!