Performance Testing: Unspoken KPIs and The Missing Correlation

 

I love Performance Engineering!  In a nutshell, the performance is all about capability in a given capacity,  in a context.  The context is critical element in Test Engineering.  In the common language of a workplace context, the performance it is about the productivity expected and delivered.

This blog post is part of 100 Days of Skilled Testing series.  The first question posted for season 2 is,

What key performance metrics or KPIs (Key Performance Indicators) do you consider when defining performance requirements?  Do you use the same metrics for client and server-side performance testing?  If not, these differ in what way?

KPI Classification

On a high level, irrespective of a boundary and interface of a software system, the Performance KPIs can be classified into two:

1. Service Oriented
• It helps to learn by correlating,
• How well a system is providing the service to the intended users in a given context?
• How a system is not providing the service to the intended users in a given context?
• For example, Response Time, Availability, etc.
2. Efficiency Oriented 
• It helps to learn by correlating,
• How well an application uses its features and resources in a given context?
• How an application is having trouble in making use of its features and resources in a given context?
• For example, Utilization of Resources, Throughput with the resources available in the context, etc.

Performance Engineering & Metrics

You will be aware of the other commonly spoken or written metrics that a performance tool offers or has it in its glossary.  I want to focus on KPIs what we are not aware or not exposed to in the communication or Performance Engineering & Testing Report.

The metrics in Performance Engineering & Testing depends on what in the system, I'm putting to an evaluation.  

For example, we do not consider how a feature is implemented and how many threads it can spawn and use in a given point of time.  Further, this leads to CPU and its logical cores, RAM, Disk I/O, and types of server or machine used.  Also, another important data is how the OS is setup with configurations where the different tiers of a system is deployed or installed.  These metrics are common either when I'm facing a client end or serve end, yet it is missed.

If you notice, these are hardly spoken or heard metrics.  But it plays a crucial role.  A performance report compiled should have this data so that the technical people can correlate with other commonly heard metrics.

Being aware of the KPIs that are commonly seen in the Performance Report is a must.  But, knowing what we are not aware or/and missing besides the known is a necessity!  This is missing in the correlation when evaluating the performance's perspective of a system.

The Missing Correlation

Identify what in your report is a need to correlate.  The statistics or numbers are representation and not a derivation.  It is of no use until I derive out of it with a correlation.

I have to derive the correlation by including and also by eliminating the representations.  For doing so, the missing correlation representation is a must so that KPIs look rational, technical, logical and analytical.

Uncover what is being missed in your correlation so the KPIs representation look senseful!