Performance Test Report: Between the Effective and Ineffective Reports

 

In this post, I'm picking the thirteenth question from season two of 100 Days of Skilled Testing.

What is an effective way of reporting performance test results and mention some tools you have used in test execution, analysis, and reporting?

I see two questions.  I see it is not a wise attempt to learn these two questions combined as one.  In my opinion, the second question added, it dilutes and make the whole question vague.

What Should a Report Do?

The report should be contextual, compelling, influencing, and targeted to the intended audience to act upon on making a decision.  The software testing report is not an exemption from it.

The performance testing report should know

• Who are its audiences?
• How they read, relate and understand the information?

If this is ignored, the report will not serve the purpose of commissioned testing.  The effectiveness of a report cannot be determined solely on how the stakeholders responds to it.  

On understanding the risks and problems in the system's current capability, mentioned in the report, the stakeholder might not respond with an action to tune the performance aspects.  This could be for multiple factors including that of business.

Note that, a skilled and problem solving engineer understands the business and how it drives.  Just being technically skilled will not help an engineer to grow in a longer run in her or his career.  The system's performance tuning decisions most times will be driven by business.

Did the report persuade the stakeholders with an awareness, mutual understanding and agreement?  The report should drive this conversation.  If not, we have a problem.

On reading the performance testing report, do the stakeholders get an informed awareness on what happened during testing in the present capability of a system's performance criteria?  Do the stakeholders understand and mutually acknowledge how it benefits and costs the business?

This is the foremost value serving expected from a testing report.  If not, I look at how the data and story is presented in the report.

The bottom line is, did we mutually acknowledge, agree and understand on current capability and consequences?  If not, the basic purpose of the report is not met.

Performance Testing Report

The software testing is a high technical activity.   You agree or not to this, but, this is the reality.

Testing for performance is technical investigation activity. It includes the orchestrated study in correlation of 

• hardware, operating system, network, tech stacks & software used in SDLC, architecture, designs, certain decisions, people, business and you - the test engineer.

The fundamental in-depth awareness and knowledge of these areas is essential and a necessity to analyze the performance's aspect.  The performance testing report will show this trait of you as a test engineer.

We have stakeholders who work in technical area and in non-tech area.  How to compile the effective performance testing report?

There is no one way or defined way of writing an effective performance testing report.  Figure out what works in context of your testing to have a effective report knowing - What Should a Report Do?

Outline of Persuading Performance Test Report

It is a technical story telling in a non-technical way with data, pictures, comparison by relating, metaphors, and contemporary history.  I compose the performance testing reports in line with business targets and objectives set.  I provide a metaphors to relate and know the value and cost.

At times, I will have two reports.  I share it with respective stakeholders.

• One with non-technical summary and conclusion
• The other with technical details, analysis and data from investigation

Sometimes, I include the above two reports in one report based on the context.

In overall, this will be in minimum as part of my performance testing report to start.

1. What part of the system is being tested?
2. Why that part of the system is being tested?
3. Mentioning the vague performance requirements gathered from stakeholders.
4. Refining and precising the performance requirements to be specific, contextual and deterministic.
5. Who are the stakeholders of this report?
• What sections the respective stakeholders to refer for the analysis and outcome?
6. Problem statement of the performance testing statement
7. Brief summary of performance testing outcome. [TLDR]
1. What aspect of system's performance is evaluated and why?
2. Brief summary of performance test carried out and outcome.
8. Detailed Report with Technical Details
1. Analysis and Technical Investigation
2. Representation of data which is analyzed
3. Identification of bottlenecks, risks, problems and its symptoms
4. Summary of the test's outcome

You don't have to stick on to one format or a template.  Figure out what works well in your case so that the intent of your tests and outcome is understood by stakeholders.  Give a structure to your report!

The performance testing reports will have metrics, graphs, numbers and proposals.  The presence of metrics, graphs, numbers and the other said, does not make the report effective.  Then, what makes it effective?  When you call it effective?  When you accept it is not effective?  Only, you can figure it out to your context.  I can assist you here; pull me in.

There is no good [effective] report or not good [ineffective] report.  The report is either

• From a team with skills, experience, trained, and practicing
• From a team which is not trained, and, not practicing