Sunday, March 3, 2024

Performance Test Report: Between the Effective and Ineffective Reports

 

In this post, I'm picking the thirteenth question from season two of 100 Days of Skilled Testing.

What is an effective way of reporting performance test results and mention some tools you have used in test execution, analysis, and reporting?

I see two questions.  I see it is not a wise attempt to learn these two questions combined as one.  In my opinion, the second question added, it dilutes and make the whole question vague.


What Should a Report Do?

The report should be contextual, compelling, influencing, and targeted to the intended audience to act upon on making a decision.  The software testing report is not an exemption from it.

The performance testing report should know
  • Who are its audiences?
  • How they read, relate and understand the information?
If this is ignored, the report will not serve the purpose of commissioned testing.  The effectiveness of a report cannot be determined solely on how the stakeholders responds to it.  

On understanding the risks and problems in the system's current capability, mentioned in the report, the stakeholder might not respond with an action to tune the performance aspects.  This could be for multiple factors including that of business.

Note that, a skilled and problem solving engineer understands the business and how it drives.  Just being technically skilled will not help an engineer to grow in a longer run in her or his career.  The system's performance tuning decisions most times will be driven by business.

Did the report persuade the stakeholders with an awareness, mutual understanding and agreement?  The report should drive this conversation.  If not, we have a problem.

On reading the performance testing report, do the stakeholders get an informed awareness on what happened during testing in the present capability of a system's performance criteria?  Do the stakeholders understand and mutually acknowledge how it benefits and costs the business?

This is the foremost value serving expected from a testing report.  If not, I look at how the data and story is presented in the report.

The bottom line is, did we mutually acknowledge, agree and understand on current capability and consequences?  If not, the basic purpose of the report is not met.


Performance Testing Report

The software testing is a high technical activity.   You agree or not to this, but, this is the reality.

Testing for performance is technical investigation activity. It includes the orchestrated study in correlation of 
  • hardware, operating system, network, tech stacks & software used in SDLC, architecture, designs, certain decisions, people, business and you - the test engineer.

The fundamental in-depth awareness and knowledge of these areas is essential and a necessity to analyze the performance's aspect.  The performance testing report will show this trait of you as a test engineer.

We have stakeholders who work in technical area and in non-tech area.  How to compile the effective performance testing report?

There is no one way or defined way of writing an effective performance testing report.  Figure out what works in context of your testing to have a effective report knowing - What Should a Report Do?


Outline of Persuading Performance Test Report


It is a technical story telling in a non-technical way with data, pictures, comparison by relating, metaphors, and contemporary history.  I compose the performance testing reports in line with business targets and objectives set.  I provide a metaphors to relate and know the value and cost.

At times, I will have two reports.  I share it with respective stakeholders.
  • One with non-technical summary and conclusion
  • The other with technical details, analysis and data from investigation
Sometimes, I include the above two reports in one report based on the context.

In overall, this will be in minimum as part of my performance testing report to start.
  1. What part of the system is being tested?
  2. Why that part of the system is being tested?
  3. Mentioning the vague performance requirements gathered from stakeholders.
  4. Refining and precising the performance requirements to be specific, contextual and deterministic.
  5. Who are the stakeholders of this report?
    • What sections the respective stakeholders to refer for the analysis and outcome?
  6. Problem statement of the performance testing statement
  7. Brief summary of performance testing outcome. [TLDR]
    1. What aspect of system's performance is evaluated and why?
    2. Brief summary of performance test carried out and outcome.
  8. Detailed Report with Technical Details
    1. Analysis and Technical Investigation
    2. Representation of data which is analyzed
    3. Identification of bottlenecks, risks, problems and its symptoms
    4. Summary of the test's outcome

You don't have to stick on to one format or a template.  Figure out what works well in your case so that the intent of your tests and outcome is understood by stakeholders.  Give a structure to your report!

The performance testing reports will have metrics, graphs, numbers and proposals.  The presence of metrics, graphs, numbers and the other said, does not make the report effective.  Then, what makes it effective?  When you call it effective?  When you accept it is not effective?  Only, you can figure it out to your context.  I can assist you here; pull me in.

There is no good [effective] report or not good [ineffective] report.  The report is either
  • From a team with skills, experience, trained, and practicing
  • From a team which is not trained, and, not practicing


Sunday, February 25, 2024

Backtracking of Testing, Security and Tools

 

When I started my software testing career in 2006, I was in this thought -- What tools should I use, so that,

  • I can do the testing that is sought after
  • I can test for performance
  • I can test for security

Moving from a search for tools to building the mindset and attitude.  It is a journey!  It took me time to see this journey.  I hopped on to this journey in 2011.  I see, this is not an ending journey, while I know where should I go and reach.  I'm on this journey.

I had no mentors.  I had no seniors in software testing to guide and discuss on my thought process.  I had developers (programmers) who had little or no interest in testing; so it did not matter to them.  But, they have helped me to be better tester.  I'm grateful to them.  Then, the community was not so connected, organized and share the knowledge as it does in 2024.  The software testing was not considered or seen as a technical activity, then.  I have stood, fought, demonstrated and delivered my testing as a technical activity.  I'm continuing it.


Today, on 24th Feb 2024, I read the below question in a community's social space and decided to write this blog post.
Hey, everyone .... Can anyone please suggest a good tool for API security testing?

This question resonates in test engineers.  Most of we test engineers still look and ask for tools when it comes to security testing.  To test engineers, the performance and security testing are still a conception and activity with tools alone.  In reality, it is not!  If you are in such thought or you come across such question to answer, this blog post is for you.


Backtracking the Problem Identification

In programming, we have an approach by name Backtracking.  It is about exploring in possible ways to find possible solutions for a problem.  And, a best solution which works in context is picked.

What's the problem here?  Testing, Security and Tools. Are you with me so far? Let us backtrack this problem.

NoteI see a difference between the words 'possible' and 'all'.  Hence, I use the words "possible ways" and "possible solutions" and not "all ways and all solutions".


Bounties and Entry

There are reputed bug bounties for security testing.  To get into this bounties one has to showcase her/his discoveries and skills with her/his recognized portfolio.

The tools are accessible to all.  The community edition and licensed edition tools are available.  We use these both editions of tools.

  • But, why not all of us with tools cannot get into such invited security bug bounties?  
    • You will answer this question if you ask yourself.  Hope this backtracking should have helped by now!

The Security Engineering is a vast practice area in Software Engineering. There are dedicated security engineers in role.  But, we test engineers can take up the testing for the security of software systems which the team is programming and building.

I advise, a practicing test engineer
  • To start with building an interest for security engineering.
  • Consistently hone and build the mindset, attitude and skills needed for the testing the security aspects.
  • Pick simple problems, solve it.  Do it consistently, while you explore the layers.

While this is done consistently, it is time to find the mentors in Security Testing. The mentors will assist you in practicing how to test effectively for security making use of simple contextual necessary tools.  Also, a mentor will let you know how to test for security without tools to an extent.  The tool is effective when known how to use it.  The tools help immensely only if I can test for security. 

To backtrack in a different perspective, did any tool that you use, find a P1 security problem [or risk] by itself in its scan?  Did your programmers acknowledge to that risk or problem?  I will pause with these two question to you.



Today, my testing for security is confined to systems that I test.  I test for web application, mobile apps, web APIs, and database.  I can assist here, if you do the home work and ping me.