The "Bottleneck" in a Test Engineer's Eyes

 

Preference to Bottle Over Jar! Why?

Have you heard Jar Neck anytime when describing a problem or solution?

• I have heard Bottleneck often and consistently; but, not Jar Neck .  Why? 
• Be it in Software Engineering or day-to-day life problem solving description,
• The Bottleneck is referenced and not a Jar Neck.

Looks like people want Bottle but not the Bottleneck speed and benefits.  Bottle without its neck is a jar?!

Bottleneck exist for better controllability.

• In a bottle, the bottleneck is a solution!  It is not a problem!
• It is to mitigate any risk and problem that arises from the flow of content in the bottle.

Yet we describe, learn and communicate the neck of a bottle as a relativity and analogy to a problem.  

Are you aware of Gateway in the software system?

• The Gateway can be seen as a neck of a bottle which controls the incoming requests and outgoing response.
• Gateway is a necessity.
• We need Gateway to be adaptable in size of its neck based on traffic volume it is handling.  Here, the gateway's neck size should adapt and scale contextually.
• When describing a problem, we are talking about how this bottleneck size which is not adaptable for the context.
• That adaptability has to be built in engineering to scale in any dimensions and magnitude.
• When this is not done, we equate the software system's problem to a bottleneck as a analogy, which is incorrect!  The bottle has got its size and its neck size fixed for a purpose and as a solution.
• The context of a bottle and today's any systems are different.
• It is good to draw similarities from General Systems Thinking and observations.
• But the solution cannot be generic to all systems; it has to be contextual.  The software system has to have its contextual solution.

So, next time when someone in your team or network talk about bottleneck, do share them bottleneck is for better controllability.  Having a contextually resizable and adaptable bottleneck is the need for Software Engineering; not the elimination of bottleneck.

In fact, a software system should have and will have a bottleneck in a point.  And, this bottleneck will be adaptable to the context for having what it should let through and process.

Is the runway of an airport a bottleneck when it is compared to a sky?  Is that a solution or a problem?  Likewise, the ship will have a defined route path and it does not sail without a route path.  Is this a bottleneck to ship and its business?  A elevator can accommodate the defined number of people or kilograms allowed, and not beyond that to move.  Is that a bottleneck?  The esophagus in human body has a size which medical science observes as normal and acceptable; any deviation from that size measurement, the medical science test investigates it as a risk and problem. Why?  Is the circumference size and length of esophagus a bottleneck to human anatomy and physiology system?

The engineering solution will and should have a bottleneck at a point.  Having a adaptable bottleneck to the context is one what tries to accomplish in a software system's scalability and operability.

Please, do not equate solving a "bottleneck" situation with Agile practice.  Does it look like a joke?  I will not be surprised if someone says bottleneck problem is solved if practiced Agile.

Software Engineering - The Unquestioned Understanding of Client in Testing

Client - The Unquestioned Understanding

When asked or said "client" or "client-side", most of us assume or take it as:

• Web page displayed in the browser
• Mobile apps - Android and iOS apps
• Desktop applications

I see this is one of the unquestioned understanding and assumption in the Software Engineering.  While it is not wrong, the client does not always mean -- a web page displayed on browser, mobile apps, desktop applications, a terminal, etc.

The client is one of the subjects which we have not attempted enough to understand in Software Testing & Engineering.

A client is one who consumes the service in form of a response, and then does what it has to do.

Client - The Contextual Entity

The entity which takes the a client place [role] is entirely based on the context.  That web page displayed on a browser is a client per some model.  So is the mobile apps.  A service looking for data from Redis is a client too.

The client can be within the backend system which requests another entity to process its request and awaits for the response.  This client is not always a mobile app, web page on a browser, or a terminal where I'm working with commands.  Do you see that?

Next time when you hear the word client, ask for the context and know who is the client that is being discussed.

Client's Awareness in Performance Tests

By now, you should be breaking your assumption and the myths around the word "client".

In testing for Performance, it is critical to be aware who is client, when and how?  Evaluating this client will not be like evaluating a web page on a browser or the mobile apps.

The fifth question from season two of 100 Days of Skilled Testing, is:

How does client-side performance testing contrast with server-side performance testing, particularly in their objectives and area of emphasis?

Hope now you will question when said client-side performance,

• What client are we talking here?
• Where do this client sit in the system's architecture.

Based on this information,

• How the tests for performance is approached and executed for a client, differs.
• What is collected and observed to evaluate the performance of a client, differs.

This blog post is not to illustrate the different clients and how evaluate it for performance.  When the question talks about a particular client and in a context, I will share the approach, and how to evaluate the same.

To end, have we explored the clientless interface?  How to test this interface?

  

Architecture: The Common Shared Understanding -- Part 1

 

When we are developing a software system, the requirement from a stakeholder is not 'Fast' or 'Scalable'  or  'Responsive'.  But, the stakeholder needs it and expects it.  If you see, on a larger picture, the software system development and maintenance is a job of balancing too.

When a Software Architect [Technical Architect and Test Architect], works on architecting the software system and testing for the same,

• It is about balancing the technical aspects with the business's requirements from stakeholders.  
• Do you see that?

Knowing the architecture of a software system and testing of same is one of a primary task for engineers on the project.  Because, we software engineers have to balance it well.  Balance, what? Balancing the technical aspects together with business's requirements from stakeholders.

This blog post is part of 100 Days of Skilled Testing series.  The second question posted for season 2 is,

How important is the understanding of application architecture to do performance testing better?

 

What is an Architecture?

In context of Software Development & Engineering, the word "architecture" is one of the ambiguous words among the teams in a project and an organization. 

As a test engineer,

• Did you ever had a discussion or arguments or debate with programmer and architect?
• I had such discussions and I continue to have it today as well in the projects that I work.
• This is to know and understand
• What I should be doing as an engineer for first and as a Test Architect in the role?

The outcome of this discussion showed me,

• We all did not have a common understanding of it
• We did not share,
• "What I understood for the architecture and this architecture?"

The primary goal of a Software System's Architecture is,

• We all engineers on a project have a same understanding of it, in the aspects it exists for.
• This understanding is arrived after we have put our thoughts into scrutiny and decided that we stick on to it, so that,
• We can balance well between the technical requirements and business expectation.

Are you with me, so far?

A Software System's Architecture is,

1. A common shared understanding of what we all have for,
•  What we are developing, testing, and to about maintain?
• And, Why? Who? When? Where? How?
2. Represents the boundaries and interfaces of what matters,
• That is [to be] orchestrated, designed, implemented, how it communicates, and what it will have, and not.
• It also can show how the teams are structured and how the team and organization is organized.
• For example, in the Service Oriented Architecture, the teams are built and structured with respect to the service they offer.
• It is a model that is better than other models in a given context of technical requirements and business needs
3. The context and awareness for,
• Why it is the way it is?
• The cost and value for being so.
• What to do when it has to be changed? Where to change?
• How simple and quick to change?
• What are the cost and value for being so?
• How can I monitor and observe all these consistently?
4. The Gateway of Testability - it tells what is the Testability available for,
• Letting know what is critical and priority to test
• Where, How, When, and What tests can be framed, designed, and executed? To what extent?
• Why these tests?
• If an architecture does not talk about and do not have the Testability, we have a serious problem!  This has to be fixed for first on priority.
• An architecture has to provide the Testability and Programmability scope and opportunities to develop a software system that is of value!

For today, this is my understanding for the "Architecture".

I'm a Test Architect in the role and I expect myself to be an hands-on engineer for first.  It is a necessity for an architect to be an hands-on engineer.

Note: Read the Part 2 of this blog post here.

Architecture: Its Aid in Performance Engineering -- Part 2

 

I hope, you and I have the common shared understanding for the word "architecture".  If not read this blog post and come here to know about the dots.

Do I Know the Dots?

Before connecting the dots, I should know,

• What are the dots and how to identify them, where and when? 
• Who can help me in doing so?

In Software Security & Engineering, we use a Threat Model to,

• Identify the risks, surface area, tests and to develop the payloads. 
• A software system's architecture will help in developing and improvising the Threat Model consistently.

I see, the same for software system's Performance Engineering & Testing.  To test better for the performance,

• I need to identify the dots, risks, surface area, tests, payloads, monitoring aid and correlation of all these.
• The understanding of software system's Architecture is a necessity to do so.
• But, what are the dots here?

The dots can be identified when I know how to use the Testability provided by the architecture.  This leads me to evaluate the performance for the boundaries and interfaces in isolation and as a whole, and then correlate.

With this, it puts me to question - What is the performance of this architecture?

• I did not say the performance of software system; I said, the architecture.  
• There should be some characteristics to identify and evaluate the performance engineering models offered by the architecture. 
• What are they?  

The architecture's characteristics helps,

• To identify and distinguish the dots in ease and to test better.
• How can I test for the performance aspects of a software system using the architecture's characteristics?
• How do I identify these characteristics in the architecture of software system?

The Characteristics of an Architecture

Last year I read an article from ByteByteGo System Alliance.  This article flashed in my mind as I read the question,

How important is the understanding of application architecture to do performance testing better?

This is one of the article which I refer to identify the performance characteristics of the architecture.  I refer to the cheatsheet shared in this article for my references.

In few projects and organizations that I have worked, most of these characteristics were put into practice and production environment.  I monitored them in usual traffic and unexpected traffic.  The feel is something that I cannot describe in words; I want to experience it.

Performance Engineering Aided by Architecture

Which characteristics of architecture is associated to the which boundaries and interfaces of a software system?  Knowing this, helps you and me in thinking - What has to be tested in performance for this interface in this boundary?

I want to share my work experience here.  But, I see, if I share something which we all can relate to, it will be of help in knowing - Why it is important to know about the architecture to do the performance testing?

Below one is a recent use cases from software industry for the same.

• I will not explain in detail; but, I will bring the key points to the context of this blog post

Amazon Prime's Audio-Video Monitoring Service Moving to Monolith Architecture

What I and You Should Know:

• The complete Amazon Prime system did not move to Monolith.
• The Prime Video's Audio-Video Monitoring Service moved to Monolith.
• Why?
• This monitoring system which was orchestrated with a Microservice architecture did not scale after a limit
• Problem Statement:
• Say, the Prime Audio Video Monitoring Service expected a load of active 100 concurrent users streaming movie Kantara in Kannada audio.
• After the 6th user started streaming the same video [in the same audio or different audio language], this monitoring system did not scale to include other 95% of concurrent user. 
• As a result, the Prime Audio Video Monitoring Service slows down [or stops,] and eventually the video streaming to the active concurrent users will take a hit. 
• This monitoring service is important so that each user gets a video and audio of the agreed upon quality and streaming.

What I understand is,

• This monitoring service was continued in production while the team looked for better solution in performance with the given architecture.
• While it did so, the cost of having this architecture was high when it had to scale up.
• Looks like Prime Video business beared this cost for sometime is what I see.

But, the online streaming business cannot settle and agree to pay high cost, while it is planning to stream the live sports action in coming days.  A need came to look into performance characteristics in the being used monitoring service's architecture.

It eventually re-orchestrated the existing components with a new architecture in place.

• It moved from the distributed microservice system to a monolith system, where the spawning of Amazon Step Function (error detector clusters) happened vertically.
• Along with this, the architecture of this monitoring service was placed in a way, such that, most of its components came into one process.
• Thus, eliminating the S3 bucket as immediate storage for video frames (as images) and audio files.
• This architecture helped the creational, behavioral, structural and functional characteristics of Prime Video's Audio-Video Monitoring service.

Prime Video says upon testing for performance and changing to monolith by rearranging the existing components,

• It saved 90% of the cost.
• 90% of cost for Amazon, is what in the numbers if it is in Indian Rupees? 
• How much a tester gets paid if just 1.5% of this 90% is paid as a salary per month?
• I will leave this to your thoughts and calculation.

If I know the architecture and where to look for what characteristics, it helps me to think of right performance tests for the context.  

The Hostar's emoji introduction in live cricket matches during 2018 and its consistent improvisation in processing for performance is a good use case, to the question -- Why it is important to know about the architecture to do the performance testing?

To conclude, architecture cannot be ignored in Testing.  It plays a critical role for aiding and identifying the testability and BCFS (behavior, creational, functional and structural) characteristics of a software system.

Performance Testing: Unspoken KPIs and The Missing Correlation

 

I love Performance Engineering!  In a nutshell, the performance is all about capability in a given capacity,  in a context.  The context is critical element in Test Engineering.  In the common language of a workplace context, the performance it is about the productivity expected and delivered.

This blog post is part of 100 Days of Skilled Testing series.  The first question posted for season 2 is,

What key performance metrics or KPIs (Key Performance Indicators) do you consider when defining performance requirements?  Do you use the same metrics for client and server-side performance testing?  If not, these differ in what way?

KPI Classification

On a high level, irrespective of a boundary and interface of a software system, the Performance KPIs can be classified into two:

1. Service Oriented
• It helps to learn by correlating,
• How well a system is providing the service to the intended users in a given context?
• How a system is not providing the service to the intended users in a given context?
• For example, Response Time, Availability, etc.
2. Efficiency Oriented 
• It helps to learn by correlating,
• How well an application uses its features and resources in a given context?
• How an application is having trouble in making use of its features and resources in a given context?
• For example, Utilization of Resources, Throughput with the resources available in the context, etc.

Performance Engineering & Metrics

You will be aware of the other commonly spoken or written metrics that a performance tool offers or has it in its glossary.  I want to focus on KPIs what we are not aware or not exposed to in the communication or Performance Engineering & Testing Report.

The metrics in Performance Engineering & Testing depends on what in the system, I'm putting to an evaluation.  

For example, we do not consider how a feature is implemented and how many threads it can spawn and use in a given point of time.  Further, this leads to CPU and its logical cores, RAM, Disk I/O, and types of server or machine used.  Also, another important data is how the OS is setup with configurations where the different tiers of a system is deployed or installed.  These metrics are common either when I'm facing a client end or serve end, yet it is missed.

If you notice, these are hardly spoken or heard metrics.  But it plays a crucial role.  A performance report compiled should have this data so that the technical people can correlate with other commonly heard metrics.

Being aware of the KPIs that are commonly seen in the Performance Report is a must.  But, knowing what we are not aware or/and missing besides the known is a necessity!  This is missing in the correlation when evaluating the performance's perspective of a system.

The Missing Correlation

Identify what in your report is a need to correlate.  The statistics or numbers are representation and not a derivation.  It is of no use until I derive out of it with a correlation.

I have to derive the correlation by including and also by eliminating the representations.  For doing so, the missing correlation representation is a must so that KPIs look rational, technical, logical and analytical.

Uncover what is being missed in your correlation so the KPIs representation look senseful!