The { } and Lessons I Observed

 

I did not know { } are called braces.  I called them curly bracket or flower bracket in my verbal communication until I started my career.

In the early days of my engineering college, I wondered why the braces are used and not the square bracket or parenthesis to open and close the class, function and a block of code.  

I was practicing C, C++ and COBOL then.  Though COBOL did not use braces, it used indent to start from specific column in a row.   But, the philosophy of a programming language construct looked relevant in COBOL as in C and C++. 

When I practiced Core Java, I could relate the programming constructs and find the similarities in C, C++, COBOL and Java.  So with C#, JS, and Shell Scripting.

All this said, the question that remained in me is -- Why these programming languages use braces to open and close the class, function, loop & conditional blocks?  Why this similarity?

When I studied Discrete Mathematical Structures in a semester, I could correlate the programming to it.  This helped me to reason and interpret why the braces in a programming language.   

Pic: Programming languages having braces to open a function.

{ } and Set

In the Discrete Mathematical Structures, I studied Sets. Further, I learned how the Sets is used to classify and organize the elements in problem solving.

While I was solving a discrete math exercise,  this struck to me -- Why the braces to represent a SET?  This triggered my associative memory and sparked a question -- Why do I see the braces in C, C++, and C#?

Here is how I interpreted and reasoned my thoughts and questions.

• The SET is to represent the elements of common nature and characteristic which has a purpose. Most of the times it is deterministic.
• But, why the different programming languages use this braces to group a set of instructions? Why not something else to represent programming instructions with a common and deterministic purpose?  
• The COBOL do not use braces, but, the philosophy and thought process of braces exist in it as starting from specific column.
• I observed Java and Shell scripting using the braces.

The point and thought to note here is, the two words -- SETS and Groups. I'm studying this since my primary school days.  But, I had not realized how they are so much into the computer science!

Today, I observe most programming languages use the braces to group the set of instructions which are for a specific purpose.  Be it a class, function, loop or a conditional block, they are within { }.  

Between the { }

The instructions written between the braces has an objective.  Be it class, function, condition or loop block.  That is, I group this instruction as a set.

Why I call it as a set?

• The instructions between the braces have the one purpose.
• Is this SRP -- Single Responsibility Principle?  I am yet to put this in a better word!

Is the set a collection of elements that has a common characteristic and purpose?  Then, between the braces, the code has a common [one deterministic] purpose.  

Does this thought of programming language creators made to use the braces?  If so, this is a beautiful learning which is missed to observe and interpret.

{ } and Programming Languages

A programming language is created [written] to overcome the limitations of other language in a context.

Yet, each person who created the respective languages did agree to a common representation and understanding of semantics and constructs like the braces.  I see this is missing in Software Testing fraternity.  

Though, the creators of programming languages have a differing thoughts on other programming languages and what it takes, they have something in common to represent their thoughts.  

Looks like the most creators seem to go with this common representation.  The braces is one such example.  

Their vocabulary is something amazes me; most don't seem to have a problem with it.  The idea and thought are the same behind the vocabulary in most cases.

Another example is how to write a comment.  The programming and scripting languages that I have used so far uses // to comment a line.

To end this post here, personally I feel, I as a Software Test Engineer have a lot to learn from the programming fraternity.  

The next blog post is continuation of the thought shared here and sharing my experience of how the software testing fraternity looks to me in comparison to programming fraternity.

Testing and Programming Are Not Separate Disciplines

 

Do you and your team see Programming and Software Testing as two separate disciplines?  You believe that programming and testing are as different as oil and water?   If so, this blog post will converse with you to realign your thoughts.

When I started to draw the correlation between Software Testing and Programming,

• That is when I realized Programming is beyond writing the code.
• Testing is an integral part of programming.
• Writing code is one part of programming.  
• Likewise, testing the written code and the associated system, is part of programming.  
• I see, the idea of TDD is an example to it.
• Testing is not the other side of a coin.
• Programming and Testing are on the same side of a coin!  Testing helped me to learn this.

Then, what is on the other side of the coin? 

• It is we on the other side of the coin, 
• Who are seeing and advocating testing and programming are the two different disciplines.

Had I understood this early in my career, it would have made a significant difference.

For the last 10 years, I have been assisting myself and mentees to experience this perspective of programming and testing.

The sad part is, the colleges and universities do not tell this to their students.  Instead, we are said the programming and testing are two different stuffs in software system and engineering.  Thereby, treating the one is superior to other.  Eventually, we in the industry continue to share the same thought and dialogues in all spaces.  Thereby, creating the gap in skills and thoughts of both programmer and tester.

Programming 

What is Programming?

A program is planned and an ordered event of actions with a purpose.  When it is ordered in a specific way, it involves logical decision of a programmer to design and implement a program in a intended way.  

In Computer Science, I accomplish this with the help of programming languages and commands which an operating system leverages and executes.

Programming is a thought process and thinking activity which evolves over the time with discipline and consistent practice.

• It involves thinking, understanding, questioning, and analyzing the intent of learning and solving a problem in hand.
• It demands questioning and learning the context.
• It involves studying of -- systems, people, environment, communication, programming languages, design, modeling, simulation, engineering, technologies, anthropology, geography, economy, business, politics, laws, regulations, computer software, software tech stacks, hardware, domains, different professions, jobs, emotions, crime, psychology, science, arts, and you can add more to this list as you unlearn and learn.
• It demands analytical thinking, critical thinking, and ability to develop clarity for making decisions to design the implementation and code.

Programming is like a manifest file in Android app, where it manifests and expects how and what the app should be like.   The app code implements the manifestation.

Coding

Visualizing and writing the programming instructions from the manifestation thought of programming is coding.  The programming instructions [logical] are written using programming languages.  I understand it, this way.

To code, one uses programming languages.  Further one has to understand the Data Structures and Computer Hardware and associated software to code efficiently and effectively.  This is where I understood the need and value of the Data Structures.  This forms the basic and fundamental necessity for one who wants to build and test the software system.

Each programming language has its own,

• Syntax
• Rules
• Limitations
• Way of execution
• Space
• Time
• Storing and retrieval of data

The below said will be well evaluated for the context,

• Which programming language to code? Why?
• What Data Structures to use for processing the data?

That said, the idea of debugging remains relatable and same in most cases irrespective of the programming language.  

That is, the debugging is at meta layer of programming.  Based on the context, how the debugging is done, can change with the programming language.  But, the idea of debugging remains the same.

Programmers learn and practices different programming languages.

Programmers solve the problem using programming language.  To solve, programmers start visualizing with programming, and then code.

I see, the programmer learn -- Is the actual problem being solved?  So that the time and efforts are well spent.

To stop here, there is,

• Clean Code, and not Clean Programming
• Code Smell, and not Programming Smell
• Code Refactoring, and not Program Refactoring

Programming and Coding are two different skills and it runs in parallel helping each other.

It is like, the programming is a blue print, and, coding is the construction work.  Both are needed!

Testing

Having one's own definition as a practitioner is good.  But, then I see, my definition also aligns with that of Dr. Cem Kaner, James Bach and Ashok Thiruvengadam.  So, I quote their thoughts here.

Testing is,

• Questioning a product to evaluate it.
• Probing the system to learn and evaluate it.
• Empirical technical Investigation
• To learn quality related information of a software system
• So that the stakeholders can make informed decision

I see, testing is an another stand to the programming.  It is not different.  

Programming and Testing are analyzing and thinking skills for first.  The thinking applied and implemented and evaluated is coding skill.

What all needed for better programming, all those are needed for value adding testing.  Refer to the above section on Programming.

Further, testing uses coding to execute the intent of a test to some extent.  And, this is a need.  Being skilled in programming and coding gives a different perspective and strength to the testing.  The value out of this will be distinct and prominent in most context.

If one can test, one can program too.  

Now, do you see that the programming and testing are not the two disciplines?  They are one!  They are on the same side of the coin.  On the other side of a coin, it is we who are not willing to see this and advocating these two are different.  

I learn, a skilled tester can add keep adding ways to sample, interpret and test when she or he can use the coding skills.

To summarize,

1. Programming and Testing are the same discipline.
• Seeing it as two different discipline is counterproductive!
• This counterproductive can be seen in the software industry, today.
2. It is useful thought process, mindset and attitude to see both programming and testing as a overlapping areas and one discipline.
• Both are on the same side of the coin in Computer Science, Software & Engineering
3. Coding skill does not mean one has programming skill.
4. Coding skill does not mean one has testing skill.
5. Programming skill shows the testing skill and vice versa.
6. Programming is a way of testing.
7. Programming is a way of debugging.
8. Testing is a way of debugging and refactoring the programming.
9. Programmer develops the tests and code.
10. If you can program, you can test.  If you can test, you can program.  If you can program, you can code.
• But, coding skill alone is not enough to program and test.
• These all are three distinct skills, where programming and testing overlaps very much on each other.

My Advise to a Software Test Engineer

I was discussing with a Software Test Engineer post meetup.  We two stayed back for a couple of hours conversing on various topics and how to approach the problems.  He was assisting a recent graduate to get started in Software Testing.  His question to me was, "What all should I share and where should I start?".

I ask the same question to myself when I sit back and observe my practice and journey.  That is, 

"Ravi, what is the one advise you give to your younger self when you are starting career and practice in Software Testing?"

That question of him resonated in me as I listened to him.  I said him, 

"I ask the same to myself. I align; I'm a beginner resetting and starting over everyday using the learning of till date.".

Note this; what I'm saying here it is my experience and learning.  I feel, it make sense to me in my contexts.  You evaluate it and take it to your context.  Nevertheless, I see it is sensible advise and it should be must in your practice the day you start to practice Software Testing & Engineering.

My Words and One Advise

This is a advise I give to younger myself who is starting the career in Software Testing & Engineering,

 "Practice programming!  This should be your day to day activity [practicing programming] to improve your testing skills.  It is critical and elemental to be a better Software Test Engineer.  Do not ignore it.  Embrace programming and leverage your testing."

Any other skills that you hear or said, you can apply them on your programming practice work.  Evaluate it.

For example, if you hear skills as questioning, thinking, critical thinking, lateral thinking, and whatever, apply them to the programming activity and the code you are writing.  Take help of mentors here.  Know the boundaries to set and unset as you progress here.

That said, do not be biased and get stuck to just programming alone. The programming should aid your testing practice and skills, and vice versa.

If I'm testing a software and the general [contextual] systems with which it associates, I should know how the software is being implemented, how it communicates, and manages.  This is essential and necessity for a Software Test Engineer.  This is a need!

Being away from programming, I will fall short in the ways I think of the tests and how I test.  The software that I'm testing is composed from the programming along with other general [contextual] systems integration [specific to the contexts].  If I cannot identify, learn, write and understand the composition of software, I have to think about my testing and question it.

For example, how would this blog post get saved and published when I publish this draft?  What goes behind and in front?  If I cannot see the layers here and understand them, I cannot test better.  Maybe, I can see them to an extent without programming skills.  But, the programming skills gives the different perspectives to the testing, automation and test engineering.  The advantages of these perspectives are unparallel and helps immensely when identifying and designing the context's priority and critical tests, and its samplings.  Hope this makes sense to you and tell how critical it is.

To end this post, I summarize it as,

• I do not have to practice programming in the scale for being a full time skilled programmer.. I practice programming to be a full time test engineer whose testing is sought after.  I stand here for today.
• Practice programming to engineer your testing better.
• Apply all other skills which you see as important to test the code that you are programming in your testing practice.
• If you are starting to practice Software Testing, start from programming and test your program.  In testing this programming of yours, apply all other skills which are said critical for testing better, and upskill them in parallel.

Is Software Testing a Cost to Business?

How quick one gets to have the awareness of the cost and value in the trade, it will be of help.  I consistently exercise the below questions in my practice.

• What are the Values added from my work to the business?
• How do I benefit from this?
• What are the Values removed from my work to the business?
• What does it bring to me?
• Who is not getting benefited from the Values I'm adding and why?
• What are the loses and its impact?
• What are the benefits and gains we are losing?
• Who is getting benefitted from the Values I'm adding and why?
• What are the benefits and gains we are making?
• What are the Costs added from my work to the business?
• What are the impacts?
• How does it impact me?
• How does it benefit me?
• What are the Costs removed from my work to the business?
• How does it benefit me and what do I gain?
• How does it benefit the business and others?  What do they gain?
• Who are all experiencing the Costs from my work and why?
• What are the pains from these costs?
• Who are not having any Costs from my work and why?
• Do they need anything from my work and its value add?

This exercise will help me to know the expectations of stakeholders and business. I align myself to deliver the expectations as I exercise these questions.  

The cost and value from my software test engineering work will have impact and influences my growth with the benefits that I will make.

Is Software Testing a Cost to Business?

I witness the discussion which says testing is a cost in software engineering business.  But, I do not hear the same statement on development.  This made me to think, why?

Here are my understanding on thinking over it for now.

1. Software development is not about just programming.
1. When said development it includes every teams and their work.
2. Programming and Testing are parallel activities in the software engineering which helps each other.
3. If testing is a cost, then for sure, programming is also a cost!  It is an associated activity.
2. In business, every activity and investment on them is a cost in multiple ways.
1. These are evaluated costs and being taken for a purpose in expectation of returns.
3. Have I come across anyone saying Automation in Testing is a cost, like I hear for testing?
1. It is seen as investment and necessity. Why?
1. May be, because, in a belief if [once] automated, that is sufficient enough it goes on for itself without human intervention.
2. If this is the thought, do you think we are doing it wrong?
4. Have I seen the discussion and statements which concludes testing for performance and security is cost?
1. If yes, how?
2. If no, why these two are not seen as cost?
5. Serving and well maintained engineering system is a trade-off
1. I see, we engineers and business choose what to trade-off
1. This decision can be on logical and non-logical basis; but, there will be a trade-off
2. Each trade-off has costs and values
3. What should I trade-off in the software test engineering and why?
4. What should I trade-off in the software engineering that we are doing as a business?  Why?
6. When one is offered a offer letter from a business, there is a CTC mention.
1. CTC means Cost to Business
1. Programmer has a CTC
2. Tester has a CTC
3. Every others in different teams have a CTC including the CxOs
4. We all [and our job] are cost to a business, who can add value and get the high returns

To end for now, I learn, everything and anything business picks up is a cost.  Because, there is an investment on it.  If there is no investment, is there anything happening or progressing and changing?

I do not want to get into discussions which says software testing is a cost.  I see such discussion is lacking in understanding the software engineering and intrinsic for first.  Software Development is not all about programming; the programming is one small part of it.

There are different teams that work together in collaboration to develop and consistently delivering the usable software systems.  In this process, everything in software engineering business is an invested and evaluated cost in the interest of returns and values that are expected.

It is time to know and ask "What way of testing and its outcome is a cost?" than saying and listening to -- 'testing is a cost', which is nonsensical.  This holds good for any activity in software engineering.  

Any serving engineering marvel is a calculated and evaluated trade-off.

Functional Testing Is Must In Performance And Security Testing

 

I'm sharing about how I missed to test for functionality while I was immerse focused on testing for performance of a Stored Procedure.  I was unhappy for a couple of days as I missed something that I practiced for years.  

I'm glad for reinforcing this learning with much more awareness into my testing's MVT and MVQT, now.

Context of Testing

A Stored Procedure was optimized for better execution time.  No change in the functionality.  This part of the system is not touched for a long time (years?).  There was no change in functionality here for long time (years?).  The time taken by SP was of concern.  I was asked to test for the optimization.

The complicated area, here, is the test data to use.  It took me days, for identifying and building the test data to test this optimization by mimicking the production incidents, use cases, and data.

When I got the test data ready, it was the fourth day of my testing this change.

Where Did I Go Blind By Being Focused?

The test data that I prepared is solely for the evaluation of the execution time.  This test data helped to test functionality as well.  But, my focus was on evaluating performance not functionality from this test data.

The change in SP did impact the functionality.  I was supposed to use the large data range to test for functionality of this feature which includes two SPs.  But, the task assigned was to test just one SP which is optimized.  I got blind here!  

Are you asking, what is the impact of this functional problem?

• In the one complete business work flow, this functional problem added the same data into different sets in the subsequent iterations.  Redundant Data -- This is not an expected behavior.

I just spoke performance, traces, data I/O and execution time, because that was a pressing problem.  Why?  That was the objective given to me.  

My testing mission fell short in redefining this objective.  If I had redefined it, I would, have added functionality in the better scale.

If I had redefined it, I would have pulled the other SP into functional testing which is also part of this feature's work flow.  These two SPs are expected to handle the data by eliminating the redundancy.

It was a simple test, but, I did not include/had that in my testing mission that day.

Why Did I Go Blind?

The performance test blinded me for functionality, as I saw the basic functional flow looked functioning.  But, the data count was going wrong when a bigger data range is used in the context.  

See here, how stupid I was in my testing!  I'm testing for a SP that has a change as part of its optimization for execution time.  I never brought the functional testing in.  Why?  I focused on the testing objective.

I just looked into one SP that is optimized.  I did not look the other SP which has to work along with this SP later to complete functional flow of the feature.  Why?  How is that even possible?  I was asking myself this.  I see, this is okay from the perspective of the testing objective I had.  But, not okay from the perspective of a test engineer who is supposed to think the impact and prevent the problems.

My immersed and concentrated focus on performance and its related activities on a SP for four long days did not let me see this.  

What Am I Saying Here?

While I have tested for DBs and ETL systems for years, I did not use my learning here.  What is that learning?

When there is a change in any part of the ETL, SP or DB of a system, testing for the functionality for the business workflow is equally important.  Vary the data dimensions and evaluate the counts.

I was completely hooked into the execution time and the test data while switching between the environments for four days.  The chaos in data between environments is something that misleads easily.  I fell to it this time.

I say to myself, if it is a fix for the performance optimization or a security [or any quality criteria], testing for functionality is equally important and of priority as running the tests for performance or security.

When a DB layer is picked for fixing and optimization, testing for functionality in a equal scale is must.  There is a change in the code or/and infrastructure and it has to be noted with additional attention.

To add on this, this time, I did not go through and analyze the SP.  I took this call from the test team.  This call of me costed and had a major part in letting me not to think of functionality.

My fellow colleague ran a test with varying data size by completing the business workflow and observed the problem, and informed me.  I give the credit to Sandeep.

If I had brought this performance test under the automation, I would not have done this.  Why?  I will evaluate and assert for each data returned for different sizes.  I did not automate here and there was no need for it in this context.

Redefine the testing objective that you have got; it helps when you see the model of a system and test.

Respect all the fix and suspect all the fix.  This helps in a longer run!  

My First Hand Analysis of CrowdStrike Falcon Update Incident

I attempted to analyze the process dump of CrowdStrike shared by my friend.  He said, there could be an attack which is leading to crash of Windows OS globally.  This made me curious to look into the dump and learn.

I had no much context around it, but, a test engineer in me did not sit quite.  I started to analyze the dump information.  Here is my first hand analysis that I made on 19th July 2024 post 10:30 AM IST.

What I Saw?

• It is a Windows OS's process dump.
• Looks like something with C or C++ application reading how the memory offsets were in the dump.
• It started to read a memory offset.
• Then the process witnessed an exception.
• Here the program could not read further
• Why it could not read further from this offset?
• My little experience of testing drivers on Windows OS for a card printer machine, refreshed and recalled what I had witnessed when testing.

Scratching and Striking My Mind

I started to ask these questions myself while I asked what could have gone wrong.  I could not stop here as I was curious what led Windows machine crash.  I referred to web and learn there was an update by CrowdStrike, and then this incident.

The bugs do exist in every software no matter the level and depth of testing, automation and engineering's excellence.  All software do crash and OS is not an exception to it.  But, what made the update to crash the Windows OS?  Pointing and blaming CrowdStrike or Microsoft is not a way for the practicing test engineer.  If these two organizations are serving its huge customer base, they have something working and reliable.  Engineering does not eliminate problems.

By now, I had a thought that it is not an attack.  It is a software bug!  Where is the bug?  What is the bug?  Was it not experienced in pipeline?

The Open Ended Questions

I had these questions as I analyzed and spoke to my friend.

• What is Falcon?
• What was this update to Falcon?
• How frequently the updates are rolled out?
• How the updates are rolled out globally?
• What pipeline do they have in testing?
• Who is impacted the most in business? Is it Microsoft or CrowdStrike?  Impacted in what way?
• What is CrowdStrike?  What they do?  Who are the customers?
• Where do the CrowdStrike's Falcon sit in the OS and what it does?
• How CrowdStrike works in the machines and what it offers?
• What do the dump say? Relook into it with different perspectives.
• How this could have been prevented?
• How will I prevent this if I join this team knowing this incident?

With these questions, I started to analyze the process dump which was shared.

I had more such questions, but these were the first few that I crossed as I started.

Analysis of Process Dump

My interpretation, tells me the below for today

1. Accept that it is an incident as any other incident which I witness in production environment.
2. Do not fall to the speculation happening around.  Remain calm and focus to interpret and understand your exploration.
3. I see, if it can start to read from an offset and then ending to experience a non-existent or invalid offset, is it a NULL Pointer?
• What is NULL Pointer?
• A NULL Pointer is a pointer that does NOT point to any memory location and hence does not hold the address of any variables.
• If I do not initialize and assign, the pointer will have NULL as its value.
• For example, int *test;
• When I want to access the pointer test (a location in memory) pointing to, I will not be sure what is in the pointer when I read it.
• I may not set it later or set it.
• In this case, the code can tell if the pointer is valid or pointing to a garbage memory
• But, if I declare it like int *test = NULL;
• I can check if was set and initialized
• It is a better practice to assign a NULL value to a pointer during initialization so that we can check if it is NULL or as any address assigned to it.
• This understanding of Pointer makes me think, is it due not initializing a pointer and so the error code c0000005 on reading a memory that is not valid.
• When we assign a NULL value to pointer, it is a null pointer in C++
• We assign null value for testing and asserting
• If the memory is allocated to a pointer or not
• If it has a return address and is a valid one or not
• If a pointer is not initialized, assigning null it prevents problems to certain extent
• With this understanding, I also read, it started to read from an offset 0x9c, and then failing.
• What is 0x9c?
• In Octal it is 234. In Decimal it is 156.
• Can there be such address in a computer's memory? I don't know.
• If it is a access violation, then is it a memory which is in preemption of the OS?
• If so the OS can terminate the program or process which is trying to access it.
• Is this killing the process and aborting the operation of Falcon's IPC and eventually Windows coming to BSOD?
• This tells me it is not a NULL Pointer in first case but not initializing a pointer to NULL.
• I infer, if the pointer was assigned to NULL, that is initialized, there could have been some hint in the state and event when accessing the memory.
• This is my analysis; but, I have not seen the test code nor aware of the product.  All this inference is based on the process dump and my experience of testing drivers.
• It got something in between from update (a config or pattern?) for which it cannot find and read in the memory?  Why?
• This indicates me, it could be a bug, that is, a logical problem.  This is my hunch for today!
4. Data in the dump
• Exception Address
• Read from Address 0x9c
• Exception Code: c0000005 (Access violation)

Testing my Interpretations

CrowdStrike as an org when it caters its SAAS to such a customer base, won't it have a testing pipeline

• It will have, I have no doubt in it.  They test and roll out the updates, I believe in it.

Did they witness any such incidents earlier?

• I searched on web for it and I did not find something similar on the Windows, earlier.

Is this a NULL Pointer?  Are you sure?

• No, I'm not sure.  But, there is something that is leading it to address which does not exist or which is invalid?  I will have to wait for their RCA to know technically what caused this.  But this is my understanding reading the dump.

How do you think it is a memory access problem?

• The error code 0xc0000005 says that.
• I referred to driver easy website for the information because my experience of testing the drivers for Windows OS and experiencing such incidents led me there.  This is what I learn:
• https://www.drivereasy.com/knowledge/solved-how-to-fix-0xc0000005-error/

Do you think the programmer would not have handled the obvious Pointer and NULL initialization?

• I believe there will be a check for Pointer and what it is pointing to.  But is it due to no initialization?  Technically this has to be analyzed which I cannot do.  I will have to wait for CrowdStrike team to share the tech details.

Is this a driver problem that killed the Windows kernel?

• I don't know.  But, the .sys file will not have driver as per my learning.  It will have information about the drivers and any configurations.
• This incident is a problem, which impacted both CrowdStrike and Microsoft.  Maybe, both will have their areas to look and fix it they see so.  But, in this context, CrowdStrike can fix it quicker and that is much better -- is what I understand.
• I'm a Windows user for long time.  I see, Windows has worked well to all my contexts so far.  The Engineers of Windows OS knows better than me here.  I'm not well aware and informed as they are.
• CrowdStrike's engineering team are skilled and they are rolling out updates often in a day.  They have a better pipeline when this is being done.
• But, the question I have is, how did this happen?
• No one lets such problem into production when they are aware of it.  Do you?
• There is something that has not come to their observation and experience.  What is that?
• Knowing this will help to prevent this and similar incidents happening in future.
• I'm waiting to know what did not come to their experience and led to this incident.

What could be in the .sys file of CrowdStrike?

• I don't know!  I want to learn that.
• But, from my testing of .sys file and drivers on Windows OS, I learn there could be a configuration details with certain pattern or information to capture at run time, and help the installed software to run.  This is my learning and awareness from my testing.
• That said, testing at OS level and Anti Virus engines are not obvious.  Testing of drivers is like the risky mines.  What is sufficient and good enough in test coverage?  It needs an expertise at OS internals level.
• Windows OS having such a fragmentation in its versions, updates and patches, it is a battle field and mines for engineers building such solutions for sure!
• I learn, the Windows OS stopped when an application tried to access the invalid region or non-existent memory.
• The update which was rolled out, did it have a configuration or a pattern that showed a logical problem when processing it?
• I have such questions and thoughts that are striking my mind as I think and build a problem model for the same.

Is this a race condition incident?

• I see, it is not a race condition incident as users across globe experienced it.

Is this specific to a Falcon version, OS version and hardware?

• Not all host machines would be on latest version of Falcon, is my presumption.
• At least, n-1 and n-2 versions should be on host machine which experienced this behavior.
• So it is not a Falcon version specific, I see.
• It looks to me as it is not specific to the Windows OS version and hardware configuration.
• It is an application software problem which occurred at driver level is what I see.
• This is an IPC communication and process is my understanding.
• The driver can receive the IPC communication in continuous mode.
• At times, this can get queued based on the application and what it does.

Where is the Problem?

Well, I'm looking and pulling from my visualization by relating with my experience of testing the driver on Windows OS.  I don't know the exact reason or close enough to tell what could have gone wrong.

Reading the process dump, it says accessing a memory that does not exist or corrupted.  One of the high possibility is, the starting offset is seen but it is not helping when reading.

• For example, Ravi has the address of India's Prime Minister house.
• But, he does not know from where to start despite having the address.
• He is void and null in knowing where to start and what to do when he is not initialized with the start location to begin the travel to the Prime Minister's house.
• In short, he do not know where the address is pointing to and what it has, though he is given a address to start.
• Can he access the Prime Minister's house premise without any access granted and authorized to do so?
• If not, won't he be arrested by police or other security forces and stop him?

Do I Know the Precise Problem?

I don't know!  I do not know the CrowdStrike product and platform.  I'm waiting to read the technical details from Crowd Strike.

I see, it comes to the data, state and event.  I would focus on how to prevent it learning which data, state and event led to this behavior.  I think of figuring out the Test Design and Strategy that can help me to identify such use cases.  I focus here and see can it brought into the automation so that it gets exercised and regressed consistently.

If it is due to the memory access that had a problem, I did such tests when testing driver for a hardware machine on Windows OS.  I will share the tests that I did in upcoming blog.

I wrote the technical analysis from process dump to CrowdStrike and Microsoft.  I did not get a response.  Anyways, I'm sharing the overall information in a non-technical way so that it is consumable to most readers here.

Note: Here are another threads of me sharing my thoughts on same:

1. https://x.com/testingGarage/status/1814215089525821763?t=XSFdx69ElL0ZmBOcEFrTjg&s=19

2. https://www.linkedin.com/posts/ravisuriya_%3F%3F%3F%3F%3F%3F%3F-%3F%3F%3F%3F%3F%3F%3F%3F%3F%3F%3F-%3F-activity-7221156949445206017-oeRa

Performance Testing - The Unusual Ignorance in Practice & Culture

 

I'm continuing to share my experiences and learning for100 Days of Skilled Testing series.  I want to keep it short and as a mini blog posts.  If you see, the detailed insights and conversations needed, let us get in touch.

The ninth question from season two of  100 Days of Skilled Testing is

What are some common mistakes you see people making while doing performance testing?  How do they avoid it?

Mistakes or Ignorance?

It is mistake when I do an action though I'm aware that it is not right in the context.

I do not want to label what I share in this blog post as mistake.  But, I call it as ignorance despite having or not having the awareness, and the experience.

The ignorance said here is not just tied to the SDLC.  It is also tied to the organization's practice and culture that can create problems.

To this blog post's context, I categorize the ignorance in these categories -- Practitioner and Organization.

1. Practitioner's ignorance
• Not understanding the performance, performance engineering, and performance testing
• When said performance testing, taking it as - "It is load testing"
• No awareness on what is performance and performance engineering
• Going to the tools immediately to solve the problem while not knowing what is the performance problem statement
• Be it web, API, mobile or anything,
• Going to one tool or tools and running tests
• No much thinking on how to design the tests in the performance testing being done
• Ignoring Math and Statistics, and its importance in Performance analysis
• No idea on the system's architecture, and how it works
• Why it is the way it is?
• The idea of end-to-end is extended and used in testing for performance and having hard time to understand and interpret the performance data
• How many end-to-end your tests have identified?
• Can we test for performance to all these identified and unidentified end-to-end?
• Relying on the resource/content in internet and applying or using it in one's context without understanding it
• No idea on the tech stack and how to utilize the testability offered by it in evaluating the performance
• Not using or asking for testability
• Getting hung to most spoken and discussed 2 or 3 tools on the internet
• Applying tools and calling out it as performance testing
• No attempting to understand the infrastructure and resources
• How it impacts and influences the performance evaluation and its data
• Idea on Saturation of resources
• Thinking it as a problem
• Thinking it as not a problem
• Not working to identify where will be the next bottleneck when solving a current bottleneck
• What to measure?
• How to measure?
• When to measure?
• What to look when measuring?
• Not understanding the OS, Hardware resources, Tech Stacks, Libraries, Frameworks, Programming Language, CPU & Cores, Network, Orchestration, and more
• Not knowing the tool and what it offers
• I learn the tool everyday; today, it is not the same to me compared to yesterday
• I discover something new that I was not aware of what it offered and exist
• I learn the new ways of using the tool in different approaches
• No story in the report with information/image that is self-describable to most who reads it
• And, more; but the above said resonates with most of us
2. Organization's ignorance
• At the org level, for first and to start, it is ignorance in Performance Engineering
• Ignoring the practice of performance engineering in what is built and deployed
• Thinking and advocating, increasing the hardware resources will increase and better the performance
• In fact, it will deteriorate over a period of time no matter how much the resources are scaled up and added
• Ignoring the performance evaluation and its presence in CI-CD pipeline
• The performance tests on CI-CD pipeline should not take beyond few minutes
• What is that "few minutes"?
• Not prioritizing the importance of having the requirements for Performance Engineering

Recently, I was asked a question - How to evaluate the login performance of a mobile app using a tool "x"?

In another case, I see, a controller having all HTTP requests made when using web browser. Running these requests and trying to learn the numbers using a tool.

I do not say this is wrong way of doing.  That is a start.

But, we should NOT stay here thinking this is a performance engineering and that is how to run tests for learning a performance aspect[s].

To end, the performance is not just - how [why, when, what, where] fast or slow?  If that is your definition, you are not wrong!  That is a start and good for start; but, do not stick on to it alone and call performance.   It is capability.  It is about getting what I want in the way I have been promised and I expect; this is contextual, subjective and relative.  The capability leads to an experience.  What is that experience experienced?

Sometimes, serving the requests by what you call as slow, is a performance.  What is slow, here?

The words fast and slow are subjective, contextual and relative.  It is one small part of performance engineering.

That said, let me know, what have you been ignoring and unaware in practice of Performance Engineering & Testing?

RAAMA: My Test Discovery Model

 

RAAMA -- I Look at You Everyday!

I have tried to put up one of my Test Discovery models in a conceptual way here with name RAAMA - Refer to, Arrange, Action, Monitor, and Assert.

Maybe this model helps you and your test engineering team as it is helping me.  Use this to your context with addition or subtraction for what you are seeking.

I refer to this RAAMA of me everyday and when I'm testing.  I'm finding the new learning and realization everyday that I was unaware earlier.  My understanding of RAAMA is not same what I had on the previous day.

My understanding of this RAAMA is incomplete and I have made PeACE with it by accepting it.  My understanding is growing and getting better everyday.  I will share a better version of it as I experience it.

Each time I look up to RAAMA and refer to it, I see a new dimension to RAAMA.  The awareness, exposure, and the questions are getting better giving the better realization of what I was ignorant and unaware.  The RAAMA is exposing me to be a better test engineer today than what I was earlier.

RAAMA - I Look at You Everyday!

RAAMA - One of my evolving models for Test Discovery

Note: I have not explained in detail what I mean for each node and its sub-nodes.  I can talk and discuss it with you if you look for it; I'm just one email away to get started.

Test Data is Not [Equal To] a Test

 

Not every release is a critical release!

There are releases which are critical from technical and business interests.  In such critical releases, most of my time is spent on understanding the technicalities of the system and the test data identification on identifying what are the priority tests.  Identify and building the test data takes major chunk of the testing time. This effort has helped me, testing, stakeholders, projects and products immensely.

Test Data != Test

Looks like the word "Test Data Management" has become a buzz and marketing word.  In how the word "Test Data Management" is pushed, to me it looks like the intent is missing for -- how to identify the tests and its test data?

• If I cannot think of tests, how can I think of Test Data?
• How can I think of tests, the right tests in the context, and the priority tests out of it? 
• If I cannot get this, how can I get the Test Data for all these tests?

Before talking of Test Data Management, we need to talk about the intent and goal of the test.  If I have a clarity here, by being aware of the product's technical internals, architecture and its eco system, I can think of test data in a better sense.  For first, the PaaS which provides Test Data Management solution has to push and enforce for learning and mentioning -- What is the intent of the test?

When I know, for what to evaluate and how should I be evaluating it, those test intent will lead me to the Test Data and its dimensions which should be vectoring the tests.  

Test Data is not [equal to] a Test.  Test Data is one of the variance, in fact, it is a multi-dimensioned variance that a test will [need to] have.  While a test has one deterministic objective, the test data will have multi-dimensioned vectors that instruments a test.

Test Data has its role in a test, and, it is a critical role.  Hence, we practicing Test Engineers talk and spend our time on Test Data in equal importance as we give our time to identifying, designing and execution of tests.

Test Data is not a test; but the test data is an expression of the test's intent.  Test Data is one of the byproducts of a test.  Having this clarity is important.

Note this, the test data is not the only expression of the test's intent.  A test will have multiple touch points; these touch points express, advocate and can show the intent of a test.  A test data is also a heuristic as a test is.

 

Test Data Management

The word "management" is underrated and not attempted to understand from context where it is being spoken.  How do this word sound -- "Test Data Leadership"?

In engineering, especially in the software engineering, the word "management" inherently talks primarily about design.  On day-to-day operation, the management designs its strategy and approach to solve the problem and challenges.  

When said "Test Data Management" in software engineering, it is about strategizing and approaching the problem of identifying and categorizing (subsets) the data to test.  A kind of leadership at one layer.  It has its role and critical to the deterministic outcome for a test.

In machine learning, we can see this categorization -- training data and test data.  Then, is training data not a test data?  It is!  The training data is designed [managed] to train and this data as well identifies the problem as the training progresses.

The data which are identified, designed, categorized and collected as Test Data, are well sampled data in the context, for the intent of a test.

To conclude, Test Data Management is one of the correlation in software engineering to solve a problem; it is contextual in how and what tests and testing is executed.  The different teams and organizations has their own way of managing the test data.  How they do it, it is their way of doing it.  We have to look Test Data Management from point of Test Engineering and not as an another buzz word to sell just as a product and business's service.

Is Performance a Perception to an Engineer and User?

 

When hearing about performance from customers and engineers on team, I see each are having a perception of it on using a product.  To one it was fast enough, to another it was as usual and for one it was too slow.  Each are expressing their perception.  But, what is the performance?

What is Performance?

I see, understanding and knowing "What is performance?" is important for everyone who is involved in building the software system and product.  In my opinion, this should be the start point.  It is beneficial, when everyone has a shared common understanding to it as a team and business.

Performance is an emotion to a user!  Technically, the performance has multiple facets to it for understanding the capability of a system and its sub-systems.

Facets of the Performance

What facet do you consider and call it out as Performance?  This is another point where most of us do not align.  For example, below are the different facets that we usually hear and read a lot:

• Heap and Memory
• Threads
• Not supporting for concurrency
• Not handled well in concurrency
• Holding up other processes and causing bottleneck cases
• Memory Leaks
• Concurrency
• Memory not reclaimed
• and, more ....
• CPU consumption
• Open connections and its I/O
• Held up in processing requests
• No enough resources to process
• and, more ....
• DB I/O
• Open connections
• Unindexed data
• SPs and Queries holding the transactions
• Incorrect configurations of DB Server and nodes
• and, more ...
• Disk I/O
• Running out of space
• RW I/O not responsive
• Network consumption
• Unmanageable transactions
• Transaction's data, size and time
• Latency
• Latency in which interfaces?
• Ineffective caching, queuing and messaging
• GUI not rendered or painted
• GUI exhibiting jank behavior
• GUI partially rendered
• GUI not in a interactive state
• GUI not responding to an action
• GUI and UI loading multiple times with no room for interaction 
• Terminal yet to return and show the prompt
• Older and deprecated libraries with latest dependencies
• Server, orchestration, and sub-systems configured incorrectly
• Hardware resources and its specifications used in a context
• Display refreshing rate and frames lost with GUI rendered
• Heat dissipation
• From the hardware
• Experienced in the environment
• Time taken for a request to reach the actual end point
• Execution time on receiving a request
• And, more...

Further we have classified it to frontend and backend; both are important and equally needed.  The webpage has got different KPIs and metrics to determine where do its performance stand.  Likewise, for backend.

Which facet of performance need to be evaluated and in which phase?  Why?  The perception will be established when testing, on how we test it, and how one uses the product.

With all these for performance, where to start and what to look at?  This is one of the question with which we are left in Waterfall and Agile.  That is where the eighth question of 100 Days in Skilled Testing comes in -- Can you share some best practices for conducting performance tests within an Agile development environment?

Have you asked these questions to yourself and team?

1. What is performance to you and to your stakeholders?
2. What should I consider in evaluating for the performance of your software system?
3. What is the practice I want to pick to evaluate the performance?
4. Where do I start and how?  

MVQT: The Testing and Tests with a MVP's Perspectives

I was leading multiple teams and its delivery in a testing service company.  Then, I came up with this thought -- Like MVP, I also have the MVT (Minimum Viable Tests) for a MVP.

Further, I expanded this thought in my day-to-day practice on tailoring to different contexts. I'm observing that it is applying well to the different contexts when I tailor it to the contexts.  After experimenting it for 10 years, I'm sharing this as a blog post.

What is a MVP?

I take this from Eric Ries.  It looks simple and precise to me.

The Minimum Viable Product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.

I see this technique [and a concept] can be applied to anything when I'm developing.  As a test engineer, I develop the tests and test code in major as part of my testing.  On applying the idea of MVP to my testing and deliveries, I see the value and result.

Reading this blog post of me to know who is a developer.

Testing, Tests, MVP and MVQT

In software test engineering, I see the MVP as Minimum Viable Questioning Tests.

The Minimum Viable 'Q' Tests (MVQT) for a focused area of a feature [or to a feature]

• Helps me to identify the priority tests that should be executed for first
• Allows me to learn information on priority which matters critically to product and stakeholders
• So that a informed decision can be made.

The Q in MVQT stands for "questioning".  I read it as Minimum Viable Questioning Tests.  I see the "Q" as a placeholder for the Quality Criteria.  That is, MVFT means Minimum Viable Questioning Functional Tests to a feature or a workflow.

The MVQT are key to know:

• Have I identified and designed the priority tests?  How do I know that I have got them?
• Did stakeholders get the information which they wanted to know on priority?
• Did MVQT help me to
• Explore and know what I wanted to know about a feature or a workflow?
• How fast I was here to know and learn this?
• How did I develop my tests incrementally?  Did I?  If not, then, is it a MVQT?
• Did MVQT help me to know
• Am I aligned and in sync with expectations of my stakeholders and customers who are using the software product I'm testing and automating?
• Did the MVQT help me 
• In collecting the critical information in a given context for the scope of testing and automation?
• Do the learning and outcome from this MVQT help to reinforce the validated learning of customer?
• Do MVQT result support the outcome of Unit Testing result?

The tests in MVQT has to be consistently revised and evaluated to keep it as a MVQT.  Note this, not all tests are MVQT.  If the number of MVQT is growing to a part of feature or to a feature, it is time to think about what is MVQT for you.

The "minimum" tests are highly effective and it helps me learn and test better technically and socially.

MVQT and Testing

• Sanity or Smoke Tests
• The set of MVQT which helps me learn can the build be taken further testing
• MVFT - Minimum Viable Questioning Functional Tests
• Apply this to a feature or a workflow or to that part which can be evaluated with minimum tests for its functionality
• To update is this aligning to the validated learning of customer [stakeholders]
• MVPT - Minimum Viable Questioning Performance Tests
• MVUT - Minimum Viable Questioning Usability Tests
• MVAT - Minimum Viable Questioning Accessibility Tests
• MVTxT - Minimum Viable Questioning Tester's Experience Tests
• MVST - Minimum Viable Questioning Security Tests
• MVAF - Minimum Viable Questioning Automation to a Feature
• MVLT - Minimum Viable Questioning Localization Tests
• MVUIT - Minimum Viable Questioning UI Tests

You add more of this to your list and context.

In a way, MVQT should ask and look for the testability, automatability and observability.  If this is not happening, then there is no possibility of saying I have got my MVQT.

More importantly, in the CI-CD ecosystem, MVQT pays a major role.  If I should have my tests in the  CI-CD pipeline, then, the MVQT is the way and it focuses on a targeted area to evaluate it.  Else, it is hard, impractical and not possible to test in CI-CD eco system by delivering continuously.

Ask and Review for MVQT

Ask for MVQT, when you review these:

• test strategy, test framing, test design, test ideas, test cases, test plan, test architecture, test engineering, testing center of excellence, and test code

For example,

• What is the minimum viable questioning performance tests that you have got to test this feature?
• What is the minimum viable questioning performance tests that you have got to test this workflow?
• What is the minimum viable questioning security tests that you have got to test this feature?
• What is the minimum viable questioning GUI tests that you have got to test this feature?
• What is the minimum viable questioning contract tests that you have got to test this end point?

Likewise, What is the minimum viable questioning automation tests that you have got to test this feature?

Ask how these tests qualify as MVQT in this context of testing and automation?

This should help you to see how effective is the test strategy in a given context.

Importantly, the MVQT and its effectiveness is a testability to test your tests.

The Credit is to Me

I'm not sure if the idea what I'm saying here in this blog post is practiced by other test engineers.  I have not seen this being discussed about it in public forum.  I have not come across it in my awareness and to the exposure I have put myself.

Hence, I will take this credit to me.  Giving the credit honestly is not a common sight and practice.  I have not got my due credits for using the ideas, thoughts and work that I have come up with.

So, I make it as a open letter and call out that credit for this idea, thought, concept, and practice will be to me when you listen, use and practice it.

The Agile & Testing: We're Pulled & Hidden from "Shift Right"

Do I Understand the Agile?

If I work in a project which follows the Waterfall approach, I still can apply the Agile Manifesto and the 12 principles behind it.

But, am I adaptable to the change and uncertainty for better ways of developing software and helps others do it?  This shows the mindset of me and others involved.  Have you heard this line, "every project that claims Agile, it has [a small] waterfall in it?".

Agile Software Development is not wrong!  It has values in what it says and for what it stands.  I have no doubts in it!  It describes how I perform a activity and respond to a situation in the software development.

Frameworks and Methodology

We have picked the frameworks which are under the Agile's umbrella. We pick the methodology that suits our work and have customized it to our contexts.

This is what Agile also say, the team will always need to adapt its use of a framework to fit properly in its context.  

Dr. Alistair Cockburn, says, the Agile methodologies are the conventions that a team choses to follow in a way that follows Agile values and principles.

The Agile is Not a Problem!

We do not collaborate well in solving the problem and say Agile is not working and creating more problems.

In reality, how we respond to the uncertainties and what got prioritized, it will derail us.  We derail from what we should be achieving as team [and as an individual] in a given time frame and context.

The Agile as I Understand

In a nutshell, the Agile is about

• Balancing effectively with collaboration in the context we are.
• Identifying, learning and solving the problem as a team no matter where we are in the project's timeline.
• Being ready to face the uncertainty and responding to the changes.
• See the progress and value in what we do as a team and bring the "continuous" in practice.
• Having the consistency in the excellence of our work and the value we deliver with it.

Read the Agile Manifesto and its 12 principles.  It is the day-to-day life of a team who works to build a software in a continuous delivery having the values to a stakeholder.

Testing and the Agile

Testing is a collaborative activity.

The Agile also talks about collaboration as an essence and necessity in the continuous delivery of the software we develop.  

The Agile talks about short interval consistent deliveries.

• How should I fit myself as a Test Engineer here?
• How should I upskill myself to deliver in short intervals as part of continuous delivery?  

I ask and say to my test teams,

• Think of delivering value consistently in the shorter intervals.
• Rather saying -- releases in short intervals, call it out as a value to a stakeholder in regular short intervals as a continuous delivery
• It sounds positive, affirmative and profound.
• Upskill and help your peers to build the skills. List the skills, let us discuss and start our practice on it.
• This mindset changes the perception in how you approach the testing.
• In these short intervals,
• Let us prevent bugs in the ways we can think of.
• Finding bugs also need resources.
• Let's start investing in building and improvising the resources to prevent bugs.
• How can we prevent and where to begin?
• For what should we test and automate other than functionality in these short intervals?
• What is the priority?
• If I do not test and automate for evaluating any particular quality criteria, does it impact the value to a customer in what we deliver?
• If I do not test and automate on a particular seam and system's boundary interface, does it impact the value to a customer in what we deliver?
• How am I collaborating with stakeholders?
• Who are my stakeholders?

In short, I see the Software Test Engineering practice in the Agile setup as this,

• We [team] exploring technically and analytically;
• Evaluating collaboratively in the software development and engineering, by having the effective staged feedback loops in place to improvise the delivery;
• By responding actively to the changes and uncertainties that comes anytime in the timeline of a delivery and practice;
• Thus, the consistent continuous delivery is delivered with value to the stakeholders.

Note: This is my observations so far having worked with 46+ startups in my software test engineering practice.

We're Pulled and Hidden From - Shift Right

I and you are said to Shift Left and start the testing from the inception of a feature's discussion.  And, further we're said, we have to shift left from right.

Did the Agile manifesto and its 12 principles say explicitly to Shift Left from Right?  Who said that?

What I see, How I see and understand the Right and Left here, matters a lot!  If I see the Right within the delivery loop, I'm setting a trap to myself to be late and lag back.  And, probably this trap gets set in the work, that is, to see the Right in the [continuous] delivery loop.  At certain context in continuous delivery loop, starting from Right is a need and necessity.

I see the Right outside the [continuous] delivery loop, as well.  In this Right, I see:

• What is coming up in the tech stack and technology?
• How this is a need?
• How this [is to be and] will be adapted by tech solutions we deliver?
• How should I upskill myself here so that I fit into the loop of continuous delivery?
• In short, the Right should be observing the trend and its analysis so that I'm aligned being agile.
• I will be skilled to test and automate in this change and uncertainties.

The Agile never said to completely move or step out of the Right.  Or, I have not read it in its manifesto and principles. 

You see and learn, what is your Right and ask am I aligning to it for being agile in the Agile.  After all, the Agile says adapting to the changes and uncertainties, yet delivering the value to the stakeholders consistently in short intervals.

If seen closely, the Agile is also about balancing, isn't it?  If I'm moving towards Left and me standing in the Left, am I balanced?

In short, we are being pulled and hidden from Shift Right.

Shifting Right is important to see what is outside the delivery loop.   Are you seeing your Right?  If yes, what is the trend analysis you see?  Where you need to upskill?

Shift Right can also be to look what you can bring into your Shift Left for better development and delivery.  For example, what you see in the market and business space for your product?  Do I have anything to add now in Left from this learning?  Do you see that?  Are not we missing this when we pulled and hidden from Shift Right?

Left or Right, if I do not respond, adapt and deliver for the changes, I need to ask, -- Am I agile and testing in the Agile?

To conclude, know what is your Shift Right while you are delivering values in your Shift Left.  Strike the balance and ace.

Performance & Tests: Getting Started and Data Analysis

 

On running tests,

• We will have data (information) as one of the byproduct.
• Analyzing the data of the integrated sub-systems in isolation and correlation,
• It will lead us to a technical analysis on each integrated system.

In the report, we draft this analysis along with actions to be taken.

Note: When said sub-systems do not ignore or skip the client or consumer; the system does not comprise just server.

No Golden Rule

There is no one way to do a testing.  Likewise, there is no one way or the golden rule to test for performance.  It is contextual and depends on what I want to learn.

In fact, in few contexts, we can have a value adding performance test with just one request.  Just, I should be well aware of -- what is that I want to know and learn from this test.

That said, there are multiple interfaces where we can observe, analyze and learn from the performance data collected.

The fourth question from season two of 100 Days of Skilled Testing, is:

What are your favorite hacks to analyze performance testing results and find anomalies?

Well, this question do not mention explicitly if it is for server or client or database or caching or messaging or for what interface of a system.  It is a question; but, to me it looks too generic and at a point it looks vague.  Having said this, that is how the learning journey and curve starts! 

Result vs Report

What is a result?

• Is it an evaluation after a data [information] is put to scrutiny?
• Or, the result is a data that is collected and not yet interpreted?

It depends on individual or team and how it being practiced.

The result is different from a report.

Getting Started and Data Analysis

I should know how the system architecture is designed and orchestrated with its boundaries and interfaces.  This helps a lot.  What kind of architecture is this?  Is it a monolith?  If it is monolith, my approach to test for performance differs.

If I'm asked to start the analysis of data for a system that I'm not aware of,

• I will start by analyzing the below indicators on knowing the architecture and the orchestration of the sub-systems for critical business workflows

1. CPU usage
2. RAM usage
3. Data I/O
4. Network usage
5. The Heat and sound dissipated from the hardware which holds and binds
• CPU, RAM, Data I/O, Network and tech stacks installed and configured

It hints me to look further and test investigate, when I observe:

• Having a steady consumption
• What is steady consumption in this context?
• Having a low consumption
• What is low consumption in this context?
• Having a unusual consumption spike and fall of it
• I follow the pattern to study further
• What is considered as knee, spike and fall, in this context?
• Having a zero consumption
• Having a maximum consumption
• What is maximum consumption in this context?

Having a high consumption doesn't mean a problem.  Likewise, having a low consumption does not mean all is well.  I have to uncover them to learn what it means in the given context.

In each of this, there will be a pattern.  I will learn them.  I will correlate with other sub-systems and learn what they were doing in the said timeline.

Do you recollect this line -- "the architecture should provide the Testability"?

• I wrote about it in one of the blog posts of Performance Engineering.

I refer to the below by traversing with the timeline,

• The logs by asking for it
• Data recorded
• Any APMs that are in place

I correlate all these with above said indicators.

This gives me a start. It is one of easiest start that I can have to get started with analysis.

Well this is to analyze at the server end.  What about the client [consumer] end?  It is simpler and will share in the coming blog posts.

Do you want to know more on this and other strategies that can be used contextually?  Let us get connected and converse.  I'm happy to share and learn on listening to you.  It is fun and awareness!