There Is No Flaky Test! Then, What Is Flaky?

In general, I understand, the flake means brittle and inconsistent or uneven.  Like the flakes one will have as food.

I'm not a native English.  It took me years to understand and correlate between the words 'flake' and 'flakiness'.

Test and Flakiness

There are no Flaky Tests.  A test is not flaky anytime.

The responses of a system to a test can be flaky.  That is, the response is not deterministic and inconsistent to one's expectation and understanding.  

In other words, the responses of the system is unreliable to the same action, data, state and other parameters.

This is not a philosophy of testing or test.  In my opinion, it is a misunderstanding that is accepted and passed on in the software engineering.

A test is not flaky; the response of a system is flaky.

Are there no flakiness when testing for services?  It exists!  That is, the inconsistent behavior of a service for the same set of data, state and other parameters. What is inconsistent in context of a service's response?

Flakiness is not bounded to just GUI  It can be with any UI [User Interface].  And, an API is an another user interface which opens up to the bunch of services.

To end, 

• It is not Flaky Test.
• It is a flaky response of a system which a test is letting you and me to experience.
• What is a flakiness to one who is testing for security and performance?
• Flakiness is not confined to the UI response or a functional response of a system.

You decide how you should be seeing a test in your suite and label the response of the system you are testing.

Who Talks About Your Work?

You talking about your work is stronger than the work talking about it.  Why?

I see, the work speaks in terms of change it brings -- the feeling and emotion of other people.  In the change, people do not see you.  The people see themselves and it is not wrong.  People see how your work made them feel; not what you feel for giving that delivering work.

How you are feeling, what you are feeling and what is that you are expecting for doing the work -- you should make it explicit and talk about it.  I see this is fair ask, be it at home or at business.  

In business, you are paid for the work.  That is how the business looks at it.  Probably nothing more than that.  If not you, someone else will come in to do that work.  But, then what is the difference in you doing and someone else coming in and doing it?  How do you put this across and persuade with an influence?  This is, you talking about your work and presence.  This matters!

At the bottom, everything has calculations -- at home or at work.  One has to talk about her or his work, presence, and value.  This is a useful lesson for family members at home and peers in work place or community -- which is not spoken or taught in school and home.

It happens that, people talk about you and not about your work.  Then who will talk about your work?  You talk on your work.

To open you up to talk about your work, 

1. To talk about your work, work and know your work
2. There is a thin line between bragging and talking about your work
• When I say talk, it is communicate -- communication
• How do you communicate about you and your work?
3. Talk about your work with,
• Data that can be correlated with change and growth
• Quantify it
• Change and growth is well received when it is quantified
• Illustrate this with data which is evident and interpretable
• I said, evident and not logical; logical does not resonate always
• Change, Impact, Consistency, Influence, Trust, Value
• Highlight the trust and its consistency from your work
• Reliability
• Challenges remaining
• Gains and loss
• Talk about loss which are competent gain
• Changed and unchanged
• Continuity -- how and why
4. Your gain and loss is not just your work, it is also what you talk about your work
5. Know your ego when you talk about your work and others work
6. Before talking about others work, know your work and talk about it
7. In short, talking about others work is -- What you are not doing!

If you do not talk about your work, neither your work nor others talk about it.  Never!

Talk about your work!

Give a Name for the Base64 Image in an Extent Report

 

I read the below question in Testing Mini Bytes community.

Hi All, I am using extents reports in my c# automation framework. I am trying to attach the screenshot but it is always showing with name as base64img. Any idea how to overwrite that base64img to customised text?

Capturing a Screenshot in the Test Run

With the Extent report, one can capture the image of screen during the test run.  This capture can be on certain condition or state being met.  That is, it can be -- on fail of a particular action; in a transition between the states; etc.

In an Extent report, when the image is captured as Base64 image and the thumbnail is set to false, the images captured will have name "base64img".  Clicking on this, I see the respective image.  

No Customized Text as Name for Base64 Image

Say, a Extent report has 10 images captured as Base64 image with thumbnail false.  All these 10 different images will have the same name -- base64img.

• This can be more chaotic to interpret when a test run has captured more than one Base64 image, and, all have the same name -- base64img.
• Do you see that? 
• How do you know what is that image without opening it just by reading base64img?
• That's hard and you have to make additional clicks to know what is in each image!
• Is this a better use of time of you who is reading the Extent report?

Naming the Base64 Image in an Extent Report

I see, the semantics in C# should also be similar to what I'm sharing below.  

The above gist should help to have a customized text given for each captured Base64 image.

In this case, the captured Base64 image name will be the customized text "Resend_OTP_Button_Not_Available" and not base64img.  This makes it more contextual and relatable for the reader of an Extent report.

Monetary Value Round Off -- A Legal and Business Problem

 

I read a blog post from Gaurav Khurana suggesting on round off of a number having decimals.  This is the post.  What made me curious is that number and what it meant.  It was a discount money having decimals.  Here, the number of decimals was not restricted to two.

Gaurav, said, it is a better experience if that discount amount is restricted to two decimals and rounded up.  Initially, I see, this is a fair expectation.

Here is the pic of a bill which Gaurav has shared.  I'm using it here on his permission.

I woke all of a sudden at 2:18 AM today, and I relooked into the post and that value whose decimal is not restricted.  I said myself there is something which cannot be rounded up or rounded down here and I could interpret it for a context.

Interpretation of the Value and its Meaning

1. The "Bank Discount" showed the below amount
• - 450.90999998999996
2. The amount is in Indian Rupees
3. This amount tells, it is the discount offered by the Bank on that transaction
4. On round up, it will be,
• - 450.91
5. If I use the round up value, then the "Balance Paid" amount will be,
• 4,058.18
6. If I do not use the round up value, then the "Balance Paid" amount is still,
• 4,058.18
7. If I do not use the decimal values in "Bank Discount", then Balance Paid is,
• 4,059.09
• That is, I will have to pay 0.91 rupee (91 paisa) more.
• In this case, someone is at loss.  Who?
• Business?
• Customer?
• In the context of this bill, it is, customer who is paying.
• Customer will pay 91 paisa, which is, almost one rupee more.
• Now, imagine, the profit which business makes from all its customers for not having that decimal values in the amount!
• Is this right?

The meaning and value of Balance Paid is not affected, in both cases, that is,

• When I use Bank Discount value as is with decimals.
• Or, When rounded up with two decimals.

My question is,

We should have check on decimals shown and used.  But, should we round up the number when it is of monetary significance and value?

Why I have this question?

• Refer to the below examples.
• 450.35
• On round up, it will be 450.5
• 450.55
• On round up, it will be 451

If observed, on round up, the amount discounted is in excess by 0.15 rupee, that is, 15 paisa; and, in the case-2, it is in excess by 0.5 rupee, that is, 50 paisa.

If the bank loses these paisa for each transactions of same customer and other customers, won't the bank is shelling out the money?  Is this good?

Likewise, if I have to pay 0.15 paisa and 0.50 paisa, am I not paying more?  Then, imagine the big amount that business is making by rounding up a monetary number and making its customer pay in excess.

My Thoughts on Monetary Value Round Off

1. The monetary value should not be rounded up or rounded down.  
1. But, it can have a decimals and limited to two or to what business sees manageable.
• As Gaurav said, decimal to two digits makes life easier for all is my understanding.
2. The customer need to see this monetary value.
2. The generated bill or transaction should have this detail.
3. When making a payment, the billing system, can show a round off value.
1. And, the business should decide upon, should the amount be round up or round down

May be to avoid such arguments and discussions in the billing counter, today, the bill amount will be a whole number and not a number with decimal.

Such round up instances can turn to a legal problems for the business.

I have noticed, the business offering round down on a monetary value, at times.  Also, I have noticed, certain consumer mobile apps showing rounded up number without breaking down the bill details.

Whatever, as a Test Engineer, I and you, have to create a report [ticket] for product's reference; keep the stakeholders informed of such behavior in the system by explaining the impact to business in monetary and legal aspects.

Browser Compatibility - What Problems Are You Witnessing In 2025?

 

One of my long standing work in Software Engineering is testing for Browser Compatibility.  But, is that a bigger problem as it was between 2000 to 2017? 

Today's browsers have to and adhere to the common guidelines and standards of W3C.  Likewise, the web development and artefacts [libraries, frameworks, language, etc.] used are expected to adhere the standards and guidelines.

In 2012, I layed out the evolving model of Test Architecture for Browser Compatibility.  The teams across Aditi Technologies did use this solution and it evolved gradually.  It is the 'go to' test solution and approach to test and automate for web application's browser compatibility, strategically.

Those were the days, where the browsers and web development did not adhere to standards and guidelines.  As a consequence, the web pages exhibited problems and unexpected [intermittent] behaviors on major browsers and its versions.

I understand and aware, that, the browser compatibility behavior can be a GUI difference to a functional blocker.  Further, it can expand to the web page's performance behavior to accessibility and security differences.  I have witnessed all this when testing for the web application and its browser compatibility, then.   

I cannot forget the versions of Internet Explorer crashing while another did not when using the web pages.  

Maybe, on reading this, you are thinking of me as an old tester.  Is it so?

2025 and Browser Compatibility

I'm curious to know and understand from you.  So, I ask this question to you.

Did you experience and report the browser compatibility behaviors recently?  If so, what is that?

Please do not share the project and tech details.  

Requesting you to share what behavior did you notice as part of the browser compatibility and which quality criteria of the web page is impacted.

I'm one ping away on Google Hangout and on an email.  Please do share!  This will help me to contribute back to the Software Engineering and Testing community.

Are We Not Innovating, Then?

 

This last Sunday, I met a friend.  In our conversation, he brought up a topic between him and his colleague.  

He said, there is no innovation happening in Software Engineering.  Instead, he see the apps development to cater business services to people.  That is, in other egineering industries there are innovation being done along with research and development.  But, he did not see anything happening in our Software Engineering and communities.

I asked him, "What makes you not to see the Hackathons [being projected] as an innovation initiative followed up with further research and development?"

He did not see the hackathons as a space for innovation.  His experience is, the organization hosts Hackathon for two or three days in a year.  

Further, he said, "We are asked to participate in the Hackathon and also to work on stories in parallel.  If the stories are not worked in these two or three days, it will be escalated.  And, the ideas and outcome of the hackathon are left as is.  Hardly one or two ideas are picked for a proof of concept work in the project."   

I feel, maybe, what he is saying make sense, as I have witnessed it.

Are We Not Innovating, Then?

I see, there are innovations happening in the Software Engineering.  But, these innovations are not something that common men can consume directly.  But, these innovations are consumed indirectly by the common men.

The innovations that I see and test are not B2C.  Sometimes, it is indirectly B2B via D2D.

Yes, D2D -- Developer to Developer.  I'm not sure if there exist a word Developer to Developer.  This is what I said to him -- D2D.  Maybe for this reason, the innovation and software engineering's problem solving and solutions do not come to the discussion and spotlight.

As an innovation byproduct in the software engineering, there are frameworks, libraries and artefacts developed by the developers of an org [and communities].  This is being consumed by the other organization's developers in their project.  As an outcome, there is a solution being built [using an innovation] and delivered by a business which is consumed by a common men.

Maintaining these frameworks, libraries and artefacts outside the payroll job time is a challenge for any developer.  But, some do it beating all the hurdles they experience.  There are challenges here when it comes to maintaining such projects by collaborating with software engineers from the communities.

Most of my research and development outcomes in the Test Engineering area are consumables of we developers and not the common men.  And, it is not known to all the  developers.  When I say developers, it includes, programmers, test engineers, DevOps and product teams.

To conclude here for now, might be the software project you are working on is also consuming an innovation that you are not aware of.  Talk to your programmers, test engineers and teams.

When Does the Community Choose Not to Respond to Questions?

I read the testing, automation and test engineering related questions posted on the social media and web.  I try to understand the question, problem expressed and collaborate to assist.  I do keep 46 minutes in a day for this activity.  

Sometimes, it is hard to assist reading the question.  I feel like not giving up; but, then I do not see the communication happening actively from other end.

What Makes It Hard To Assist?

The questions asked,

• It will lack the context.
• It does not tell who is the person and what she or he is trying to accomplish.
• It will not have information on
• the environment.
• what is the challenge he or she is facing.
• what she or he tried so far.
• It will not have minimal data as
• screenshot, exception stack trace, the complete error message and details following it, data being used, code outline, and more details to the context.

The above are minimal data needed on removing any sensitive information.  Know and understand what is sensitive information for your context when you are sharing.

Then, what do the question will have?

• Most will have a phrase or a couple or three sentences of what they are doing and what is seen on the screen.  And, asking what to do for what is seen on the screen?

Why it is hard to assist with vague details?

People who want to assist won't have any context about you -- who you are, what you are doing, or why are you doing it that way.  They won't understand your purpose or what you are actually experiencing based on the limited and vague information in the questions.

Without clear details, it is hard to connect the dots or pinpoint the problem.  Instead, people who want to assists will be guessing, making assumptions, and probably considering multiple possibilities.  Is this a way to use the time in community?

When questions follow a similar pattern but vary in challenges and context, it becomes demotivating to decode them.  Over time, people will lose interest in reading unclear questions, leading to fewer responses and missed opportunities for meaningful discussions.

What's happening at other end?

People who want to collaborate and assist will take the time to read the question.  But, when a question is too vague to understand, they often give up.  And, they feel bad for doing so.  The question remains unanswered.

When a solution is provided, there's often no update on whether it worked or not.  Those who contributed keep checking back, only to find no response.  Is that fair to those who invested their time to help?  Would you feel good if you were in their position?

Remember, in the community one can't buy someone's time to listen or solve a problem.  It has to be earned!  And, it has to be earned every single time.

How to Frame and Post the Question?

There is no one excellent way of doing it.  Then, what should I do to post my questions?

1. Know the community.
2. Read through the questions shared earlier in that community.
3. Look at the questions that have found resolutions, acknowledged and accepted.
1. Observe how the question or problem experienced is described.
2. Look at the details shared and how it is shared.
3. Look at the context details shared and how it is shared.
4. Observe how the interactions and conversation is taken forward from the two sides.
1. Closely notice the words and how the energy is kept high in both ends and how each side is pushing for it.
2. Importantly, look at the time taken to respond from both ends.

I do not want to share an example or reference saying this is the way to do it.

You figure out for your problem and to its context.  Share the minimal information said above and ask for the help.  Consistently improvise on how you ask, share and describe the problem.

Caution! Know This Before You Prompt For Your Testing

My Fellow Test Engineers,

If you are using Gen AI and LLMs as an aid to do better work, yeah, we should be exploring it.  I have no second thoughts in it.

Are you giving anything bounded by NDA to Gen AI models and LLMs?

• Like a part of software requirement, video, an image, code or anything that hints what the system is?
• That is, is this part of your prompt text which you are giving to the Gen AI model(s) and other LLMs?  If so, do exercise the below questions:
1. Do my organization and its business see any threat and risk from me for doing so?
2. Does it violate the NDA (Non-Disclosure Agreement)? If yes, how will I be impacted and prosecuted by the employer's business and organization?
3. Will I be terminated from my job or contract for doing this?
4. Will I have to face the legal consequences?

I tell you, for today's LLMs, it is sufficient to have gist of requirement or code or video or an image to isolate one's request. From there, it can monitor fairly better with additional anticipation.

My questions:

1. How would you trust the models behind?
2. Do you know how it is observing especially when you use your business identity to login and prompt?
3. Though you have customized the LLM's model and it is in within your environment, how do you know it is not connecting to its vendor environment and updating the gist?

Let us leverage the prompt engineering, Gen AI and associated LLMs. But, be aware and conscious of what we are giving out!

Also, be aware of what you tell to fellow test engineers. Not all Test Engineers or SDETs think of the consequences; instead just blindly mimic or follow what is being done or said to do by trying and using it.

Let us think about what we are passing to community by just saying to use prompts for --  to write test cases for this given requirement; read the given requirement and give a test design and strategy; review the code snippet and more.  Such loose and vague messages can be harmful if it is blindly followed!

By prompting the text and calling it out as prompt engineering, we might be giving out what we are not supposed to in the context of our employer's work. Caution!

Do Your Bug Report Annoy You and Fellow Testers?

I read the quotes or thoughts often about the code being written. Like, write the code for other programmer and not just for you; so that, the other programmer can pick it in ease and work from there.  You should have come across similar thoughts on code.

Have you ever come across thought[s] that speak about the bug report being written?

The bug report you write, is it for you alone? Or, is it for the audience to whom you wrote? Or, is it for someone who picks it up and work upon later?

How good are the audience of your bug report on reading it?  How did other fellow testers feel reading your bug report?  How easy it was for you to read your own bug report and work on it later?  How smooth it was for other tester to understand your bug report and test the fix?

I experience this:

• A bug report with a precise and helpful technical details did not serve the audience and fellow testers
• A bug report with no precise and helpful technical details did not serve the audience and fellow testers
• A bug report with plain English and attachments did not serve

While I say, that, I see this helped most audience sometime:

• A bug report in plain English giving the context, little or no technical information and associated details

It has happened, that I have rewritten my bug reports on reading it after an hour.  And, I have rewritten the bug report of others as well after testing the fix.  In both cases, I "prevent" the pain which I and others go through to some extent.  At least, I hope so!

To end, I recall this quote of Martin Fowler

Any fool can write code that a computer can understand. Good Programmers write code that humans can understand.

I see, this holds good for a bug report as well.  All and any of us can write a bug report.  A skilled engineer [and test engineer] can write a bug report which does not bring unwanted pain to her or his audience.

Anytime, did you read your own bug report after 3 months of writing it? How deep was the pain and annoyance to know what it was all about?  Give the same bug report to your fellow tester or programmer or product owner; ask, what did they know from it.  

Performance Test Report: Between the Effective and Ineffective Reports

 

In this post, I'm picking the thirteenth question from season two of 100 Days of Skilled Testing.

What is an effective way of reporting performance test results and mention some tools you have used in test execution, analysis, and reporting?

I see two questions.  I see it is not a wise attempt to learn these two questions combined as one.  In my opinion, the second question added, it dilutes and make the whole question vague.

What Should a Report Do?

The report should be contextual, compelling, influencing, and targeted to the intended audience to act upon on making a decision.  The software testing report is not an exemption from it.

The performance testing report should know

• Who are its audiences?
• How they read, relate and understand the information?

If this is ignored, the report will not serve the purpose of commissioned testing.  The effectiveness of a report cannot be determined solely on how the stakeholders responds to it.  

On understanding the risks and problems in the system's current capability, mentioned in the report, the stakeholder might not respond with an action to tune the performance aspects.  This could be for multiple factors including that of business.

Note that, a skilled and problem solving engineer understands the business and how it drives.  Just being technically skilled will not help an engineer to grow in a longer run in her or his career.  The system's performance tuning decisions most times will be driven by business.

Did the report persuade the stakeholders with an awareness, mutual understanding and agreement?  The report should drive this conversation.  If not, we have a problem.

On reading the performance testing report, do the stakeholders get an informed awareness on what happened during testing in the present capability of a system's performance criteria?  Do the stakeholders understand and mutually acknowledge how it benefits and costs the business?

This is the foremost value serving expected from a testing report.  If not, I look at how the data and story is presented in the report.

The bottom line is, did we mutually acknowledge, agree and understand on current capability and consequences?  If not, the basic purpose of the report is not met.

Performance Testing Report

The software testing is a high technical activity.   You agree or not to this, but, this is the reality.

Testing for performance is technical investigation activity. It includes the orchestrated study in correlation of 

• hardware, operating system, network, tech stacks & software used in SDLC, architecture, designs, certain decisions, people, business and you - the test engineer.

The fundamental in-depth awareness and knowledge of these areas is essential and a necessity to analyze the performance's aspect.  The performance testing report will show this trait of you as a test engineer.

We have stakeholders who work in technical area and in non-tech area.  How to compile the effective performance testing report?

There is no one way or defined way of writing an effective performance testing report.  Figure out what works in context of your testing to have a effective report knowing - What Should a Report Do?

Outline of Persuading Performance Test Report

It is a technical story telling in a non-technical way with data, pictures, comparison by relating, metaphors, and contemporary history.  I compose the performance testing reports in line with business targets and objectives set.  I provide a metaphors to relate and know the value and cost.

At times, I will have two reports.  I share it with respective stakeholders.

• One with non-technical summary and conclusion
• The other with technical details, analysis and data from investigation

Sometimes, I include the above two reports in one report based on the context.

In overall, this will be in minimum as part of my performance testing report to start.

1. What part of the system is being tested?
2. Why that part of the system is being tested?
3. Mentioning the vague performance requirements gathered from stakeholders.
4. Refining and precising the performance requirements to be specific, contextual and deterministic.
5. Who are the stakeholders of this report?
• What sections the respective stakeholders to refer for the analysis and outcome?
6. Problem statement of the performance testing statement
7. Brief summary of performance testing outcome. [TLDR]
1. What aspect of system's performance is evaluated and why?
2. Brief summary of performance test carried out and outcome.
8. Detailed Report with Technical Details
1. Analysis and Technical Investigation
2. Representation of data which is analyzed
3. Identification of bottlenecks, risks, problems and its symptoms
4. Summary of the test's outcome

You don't have to stick on to one format or a template.  Figure out what works well in your case so that the intent of your tests and outcome is understood by stakeholders.  Give a structure to your report!

The performance testing reports will have metrics, graphs, numbers and proposals.  The presence of metrics, graphs, numbers and the other said, does not make the report effective.  Then, what makes it effective?  When you call it effective?  When you accept it is not effective?  Only, you can figure it out to your context.  I can assist you here; pull me in.

There is no good [effective] report or not good [ineffective] report.  The report is either

• From a team with skills, experience, trained, and practicing
• From a team which is not trained, and, not practicing

Backtracking of Testing, Security and Tools

 

When I started my software testing career in 2006, I was in this thought -- What tools should I use, so that,

• I can do the testing that is sought after
• I can test for performance
• I can test for security

Moving from a search for tools to building the mindset and attitude.  It is a journey!  It took me time to see this journey.  I hopped on to this journey in 2011.  I see, this is not an ending journey, while I know where should I go and reach.  I'm on this journey.

I had no mentors.  I had no seniors in software testing to guide and discuss on my thought process.  I had developers (programmers) who had little or no interest in testing; so it did not matter to them.  But, they have helped me to be better tester.  I'm grateful to them.  Then, the community was not so connected, organized and share the knowledge as it does in 2024.  The software testing was not considered or seen as a technical activity, then.  I have stood, fought, demonstrated and delivered my testing as a technical activity.  I'm continuing it.

Today, on 24th Feb 2024, I read the below question in a community's social space and decided to write this blog post.

Hey, everyone .... Can anyone please suggest a good tool for API security testing?

This question resonates in test engineers.  Most of we test engineers still look and ask for tools when it comes to security testing.  To test engineers, the performance and security testing are still a conception and activity with tools alone.  In reality, it is not!  If you are in such thought or you come across such question to answer, this blog post is for you.

Backtracking the Problem Identification

In programming, we have an approach by name Backtracking.  It is about exploring in possible ways to find possible solutions for a problem.  And, a best solution which works in context is picked.

What's the problem here?  Testing, Security and Tools. Are you with me so far? Let us backtrack this problem.

Note: I see a difference between the words 'possible' and 'all'.  Hence, I use the words "possible ways" and "possible solutions" and not "all ways and all solutions".

Bounties and Entry

There are reputed bug bounties for security testing.  To get into this bounties one has to showcase her/his discoveries and skills with her/his recognized portfolio.

The tools are accessible to all.  The community edition and licensed edition tools are available.  We use these both editions of tools.

• But, why not all of us with tools cannot get into such invited security bug bounties?  
• You will answer this question if you ask yourself.  Hope this backtracking should have helped by now!

The Security Engineering is a vast practice area in Software Engineering. There are dedicated security engineers in role.  But, we test engineers can take up the testing for the security of software systems which the team is programming and building.

I advise, a practicing test engineer

• To start with building an interest for security engineering.
• Consistently hone and build the mindset, attitude and skills needed for the testing the security aspects.
• Pick simple problems, solve it.  Do it consistently, while you explore the layers.

While this is done consistently, it is time to find the mentors in Security Testing. The mentors will assist you in practicing how to test effectively for security making use of simple contextual necessary tools.  Also, a mentor will let you know how to test for security without tools to an extent.  The tool is effective when known how to use it.  The tools help immensely only if I can test for security. 

To backtrack in a different perspective, did any tool that you use, find a P1 security problem [or risk] by itself in its scan?  Did your programmers acknowledge to that risk or problem?  I will pause with these two question to you.

Today, my testing for security is confined to systems that I test.  I test for web application, mobile apps, web APIs, and database.  I can assist here, if you do the home work and ping me.

Database: Finding the Tables Having Specified Column Name

 

In today's pair testing session with a mentee, we were testing for Database I/O.  We were on PostgreSQL.  One of the questions a mentee had is,

How can I figure out the tables having this column name?

Running through every tables and exploring if the column being looked for is present or not, is time consuming.  It is not a approach to take as well.

I went through this when I started the ETL testing practice in 2011.

Here is the query that works on PostgreSQL to find table names which has specified column name.

Query:

select table_name, column_name
from Information_Schema.Columns
where table_catalog='database_name' and column_name like '%column_name%'

It is a better approach to know the precise column name and using the condition as -- column_name='EmployeeId'.

This query should work on MySQL and MSSQL Server.  If not working on MSSQL, need to look into the FROM and WHERE clauses if it is vendor specific.

Performance Testing - What to Know Before User Behavior and Traffic Pattern?

 

This blog post is in series of 100 Days of Skilled Testing.  I see, I do not have to pick every questions asked in this series.  I pick and share to which I see, I can add value.

The twelfth question from the season two of 100 Days of Skilled Testing, is:

What strategies do you use to simulate realistic user behavior and traffic patterns when conducting performance tests?

The twelfth question asked is vague and it needs to be refined for preciseness to pick it up and continue.

The Question and the Gap

I see the below are missing in the above asked question:

1. What aspect of performance is under evaluation?
2. What is the system that is being evaluated for a performance's aspect?
3. What part of the system is being evaluated for a performance's aspect?
• Queuing? Messaging? Database I/O? Memory? Space? CPU? Client Performance? Functional Module?
4. Who are the users?  What are their personas?
5. How and where the users are accessing the system?
6. What is the context of users accessing this system?
7. What is the geo location of users who are accessing this system?
8. How long these users are connected by accessing this system?
9. Are there any differences among these users in their roles and privileges in accessing this system?
10. Can the user access system through multiple interfaces?
11. Are you assuming the user is on web browser and mobile apps to access this system?
12. Is this system you are referring to, is a software system? Or any other system that is controlled environment like - access door, elevators, etc. ?
13. You are asking to simulate the user behavior and traffic pattern.  Should I assume, I and you know or agree to any volume of user?  And, all these users are here for the same purpose when accessing the system?
14. Are you considering any time or at a particular time when talking about the traffic pattern?
15. Are there any unrealistic users who is accessing your system?  You say 'realistic user'.
• Do you see that bots and non-human are also allowed as a user in your traffic?
16. Have you evaluated this earlier in your system?
• If yes, do you have the history and data for user behavior and traffic pattern?
• If you don't have, do you allow to use or have your competitor's user behavior and traffic pattern data? 
17. What is the tech stack of your system?
• What part of your tech stack, you want to evaluate with this user behavior and traffic pattern?
18. What is the architecture of your system?
19. What part of your system and its architecture is being evaluated with this user behavior and traffic pattern?
20. Are you running this exercise for the first time?  If not, where I can refer to previous exercises?
21. How the interaction and events are handled from its start to completion?
• What all are needed to complete the transaction in work flow?
• How this transaction can go invalid for lack or incorrect data, state and action?
22. What is spike, drop, saturation, expected, unexpected, and average numbers in the traffic coming in?
23. What do you understand by traffic?  Do you mean number of requests coming in?
• Do you mean the being committed I/O operations?
• Do you mean the response received at the other end?
• What is the definition of 'traffic' in this context?
24. What is that you want to study and evaluate by the User Behavior and Traffic Pattern information gathered in this context?

Using the above questions, I will get an idea to proceed.

I will build a model from information I collect using above asked questions.  This model we will used to further in testing for a performance's aspect.  The value added to the performance test depends on this model as well.  To get a better model in context, it is useful to address the gaps.  From here, I start to think further.

What do you ask and look for when building a model for User Behavior and Traffic Pattern?

RAAMA: My Test Discovery Model

 

RAAMA -- I Look at You Everyday!

I have tried to put up one of my Test Discovery models in a conceptual way here with name RAAMA - Refer to, Arrange, Action, Monitor, and Assert.

Maybe this model helps you and your test engineering team as it is helping me.  Use this to your context with addition or subtraction for what you are seeking.

I refer to this RAAMA of me everyday and when I'm testing.  I'm finding the new learning and realization everyday that I was unaware earlier.  My understanding of RAAMA is not same what I had on the previous day.

My understanding of this RAAMA is incomplete and I have made PeACE with it by accepting it.  My understanding is growing and getting better everyday.  I will share a better version of it as I experience it.

Each time I look up to RAAMA and refer to it, I see a new dimension to RAAMA.  The awareness, exposure, and the questions are getting better giving the better realization of what I was ignorant and unaware.  The RAAMA is exposing me to be a better test engineer today than what I was earlier.

RAAMA - I Look at You Everyday!

RAAMA - One of my evolving models for Test Discovery

Note: I have not explained in detail what I mean for each node and its sub-nodes.  I can talk and discuss it with you if you look for it; I'm just one email away to get started.

A Test Is Not a Metric

 

A test execution by human or automation will provide information to be aware & learn.

If there is a metric, it should be for what I got to be aware of & learned.  Not for the number of tests.  If it is for a test, it is blunder, before taking it as a metric.

One can identify infinite tests; automate adding annotation @test in big numbers. Should this number be a metric?

That is a question to ask when you see a metric on this number.

What is of value to you in this outcome from a test?  That value should have a metric, and not test or the number of tests.

Identifying the metric in a context which serves, is not easy.

Number of pass/fail or green/red is a measurement; not a metric!

Anything measured cannot be a metric.

A metric helps to measure in a way that establishes rational correlation & upholds its necessity in business.

Share this awareness!

Test Data is Not [Equal To] a Test

 

Not every release is a critical release!

There are releases which are critical from technical and business interests.  In such critical releases, most of my time is spent on understanding the technicalities of the system and the test data identification on identifying what are the priority tests.  Identify and building the test data takes major chunk of the testing time. This effort has helped me, testing, stakeholders, projects and products immensely.

Test Data != Test

Looks like the word "Test Data Management" has become a buzz and marketing word.  In how the word "Test Data Management" is pushed, to me it looks like the intent is missing for -- how to identify the tests and its test data?

• If I cannot think of tests, how can I think of Test Data?
• How can I think of tests, the right tests in the context, and the priority tests out of it? 
• If I cannot get this, how can I get the Test Data for all these tests?

Before talking of Test Data Management, we need to talk about the intent and goal of the test.  If I have a clarity here, by being aware of the product's technical internals, architecture and its eco system, I can think of test data in a better sense.  For first, the PaaS which provides Test Data Management solution has to push and enforce for learning and mentioning -- What is the intent of the test?

When I know, for what to evaluate and how should I be evaluating it, those test intent will lead me to the Test Data and its dimensions which should be vectoring the tests.  

Test Data is not [equal to] a Test.  Test Data is one of the variance, in fact, it is a multi-dimensioned variance that a test will [need to] have.  While a test has one deterministic objective, the test data will have multi-dimensioned vectors that instruments a test.

Test Data has its role in a test, and, it is a critical role.  Hence, we practicing Test Engineers talk and spend our time on Test Data in equal importance as we give our time to identifying, designing and execution of tests.

Test Data is not a test; but the test data is an expression of the test's intent.  Test Data is one of the byproducts of a test.  Having this clarity is important.

Note this, the test data is not the only expression of the test's intent.  A test will have multiple touch points; these touch points express, advocate and can show the intent of a test.  A test data is also a heuristic as a test is.

 

Test Data Management

The word "management" is underrated and not attempted to understand from context where it is being spoken.  How do this word sound -- "Test Data Leadership"?

In engineering, especially in the software engineering, the word "management" inherently talks primarily about design.  On day-to-day operation, the management designs its strategy and approach to solve the problem and challenges.  

When said "Test Data Management" in software engineering, it is about strategizing and approaching the problem of identifying and categorizing (subsets) the data to test.  A kind of leadership at one layer.  It has its role and critical to the deterministic outcome for a test.

In machine learning, we can see this categorization -- training data and test data.  Then, is training data not a test data?  It is!  The training data is designed [managed] to train and this data as well identifies the problem as the training progresses.

The data which are identified, designed, categorized and collected as Test Data, are well sampled data in the context, for the intent of a test.

To conclude, Test Data Management is one of the correlation in software engineering to solve a problem; it is contextual in how and what tests and testing is executed.  The different teams and organizations has their own way of managing the test data.  How they do it, it is their way of doing it.  We have to look Test Data Management from point of Test Engineering and not as an another buzz word to sell just as a product and business's service.

Is Performance a Perception to an Engineer and User?

 

When hearing about performance from customers and engineers on team, I see each are having a perception of it on using a product.  To one it was fast enough, to another it was as usual and for one it was too slow.  Each are expressing their perception.  But, what is the performance?

What is Performance?

I see, understanding and knowing "What is performance?" is important for everyone who is involved in building the software system and product.  In my opinion, this should be the start point.  It is beneficial, when everyone has a shared common understanding to it as a team and business.

Performance is an emotion to a user!  Technically, the performance has multiple facets to it for understanding the capability of a system and its sub-systems.

Facets of the Performance

What facet do you consider and call it out as Performance?  This is another point where most of us do not align.  For example, below are the different facets that we usually hear and read a lot:

• Heap and Memory
• Threads
• Not supporting for concurrency
• Not handled well in concurrency
• Holding up other processes and causing bottleneck cases
• Memory Leaks
• Concurrency
• Memory not reclaimed
• and, more ....
• CPU consumption
• Open connections and its I/O
• Held up in processing requests
• No enough resources to process
• and, more ....
• DB I/O
• Open connections
• Unindexed data
• SPs and Queries holding the transactions
• Incorrect configurations of DB Server and nodes
• and, more ...
• Disk I/O
• Running out of space
• RW I/O not responsive
• Network consumption
• Unmanageable transactions
• Transaction's data, size and time
• Latency
• Latency in which interfaces?
• Ineffective caching, queuing and messaging
• GUI not rendered or painted
• GUI exhibiting jank behavior
• GUI partially rendered
• GUI not in a interactive state
• GUI not responding to an action
• GUI and UI loading multiple times with no room for interaction 
• Terminal yet to return and show the prompt
• Older and deprecated libraries with latest dependencies
• Server, orchestration, and sub-systems configured incorrectly
• Hardware resources and its specifications used in a context
• Display refreshing rate and frames lost with GUI rendered
• Heat dissipation
• From the hardware
• Experienced in the environment
• Time taken for a request to reach the actual end point
• Execution time on receiving a request
• And, more...

Further we have classified it to frontend and backend; both are important and equally needed.  The webpage has got different KPIs and metrics to determine where do its performance stand.  Likewise, for backend.

Which facet of performance need to be evaluated and in which phase?  Why?  The perception will be established when testing, on how we test it, and how one uses the product.

With all these for performance, where to start and what to look at?  This is one of the question with which we are left in Waterfall and Agile.  That is where the eighth question of 100 Days in Skilled Testing comes in -- Can you share some best practices for conducting performance tests within an Agile development environment?

Have you asked these questions to yourself and team?

1. What is performance to you and to your stakeholders?
2. What should I consider in evaluating for the performance of your software system?
3. What is the practice I want to pick to evaluate the performance?
4. Where do I start and how?  

Who is a Developer?

 

Who is a Developer? 

I understand the developer is the one who builds.

• A product owner builds the product by bridging the gap between market, business and consumer.
• A programmers builds the product by programming.
• A tester builds the product by testing it with her/his tests and programming the test code.
• A devops engineer builds the product by building and managing the pipelines and environment.
• A DBA builds the product by writing and managing the database environments.\
• A support engineer or executive builds the product by troubleshooting and listening to the problems being reported.

Who else do you see in developing the software system and its product as a solution?

If you see,

• With the skills and expertise we have and build consistently,
• Each of us will work with our expertise in a particular space of the problem solving as a team and business.

Each of us develop some kind of artefact that are combined to build and ship a usable software.  To me we all are developers with a specific roles.  As a developer we upskill consistently and catchup with current and upcoming changes.

Note: The above said are not the only tasks of these roles.  Each do beyond this in developing & continuous delivery of a usable software.

The Actual Performance Bottleneck is a Test Engineer's Awareness & Practice

 

The bottlenecks are necessity.  We cross bottlenecks everyday; just, we do not observe it.  If you have not read my today's thought on the bottleneck, read it here.  

The seventh question from season two of 100 Days of Skilled Testing is:

What are the common performance bottlenecks, and how do you go about pinpointing and resolving them?

I have a different opinion and thoughts to share on this question.  It may look weird, but, I see this is the reality.

Let me share what happens to a practicing Test Engineer when going through a bottleneck in practicing a subject to upskill.

• The time taken to accomplish tasks and its milestones looks unusually high - the other extreme end.
• Some attempts will not even kick off on entering the bottleneck for several reasons. 
• One of the reason is the fear and what my peers think if I fail to accomplish it.
• I lose the focus.
• It leads to lose of interest, unhealthy confidence and the inconsistency.
• The procrastination kicks in.
• Without no self-motivation, determination and consistency, we give up; clog and remain in the bottleneck.  One will fail to make use of the bottleneck to scale self.
• One do not go back or come out of it.
• We start to blame the bottlenecks.
• But, we fail to understand that the bottlenecks are part of the system.

What makes one to have a hard time with available bottleneck is,

• Ignoring the bottleneck.
• Not being aware of the benefits and impact from the bottleneck's existence.
• Its effects are adverse in a longer run
• Likewise, its benefits are immense in a longer run

If seen, the existence of bottleneck has two extreme ends -- good and not so good.  The bottlenecks cannot be eliminated or eradicated; it can only be managed with awareness and knowing how to adapt and scale through it.

Me, The Actual Performance Bottleneck

This may appear as a critical self evaluation.  But, it is not.    In my experience and practice, I'm learning consistently,

• If there is any difficulty in communicating about the tests and identifying the information from my tests,
• It is do with me for first.
• I will have to communicate it and why it is so.
• If I'm not exposing myself to a subject's area and its practice by putting the subject to test,
• I'm the bottleneck.
• Further, the other bottlenecks that I encounter from all other systems, it easily compounds the problem and its impact.

Did you see the bottleneck is a necessity? It drives me to scale and be operable.  If no bottleneck, may be, I will not expose myself to the awareness to identify and learn the information.

Call out any testing with a name, for first I will be the bottleneck to myself and to my testing, if I'm not aware of it and not practicing in it.

For example, I'm asked and taught to do functional testing at a GUI layer.  Why I'm not asked and taught to do the performance and security tests at a GUI layer?  Or, why I do not think of it?  Should someone say me to think of it, practice it, and, do testing for the same?  If the floor and industry do not push me to do so, I will have to open myself to create that opportunity and awareness.  Did you see that, this is my performance bottleneck?

To test, evaluate and identify [pinpoint] the performance bottleneck in the software system, I will have to practice Performance Engineering & Testing.  Only then, I will be able to identify its existence precisely before pinpointing.  If I do not practice by building and refining the awareness, I cannot explain what I'm pinpointing.

I pinpoint myself [the bottleneck] first, if I identify wisely what is the performance bottleneck.  If I fix my bottlenecks, I will be able to identify bottlenecks in the system that I test.

Let me practice the performance engineering and help my team, and community to practice it.  If not, I will not be in position to name a bottleneck, forget pinpoint it. I will just follow, what others say and think it is right.

Note: I share the above from my personal experience in testing practice and also from reading [& answering] the performance testing questions that are posted in the communities social spaces.  I see fixing this and practicing better here is important for first, than identifying and solving the bottlenecks in software system.

The "Bottleneck" in a Test Engineer's Eyes

 

Preference to Bottle Over Jar! Why?

Have you heard Jar Neck anytime when describing a problem or solution?

• I have heard Bottleneck often and consistently; but, not Jar Neck .  Why? 
• Be it in Software Engineering or day-to-day life problem solving description,
• The Bottleneck is referenced and not a Jar Neck.

Looks like people want Bottle but not the Bottleneck speed and benefits.  Bottle without its neck is a jar?!

Bottleneck exist for better controllability.

• In a bottle, the bottleneck is a solution!  It is not a problem!
• It is to mitigate any risk and problem that arises from the flow of content in the bottle.

Yet we describe, learn and communicate the neck of a bottle as a relativity and analogy to a problem.  

Are you aware of Gateway in the software system?

• The Gateway can be seen as a neck of a bottle which controls the incoming requests and outgoing response.
• Gateway is a necessity.
• We need Gateway to be adaptable in size of its neck based on traffic volume it is handling.  Here, the gateway's neck size should adapt and scale contextually.
• When describing a problem, we are talking about how this bottleneck size which is not adaptable for the context.
• That adaptability has to be built in engineering to scale in any dimensions and magnitude.
• When this is not done, we equate the software system's problem to a bottleneck as a analogy, which is incorrect!  The bottle has got its size and its neck size fixed for a purpose and as a solution.
• The context of a bottle and today's any systems are different.
• It is good to draw similarities from General Systems Thinking and observations.
• But the solution cannot be generic to all systems; it has to be contextual.  The software system has to have its contextual solution.

So, next time when someone in your team or network talk about bottleneck, do share them bottleneck is for better controllability.  Having a contextually resizable and adaptable bottleneck is the need for Software Engineering; not the elimination of bottleneck.

In fact, a software system should have and will have a bottleneck in a point.  And, this bottleneck will be adaptable to the context for having what it should let through and process.

Is the runway of an airport a bottleneck when it is compared to a sky?  Is that a solution or a problem?  Likewise, the ship will have a defined route path and it does not sail without a route path.  Is this a bottleneck to ship and its business?  A elevator can accommodate the defined number of people or kilograms allowed, and not beyond that to move.  Is that a bottleneck?  The esophagus in human body has a size which medical science observes as normal and acceptable; any deviation from that size measurement, the medical science test investigates it as a risk and problem. Why?  Is the circumference size and length of esophagus a bottleneck to human anatomy and physiology system?

The engineering solution will and should have a bottleneck at a point.  Having a adaptable bottleneck to the context is one what tries to accomplish in a software system's scalability and operability.

Please, do not equate solving a "bottleneck" situation with Agile practice.  Does it look like a joke?  I will not be surprised if someone says bottleneck problem is solved if practiced Agile.