Backtracking of Testing, Security and Tools

 

When I started my software testing career in 2006, I was in this thought -- What tools should I use, so that,

• I can do the testing that is sought after
• I can test for performance
• I can test for security

Moving from a search for tools to building the mindset and attitude.  It is a journey!  It took me time to see this journey.  I hopped on to this journey in 2011.  I see, this is not an ending journey, while I know where should I go and reach.  I'm on this journey.

I had no mentors.  I had no seniors in software testing to guide and discuss on my thought process.  I had developers (programmers) who had little or no interest in testing; so it did not matter to them.  But, they have helped me to be better tester.  I'm grateful to them.  Then, the community was not so connected, organized and share the knowledge as it does in 2024.  The software testing was not considered or seen as a technical activity, then.  I have stood, fought, demonstrated and delivered my testing as a technical activity.  I'm continuing it.

Today, on 24th Feb 2024, I read the below question in a community's social space and decided to write this blog post.

Hey, everyone .... Can anyone please suggest a good tool for API security testing?

This question resonates in test engineers.  Most of we test engineers still look and ask for tools when it comes to security testing.  To test engineers, the performance and security testing are still a conception and activity with tools alone.  In reality, it is not!  If you are in such thought or you come across such question to answer, this blog post is for you.

Backtracking the Problem Identification

In programming, we have an approach by name Backtracking.  It is about exploring in possible ways to find possible solutions for a problem.  And, a best solution which works in context is picked.

What's the problem here?  Testing, Security and Tools. Are you with me so far? Let us backtrack this problem.

Note: I see a difference between the words 'possible' and 'all'.  Hence, I use the words "possible ways" and "possible solutions" and not "all ways and all solutions".

Bounties and Entry

There are reputed bug bounties for security testing.  To get into this bounties one has to showcase her/his discoveries and skills with her/his recognized portfolio.

The tools are accessible to all.  The community edition and licensed edition tools are available.  We use these both editions of tools.

• But, why not all of us with tools cannot get into such invited security bug bounties?  
• You will answer this question if you ask yourself.  Hope this backtracking should have helped by now!

The Security Engineering is a vast practice area in Software Engineering. There are dedicated security engineers in role.  But, we test engineers can take up the testing for the security of software systems which the team is programming and building.

I advise, a practicing test engineer

• To start with building an interest for security engineering.
• Consistently hone and build the mindset, attitude and skills needed for the testing the security aspects.
• Pick simple problems, solve it.  Do it consistently, while you explore the layers.

While this is done consistently, it is time to find the mentors in Security Testing. The mentors will assist you in practicing how to test effectively for security making use of simple contextual necessary tools.  Also, a mentor will let you know how to test for security without tools to an extent.  The tool is effective when known how to use it.  The tools help immensely only if I can test for security. 

To backtrack in a different perspective, did any tool that you use, find a P1 security problem [or risk] by itself in its scan?  Did your programmers acknowledge to that risk or problem?  I will pause with these two question to you.

Today, my testing for security is confined to systems that I test.  I test for web application, mobile apps, web APIs, and database.  I can assist here, if you do the home work and ping me.

Database: Finding the Tables Having Specified Column Name

 

In today's pair testing session with a mentee, we were testing for Database I/O.  We were on PostgreSQL.  One of the questions a mentee had is,

How can I figure out the tables having this column name?

Running through every tables and exploring if the column being looked for is present or not, is time consuming.  It is not a approach to take as well.

I went through this when I started the ETL testing practice in 2011.

Here is the query that works on PostgreSQL to find table names which has specified column name.

Query:

select table_name, column_name
from Information_Schema.Columns
where table_catalog='database_name' and column_name like '%column_name%'

It is a better approach to know the precise column name and using the condition as -- column_name='EmployeeId'.

This query should work on MySQL and MSSQL Server.  If not working on MSSQL, need to look into the FROM and WHERE clauses if it is vendor specific.

Performance Testing - What to Know Before User Behavior and Traffic Pattern?

 

This blog post is in series of 100 Days of Skilled Testing.  I see, I do not have to pick every questions asked in this series.  I pick and share to which I see, I can add value.

The twelfth question from the season two of 100 Days of Skilled Testing, is:

What strategies do you use to simulate realistic user behavior and traffic patterns when conducting performance tests?

The twelfth question asked is vague and it needs to be refined for preciseness to pick it up and continue.

The Question and the Gap

I see the below are missing in the above asked question:

1. What aspect of performance is under evaluation?
2. What is the system that is being evaluated for a performance's aspect?
3. What part of the system is being evaluated for a performance's aspect?
• Queuing? Messaging? Database I/O? Memory? Space? CPU? Client Performance? Functional Module?
4. Who are the users?  What are their personas?
5. How and where the users are accessing the system?
6. What is the context of users accessing this system?
7. What is the geo location of users who are accessing this system?
8. How long these users are connected by accessing this system?
9. Are there any differences among these users in their roles and privileges in accessing this system?
10. Can the user access system through multiple interfaces?
11. Are you assuming the user is on web browser and mobile apps to access this system?
12. Is this system you are referring to, is a software system? Or any other system that is controlled environment like - access door, elevators, etc. ?
13. You are asking to simulate the user behavior and traffic pattern.  Should I assume, I and you know or agree to any volume of user?  And, all these users are here for the same purpose when accessing the system?
14. Are you considering any time or at a particular time when talking about the traffic pattern?
15. Are there any unrealistic users who is accessing your system?  You say 'realistic user'.
• Do you see that bots and non-human are also allowed as a user in your traffic?
16. Have you evaluated this earlier in your system?
• If yes, do you have the history and data for user behavior and traffic pattern?
• If you don't have, do you allow to use or have your competitor's user behavior and traffic pattern data? 
17. What is the tech stack of your system?
• What part of your tech stack, you want to evaluate with this user behavior and traffic pattern?
18. What is the architecture of your system?
19. What part of your system and its architecture is being evaluated with this user behavior and traffic pattern?
20. Are you running this exercise for the first time?  If not, where I can refer to previous exercises?
21. How the interaction and events are handled from its start to completion?
• What all are needed to complete the transaction in work flow?
• How this transaction can go invalid for lack or incorrect data, state and action?
22. What is spike, drop, saturation, expected, unexpected, and average numbers in the traffic coming in?
23. What do you understand by traffic?  Do you mean number of requests coming in?
• Do you mean the being committed I/O operations?
• Do you mean the response received at the other end?
• What is the definition of 'traffic' in this context?
24. What is that you want to study and evaluate by the User Behavior and Traffic Pattern information gathered in this context?

Using the above questions, I will get an idea to proceed.

I will build a model from information I collect using above asked questions.  This model we will used to further in testing for a performance's aspect.  The value added to the performance test depends on this model as well.  To get a better model in context, it is useful to address the gaps.  From here, I start to think further.

What do you ask and look for when building a model for User Behavior and Traffic Pattern?

Performance Testing - The Unusual Ignorance in Practice & Culture

 

I'm continuing to share my experiences and learning for100 Days of Skilled Testing series.  I want to keep it short and as a mini blog posts.  If you see, the detailed insights and conversations needed, let us get in touch.

The ninth question from season two of  100 Days of Skilled Testing is

What are some common mistakes you see people making while doing performance testing?  How do they avoid it?

Mistakes or Ignorance?

It is mistake when I do an action though I'm aware that it is not right in the context.

I do not want to label what I share in this blog post as mistake.  But, I call it as ignorance despite having or not having the awareness, and the experience.

The ignorance said here is not just tied to the SDLC.  It is also tied to the organization's practice and culture that can create problems.

To this blog post's context, I categorize the ignorance in these categories -- Practitioner and Organization.

1. Practitioner's ignorance
• Not understanding the performance, performance engineering, and performance testing
• When said performance testing, taking it as - "It is load testing"
• No awareness on what is performance and performance engineering
• Going to the tools immediately to solve the problem while not knowing what is the performance problem statement
• Be it web, API, mobile or anything,
• Going to one tool or tools and running tests
• No much thinking on how to design the tests in the performance testing being done
• Ignoring Math and Statistics, and its importance in Performance analysis
• No idea on the system's architecture, and how it works
• Why it is the way it is?
• The idea of end-to-end is extended and used in testing for performance and having hard time to understand and interpret the performance data
• How many end-to-end your tests have identified?
• Can we test for performance to all these identified and unidentified end-to-end?
• Relying on the resource/content in internet and applying or using it in one's context without understanding it
• No idea on the tech stack and how to utilize the testability offered by it in evaluating the performance
• Not using or asking for testability
• Getting hung to most spoken and discussed 2 or 3 tools on the internet
• Applying tools and calling out it as performance testing
• No attempting to understand the infrastructure and resources
• How it impacts and influences the performance evaluation and its data
• Idea on Saturation of resources
• Thinking it as a problem
• Thinking it as not a problem
• Not working to identify where will be the next bottleneck when solving a current bottleneck
• What to measure?
• How to measure?
• When to measure?
• What to look when measuring?
• Not understanding the OS, Hardware resources, Tech Stacks, Libraries, Frameworks, Programming Language, CPU & Cores, Network, Orchestration, and more
• Not knowing the tool and what it offers
• I learn the tool everyday; today, it is not the same to me compared to yesterday
• I discover something new that I was not aware of what it offered and exist
• I learn the new ways of using the tool in different approaches
• No story in the report with information/image that is self-describable to most who reads it
• And, more; but the above said resonates with most of us
2. Organization's ignorance
• At the org level, for first and to start, it is ignorance in Performance Engineering
• Ignoring the practice of performance engineering in what is built and deployed
• Thinking and advocating, increasing the hardware resources will increase and better the performance
• In fact, it will deteriorate over a period of time no matter how much the resources are scaled up and added
• Ignoring the performance evaluation and its presence in CI-CD pipeline
• The performance tests on CI-CD pipeline should not take beyond few minutes
• What is that "few minutes"?
• Not prioritizing the importance of having the requirements for Performance Engineering

Recently, I was asked a question - How to evaluate the login performance of a mobile app using a tool "x"?

In another case, I see, a controller having all HTTP requests made when using web browser. Running these requests and trying to learn the numbers using a tool.

I do not say this is wrong way of doing.  That is a start.

But, we should NOT stay here thinking this is a performance engineering and that is how to run tests for learning a performance aspect[s].

To end, the performance is not just - how [why, when, what, where] fast or slow?  If that is your definition, you are not wrong!  That is a start and good for start; but, do not stick on to it alone and call performance.   It is capability.  It is about getting what I want in the way I have been promised and I expect; this is contextual, subjective and relative.  The capability leads to an experience.  What is that experience experienced?

Sometimes, serving the requests by what you call as slow, is a performance.  What is slow, here?

The words fast and slow are subjective, contextual and relative.  It is one small part of performance engineering.

That said, let me know, what have you been ignoring and unaware in practice of Performance Engineering & Testing?

Deep Link and its Testing via Automation

 

I get these question consistently from my fellow testers and community.

1. How to automate the mobile apps and web applications using Deep Links?
2. How to automate the business flows using Deep Links?
3. How to achieve end-to-end business flows testing on using Deep Links?
4. How to automate scenarios in mobile apps using Deep Links?
5. What is the best approach to automate the mobile apps using Deep Links?
6. What is the best practice to automate using the Deep Links?

And, more questions on same pitch.

No Deep Dive into - What is Deep Link?

A hyperlink in HTML is a kind of deep link within a website or to another website.

Deep Link is known with different names for web, Android app and iOS app.  All these names have the same understanding and intent at some point.

The Deep Links are URIs that takes me directly to a specific part (activity or fragment) of the app that I'm using or testing.  The Deep Link will have an intent which tells where I will be taken on using it.

When we converse on diving deep technically into testing and automation of Deep Link, will share more insights into its internals.

Deep Link and Challenges

This question is discussed with me often:

How to do end-to-end testing using the Deep Link?

Automation of a mobile app using Deep Link poses a challenge which is not experienced in web application.  

One such challenge is, say you have not installed the mobile app.  [This is solvable!]

• On using a Deep Link, I should be taken to Apple Store or Play Store based on the app.
• I have to install the app.
• Post this, in the traditional automation, I should start traversing the business work flows via GUI.
• Is this adding to the flakiness aspect of automation via GUI?

When we talk so much about flakiness and how to avoid (not prevent), should we exercise business workflows when automating using Deep Link?  What you are thinking?  Let me know!

Scoping of Automation Using Deep Link

Back to the fundamentals.

• We have to automate, no escape from it.  Let us automate what must be automated!
• Let us not fall into trap of "Automate everything!"
• For today, I'm in this mindset and attitude,
• What we automate depends on the objective or goal that we want to accomplish.
• Each test should have precise and deterministic goal.
• A test via automation is not an exemption to it.
• A test defined in automation should be precise, deterministic and have a single objective - Single Responsibility Principle.

What is the objective of my testing via automation for the Deep Link?  This define the scope and extent of my automation.  This will minimize the number of checks that I do using Deep Link.

The purpose of Deep Link is to take me to specific part of the mobile app.

• Should I start the end-to-end or exercising the workflow to be included in the Deep Link tests?
• If included, am I not complicating the testing via automation?

Automation using Deep Link

I ask this question to myself and to my team.

What is the goal of testing via automation using Deep Link?

This question helps me to pick minimal and necessity flow actions.   It has lead and leads me to define minimal tests for Deep Link based on what we want to learn from automation of same.

To me, the purpose of Deep Link is not end-to-end testing.  It's purpose is,

Am I taken to the intended state and data when used the Deep Link?

I have kept the test intent to this.

With this, I have come with tests that has minimal must evaluation and assertion to learn if the app is responding or not to the Deep Link.  This is what the business wants when the Deep Links are created.

The app usage and workflow function is not a problem statement of Deep Link in a general context.

Deep Link is not for end-to-end.  It is to take to you from a point to another point, that's it.

Are you automating using Deep Link?