CICD Load Testing With Zerocode Samurai - Part 1

The opportunity for a better approach to automated load testing

In this blog we cover some of the basic challenges in setting up meaningful automated load testing and provide some practical examples of how to use modern tooling to increase efficiency and impact. It is part of a series on automated load testing that will be published over the coming weeks.

Load testing (also known as Stress testing or Performance testing) is all about validating an application’s response against expectations, when a specified number of users access the application simultaneously. Load testing is a crucial part of any test activity, helping to ensure an application can serve all of its users to the standards expected. Load tests are established using simulated or virtual users that are designed to replicate the typical user behaviours, locations and requests. Today, as with most software testing, organisations are increasingly looking to automate load testing.

However, without the appropriate approach and tooling it can be difficult to deliver meaningful load testing (let alone automated load testing), particularly for more complex applications. This exposes organisations to risks around application unavailability which inevitably impacts on profitability and partly drives the business case for proper investment in a robust approach to Load Testing.

Another consideration is that fact that many businesses are moving more of their operations to cloud based services that charge based on planned resource usage. Comprehensive and well executed approaches to load testing can bring better insights into what capacity is required and therefore what level of resources a business will need to purchase from a cloud provider. The data collected from automated load testing can also be utilised to fine tune applications.

Despite the critical need for well designed load testing and the opportunity for automating load testing, many organisations struggle to establish meaningful load testing, wasting time and money in the process. In approaching this task, developers, testers or automation engineers are often overwhelmed, and then take overly complicated paths in their load testing, or compromise on the scope of test.

Some of the most common challenges are discussed below.

Top issues faced in establishing load testing

There are many common issues that must be overcome in establishing a good operating model for automated load testing.

Issue 1 – Taking a short-termist approach to quality assurance
- Limited scope, time or budget to establish an effective load test setup, including people, process and tooling
- Incomplete testing strategy and test plan, leading to test results that fail to validate an application’s performance under load

Issue 2 – Not using fit for purpose tools
- Lack of understanding of the tools available and how to set them up
- Difficulty integrating load tests into the CICD pipeline
- Lack of resource availability for ramping up desired load generation
- Lack of result visualisation and analysis tools with drill-down or filtering capabilities

Issue 3 – Not recognising the need for up-skilling
- Lack of skillsets to set up the required target environment
- Inability to establish meaningful load test cases or test scenarios designs
- Steep learning curve for the test engineers to setup and execute load tests

Issue 4 – Using outdated approaches to load testing
- Thinking performance testing is “Always outside the scope of the development / IDE”
- Difficulty in randomising virtual user behaviours
- Difficulty in making external APIs available as mocks

Addressing the issues

With the right approach and tooling, many of the common issues can be overcome. Zerocode Samurai has been established to directly address many of the issues by providing a better overall approach for establishing and running load testing.

Step 1 – Introduce a Shift-Left mindset

Load testing scope should be discussed and agreed upon by the stakeholders early in the design and build process to be able to properly plan for the required test setup. This approach helps to tackle the all too frequent situation of compromising test activity based on the tools, skills and time available rather than establishing a proper quality driven approach to load testing.

Step 2 – Establish a comprehensive test plan / identify gaps

Load testing requires considerable preparation before executing tests in order to get meaningful results. In establishing a comprehensive test plan there are a number of basic steps to consider.

Determining which parts of the application / service need test coverage
Establishing how often the tests should be run
What external or dependent APIs need to be available

It’s good practice to involve wider team members including the Dev team, Test team, Technical BAs and Test Architects to come up with the best strategy. DevOps and ITOps also should be involved during the infrastructure setup brainstorming sessions to ensure automation and associated tooling considerations are taken into account.

Once the basic test plan has been established, a gap analysis can be undertaken on skillsets and tooling.

Step 3 – Review the tooling and skillsets required

Load testing is often considered separately to functional testing. Therefore, standalone load test tools with their own specific languages and setups are often considered. This approach introduces cost, both in terms of the tools themselves and the skillsets to run them.

There are significant benefits in considering an end-to-end approach to testing, and an integrated tool to manage multiple test requirements beyond performance testing. For example, working in a common language / skillset, re-usability of tests and simpler maintenance and deployment of tests are a few reasons.

It is a common misconception to think that load generation can only be achieved using external apps or independent tools in the market. Load generation can very easily be achieved using the IDE itself, in a simpler and more flexible manner. In fact, the test engineer can have more control over the load generation mechanism to satisfy the project requirements and goals using this approach.

Using an integrated IDE based approach also helps to avoid problems like randomising the payload in a nested hierarchy or in the input parameters to match their complex project requirements or acceptance criteria.

Another consideration in selecting the right test tooling is reporting. Load test results can often be difficult to interpret. Granular, step level results and drill-down capability into test results can help to quickly identify where tests are failing against acceptance criteria. This helps in extracting the precise logs (request, response, timestamp, virtual user info etc) of the problematic step so that the test engineer feed it back to the Development team without wasting much time.

Zerocode Samurai is built as an integrated tool, allowing re-use of functional tests for performance testing. Samurai is also built around a JSON / YAML framework which allows working in a single common language, with step level insights into failing tests. These features make establishing representative performance testing simpler and more cost effective.

Step 4 – Introduce the tools and skillsets

Having established the right tooling and associated skillsets for your organisation, the implementation activity starts.

Introducing new tools and up-skilling resources takes time (particularly if the organisation is fairly new to automated testing). introducing the tool, learning the setup, the scripting language and the operating model of the tool’s ecosystem to create and fire virtual users/load scenarios can take a long time so it is important to plan accordingly. Taking help from an expert external vendor such as Zerocode can help to significantly reduce the time taken to introduce the required change.

Step 5 – Establish meaningful tests

When introducing automated tests, organisations tend to focus on areas that are more straightforward to automate. For load testing it is possible, with tools like Zerocode Samurai, to simply reuse the existing functional test scenarios and feed them to the load generating engine. This means it is far easier to quickly generate representative tests that embody the logic of the functional tests.

This concept is important with APIs as, most of the time, APIs are never independent. This means they are broadly dependent on Databases, GraphQL, async messaging, also often real-time data streams, HDFS/Hadoop or Elastic Search etc. When using the right tooling it becomes easier to, establish complex scenarios with POST, PUT, GET, DELETE operations, another scenario with DB CRUD operation chained with HTTP APIs with meaningful payloads to validate the various integration points to measure the performance of your application.

Another consideration in establishing meaningful load tests, is reflecting the target setup of the production environment. For instance, If the production environment setup has an Elastic Load Balancer (ELB) and the performance environment is not setup with an ELB, then you might get a misleading throughput of the application performance. This is also true for database and application configurations.

The following would be clearly an incorrect set up to conduct a performance testing which might not yield an accurate throughput of an application.

	Config / Infra	LIVE / Production	Performance / Pre-Prod	Correct Setup
1	ELB present	Yes	No
2	CPU + RAM	4 core + 32 GB	~~1 Core + 2 GB~~
3	App, DB Geo location	London, London	~~Hongkong, London~~
4	ELB present	Yes	Yes
5	CPU + RAM	4 core + 32 GB	4 core + 32 GB
6	App, DB Geo location	London, London	London, London

Note: You must take other parameters into consideration such as n/w latency, auto-scaling desired state etc.

A further consideration is the external dependent APIs. Assuming all the external dependent APIs are performing as expected, the test engineer will be generating the load on the target server. To achieve this we need to simulate or mock the external APIs and make them available during the performance testing. The challenge is to write bespoke code to create the mocks of these external APIs, or use another tool to establish these mock services. There is further difficulty in making these mocks available for standalone/in-memory testing as well as for the CICD pipeline.

Using an integrated approach, like the approach used in Zerocode Samurai, helps users to remove a lot of the complexity and time in dealing with the above considerations.

Step 6 – Integrating into the CICD pipeline

Once the test strategy, test plan, tooling and skillsets are in place and a suite of tests has been established, a robust integration into a CICD pipeline can be established. With the right tooling, this should be a fairly straightforward, natural progression from the previous steps taken for load testing.

However, most of the standalone load testing Apps lack the ability to easily integrate into a CICD pipeline, hence the Shift-Left goal of the project is compromised at a late stage. In deciding on the right setup, it is important to check how the tool will support the ultimate transition into CICD to avoid a lot of wasted time and effort.

Zerocode Samurai is built to seamlessly transition between local load testing, cloud load testing and ultimately to continuous load testing as part of a CICD setup.

A Real World Use Case

A digital transformation project was asked to conduct load testing of the application detailed below. The initial load requirement for the first year of going LIVE was to process 60,000 digital applications/requests per hour to onboard the customers within a 2 day service level agreement (17 per second).

The example below shows the heterogeneous APIs communicating over various protocols in a microservices landscape involving REST API, SOAP methods, Database, API Gateways, JWT auth tokens, and Kafka etc.

Note that the application (App1), Database and Kafka cluster are located in different Availability Zones (AZ) (If the LIVE setup is different from this e.g. they are in the same AZ or close to each other, then the load testing environment is incorrectly setup.)

London
Hongkong
USA

The project wanted to perform regular execution of the performance test suite during the sprint cycles to ensure the NFR was upheld at each stage of development. Setting up in this way helped to prevent the leakage of any performance defects to higher environments.

Now, let’s dive into more detail on how to pick the scenarios in the first instance. Later we discuss how n/w latency, Geo locations, CPU cores, RAM and threads per core etc. affect the load testing.

Zerocode Samurai was used as the test tooling in the following example.

Step 1 – Define the Acceptance Criteria (AC) – The API needs to process 60,000 digital applications per hour

In this test case, you need to figure out how many API invocations in parallel are required per second, in order to analyse its throughput. The below step/scenario needs to be executed for a single user, then we need to parallelise the invocations.

Using Zerocode Samurai, the test can then easily can be fed to a load-generator to parallelise the invocations (discussed later).

With Zerocode Samurai, you can choose to write the single executable invocation in YAML or JSON format as below.

YAML

---
name: app_screening
url: "/api/v1/screen"
method: POST
request:
  body:
    fullName: Osama Obama
verify:
  status: 201
  body:
    outcome: NO HIT

JSON

{
  "name": "app_screening",
  "url": "/api/v1/screen",
  "method": "POST",
  "request": {
    "body": {
      "fullName": "Osama Obama"
    }
  },
  "verify": {
    "status": 201,
    "body": {
      "outcome": "NO HIT"
    }
  }
}

Then, you need to fit this into the performance test pack which we will see in the later sections.

Let’s save this test file in,

load_tests/create_screening_test.yml

Step 2 – Establish the resource requirements to launch 17 parallel users/sec firing this API to the target server

Usually, in low-end requirements like this, parallel users are simulated via the Operating System (OS) threads, where each thread represents a single virtual user. For the above use-case, a laptop or PC with 2 core CPU and 4 GB of RAM should be enough to parallelise the API invocations to launch up to 30 to 50 users easily. The beauty is, no expensive local infrastructure or tooling is required.

The below table provides a ballpark estimation of the number of parallel users that can be simulated given a CPU/core or RAM availability.

Linux/Ubuntu

CPU Core	Frequency(GHz)	Memory(RAM GB)	Users Range/sec	Users Range/min	Users Range/hour	Assumption
1	2.5GHz	2	9 – 11	54 – 66	3240 – 3960	Around 1.8GB of RAM, 99% of CPUs is free for load testing
2	2.3GHz	4	25 – 35	1500 – 2100	90000 – 126000	Around 3.8GB of RAM, 99% of CPUs is free for load testing

Windows

CPU Core	Frequency(GHz)	Memory(RAM GB)	Users Range/sec	Users Range/min	Users Range/hour	Assumption
2	2.5 GHz	4	25 – 35	1500 – 2100	90000 – 126000	Around 3.8GB of RAM, 70% of CPUs is free for load testing
4	2.7 GHz	16	50 – 80	3000 – 4800	180000 – 288000	Around 8GB of RAM, 70% of CPUs is free for load testing

Mac

CPU Core	Frequency(GHz)	Memory(RAM GB)	Users Range/sec	Users Range/min	Users Range/hour	Assumption
2	2.3GHz	4	25 – 35	1500 – 2100	90000 – 126000	Around 3.8GB of RAM, 99% of CPUs is free for load testing
4	2.5	16	28 – 40	1680 – 2400	100800 – 144000	Around 12GB of RAM, 99% of CPUs is free for load testing

As a ballpark figure, if your load requirement is around 10 parallel users or less per second, you can easily go for a machine with 1 core and 2 GB RAM. The point here is you can satisfy this AC by using your laptop (which could have a higher configuration than this).

For the CICD pipeline, you can go for a 1 core + 2GB RAM to save some running costs.

Similarly, for 25 to 35 parallel users per sec, you can choose 2 core and 4GB RAM safely both for local running and in CI CD pipeline.

Step 3 – Configuration for launching 17 parallel users per second

number.of.threads=17

and we need to launch this e.g. within 1sec

ramp.up.period.in.seconds=1

Next, we want to run/generate this load on the server for a certain amount of time. We can achieve this by running this setup in a loop. e.g. around 5min = 5 * 60sec = 300 or more (apply similar calculation to convert to seconds)

loop.count=300

The load generation setup is done like below.

@LoadWith("load_generation.properties") // See below the properties file
@TestMapping(testClass = APIScreenTest.class, testMethod = "testScreen")
@RunWith(SamuraiLoad.class) 
public class LoadTest {
     // No code goes here
}

The API functional test for a single invocation of the above scenario is setup as below

@TargetEnv("screening_api_server.properties")
@UseHttpClient(SslTrustHttpClient.class)
@RunWith(Samurai.class) 
public class APIScreenTest { 
     @Test
     @Scenario("load_tests/create_screening_test.yml")
      public void testScreen() throws Exception { /* No code goes here */ }
}

Below is the documentation for how to adjust the load parameters.

load_generation.properties

# You can enter as many threads to stimulate a load test. A single user is represented by each Thread. So if you wish
# to simulate a load test with 5 concurrent users then you need to enter 5 as the value for this property. A high end
# machine will be able to spawn more number of threads. To keep the consistent(or nearly consistent) gap between the
# threads, adjust this number with 'ramp.up.period.in.seconds' and the actual response time of the API end point.</em> number.of.threads=17 

# It indicates the time taken to create all of the threads needed to fork the requests. If you set 10 seconds as the
# ramp-up period for 5 threads then the framework will take 10 seconds to create those 5 threads, i.e. each thread
# will be at work approx 2 secs gap between the requests. Also by setting its value to 0 all the threads can be created
# at once at the same time to simulate a Spike.
ramp.up.period.in.seconds=1

# By specifying its value framework gets to know that how many times the test(s), i.e. the number of requests will be
# repeated per every 'ramp.up.period.in.seconds'.
# Supposing number.of.threads = n, ramp.up.period.in.seconds = y, loop.count = i
# then (n * i) = number of requests will be fired over (y * i) seconds.
loop.count=300

Step 4 – Randomizing the virtual user behavior or input parameters

Because we fire parallel virtual users, we should be making sure each of them is firing with a different kind of data/payload so that the processing becomes unique at the server end.

This also helps the target application not to use caching during the load testing

We can easily do that as below (you can randomise any field in various ways. See documentation).

--- 
name: app_screening 
url: "/api/v1/screen" 
method: POST 
request: 
  body: 
    fullName: Osama ${RANDOM.NUMBER} 
verify: 
  status: 201 
  body: 
    status: NO HIT

where

fullName: Osama ${RANDOM.NUMBER}

generates “Osama 123234432234″… “Osama 3983456546”

fullName: ${RANDOM.NUMBER} Obama

will generate “18912344123 Obama”… “597123456 Obama”

fullName: ${RANDOM.STRING:5} Obama

will generate “hello Obama”… “xyern Obama” etc.

Step 5: Test Execution

Now let’s execute the load test from :

The local laptop

mvn test -Dtest=LoadTest

To adjust the load generation parameters, you can simply pass it via the CLI as below.

CLI

mvn test -Dtest=LoadTest -Dusers=17 -Dduration=1 -Dloop=300

The Dev (Small Performance) environment via CICD jobs

mvn test -Dtest=LoadTest -Denv=dev

ramp up more users(if required):

mvn test -Dtest=LoadTest -Denv=dev -Dusers=30 -Dduration=1 -Dloop=300

Gradle

gradle test

CPU, memory utilization

You can simply use the “htop“ command to view the CPU, memory utilisation of your laptop or VMs.

The advantage here was one or more functional test cases were reused to cover load testing ACs. If different tools were used, then those would expect you to create load test scenarios from scratch, using their custom scripting, which is an additional overhead.

As explained above, no extra tooling or complicated programming was required to create load tests and load configurations. It was all done very easily and in a declarative manner using Zerocode Samurai.

The load test was easily run locally against a deployment inside the laptop and then with the environment switch, it was executed against various other environments from the laptop.

Also, you observed that the load parameters were easily adjusted via CLI parameters for transitioning and running in other relevant environments or in a CICD pipeline to satisfy the ACs.

It was very easy to ramp up, ramp down and create spikes for the performance requirements.

So that’s it! We covered a simple use case of a load testing requirement.

To be continued…

In the coming weeks, we will further discuss practical approaches to more complex load testing including throughput visualisation, medium range load testing and high end distributed load testing.

We will also demonstrate how Zerocode Samurai can be used to generate load for Database or Kafka clusters very easily, covering parts of testing that are often difficult to manage.

You can Create an Account to stay up to date with our latest content. you can also sign up for a free trial of Zerocode Samurai and try the above to see how simple load testing can be.

Load generation on database server

Load generation on Kafka cluster

Download a Free Trial of Samurai today.

Start Now

CICD Load Testing With Zerocode Samurai – Part 1