A growing number of companies are choosing to integrate Kafka with their micro-services applications, taking advantage of its high performance, fault-tolerant, publish-subscribe event streaming framework. Developers embrace the ability to build distributed applications rapidly and Kafka’s efficient use of hardware helps to keep costs down.
However, the advantages from a development and operational perspective come with some challenges around testing. Designing and executing meaningful Kafka tests and covering multiple test objectives presents significant challenges for many testers and automation engineers.
Some of the most common challenges in testing Kafka based applications or Micro-Services applications integrated with Kafka are:
- Challenge 1 – Installation and configuration of the Kafka framework
- Challenge 2 – Setting up a fit for purpose Kafka test environment
- Challenge 3 – Establishing meaningful tests around the Kafka Produce / Consume functions
- Challenge 4 – Establishing the CI CD pipeline for continuous Kafka testing
Challenge 1 – Installation and configuration of the Kafka framework
Establishing the Kafka framework can present a number of challenges, especially for solutions that aren’t built on managed / hosted services. Developers have to become familiar with the Kafka server as well as Zookeeper, which can be time consuming. For example, setting up a multinode Kafka cluster manually involves establishing the correct configuration of server.properties in terms of replication factors, partitions and a number of other parameters.
Challenge 2 – Setting up a fit for purpose Kafka test environment
Establishing a local Kafka testing environment is good practice as part of an optimised test approach. But this often is not straight-forward for an Automation Test Engineer due to the complex nature of the Kafka broker setup that is integrated with producing and consuming micro-services applications / databases.
Challenge 3 – Establishing meaningful tests around the Kafka Produce / Consume function
Testing challenges arise when we try to write a test that specifies which event to test, from which offset, when exactly to test and how to validate the Application logic. In Kafka, producers keep on producing to the latest offset of a partition of the Kafka topic. In practice this means you can end up returning a test result that is different to the defined test scenario.
Challenge 4 – Establishing the CI-CD pipeline for continuous Kafka testing
DevOps teams can run into challenges in setting up an efficient CI-CD pipeline. Making sure builds are repeatable and easily manageable can be time-consuming, in part due to the required security configuration (Kafka SSL, SASL_PLAINTEXT, SASL_SSL, and SSL). These configuration requirements are more difficult to setup locally compared to Kafka with PLAINTEXT configuration.
A Real World Problem
Zerocode were asked to implement a test setup around a data ingestion pipeline using various heterogeneous source databases’ with data being passed into a target BigData store via a number of MicroServices applications. The diagram below illustrates the overall event streaming setup.
There were a number of producer applications as well as Databases, writing the events to the relevant topic in the Kafka cluster. The consumer applications read the messages from the relevant topic, enrich the message and publish the enriched event to another relevant topic.
The challenges associated with establishing a robust test setup around Kafka are often met by investing significant time in understanding the Kafka framework and, as a result, trying to adapt existing practices and tools or writing custom code to meet test objectives.
In other words, the challenge or obstacle to good Kafka testing is often seen as a lack of knowledge around Kafka itself and overcoming this obstacle necessitates quick up-skilling.
The way in which Kafka servers work presents challenges for traditional test approaches. Developing knowledge of Kafka and then adapting existing tooling, or writing custom code, isn’t the answer. This will present other problems as custom code can become cumbersome, often lacks proper code review / unit-testing and the code itself can get entangled with the test scripts, making the code and test scripts very difficult to maintain.
The problem is not a lack of Kafka knowledge–the real problem is not having a test approach and tooling that is appropriate for Kafka’s unique features.
Despite the obvious complexities with Kafka, developers and test automation engineers are left with few options but to resort to sub-optimal activities including,
Customisation
- Writing bespoke or custom Kafka Client code for producers, consumers
- Writing bespoke or custom HTTP Client code for MicroServices APIs
- Writing bespoke or custom Database Client code to deal with DBs (but most often Database validations are skipped by the test
- scenarios due to lack of tools)
- Writing boilerplate code to externalize the broker, producer, and consumer configurations
- Custom polling logic from Kafka topic consume the messages to fit the project requirement
- Custom retry logic for HTTP and Kafka in order to deal with REST or SOAP APIs and Kafka records
Manual Effort
- Manually parsing the result returned from the Kafka brokers after producing or consuming
- Writing message/records parsers to frame the message to produce it to the relevant topic
These activities are time consuming, subject to potential coding issues and establish an ongoing overhead that must be managed.
Best Practices
Approaching the test challenges around Kafka in the right way to begin with will lead to improved delivery timelines and create fewer ongoing management issues.
Avoiding delays to release
Trying to apply more traditional approaches to testing modern micro-service driven systems that are delivered through Agile methodologies creates a disconnect between development and quality assurance, that ultimately impacts on release timelines.
Teams that recognise this look at test process and tooling as part of the overall software delivery process before the inevitable issues arise. They benefit significantly from reduced delivery timelines, greater test coverage and fewer production issues that divert resources. Ultimately the operating model becomes far more efficient and shifting-left becomes a normal part of delivery.
Avoiding Technical Debt
If you contrast well thought through approaches to test automation with poor approaches you find that poor approaches tend to result in complex process and tooling (a complex operating model). Managing a complex operating model often results in sub-standard practices when trying to integrate the various components and processes. Decisions are taken in the interest of time rather than technical considerations, building a technical debt that must be managed.
Teams that have processes to consider and select the right test approach fare better in the longer term.
Maintaining Quality Assurance Standards
Ultimately the goal of quality assurance is to uphold quality standards. The more time spent establishing, managing and addressing issues with a sub optimal test setup will undermine the central objective of the quality assurance function. Having processes in place to consider the right approach pays dividends in terms of time and cost. For some organisations this is a mindset shift, however there are countless examples of organisations moving to a shift-left approach and reaping the benefits such an approach has to offer.
In developing the right test approach for Kafka we considered how to comprehensively test the usage of Kafka, across functional, security and load objectives. Our aim was to reduce the complexity of initial setup and keep test scenarios clean and separate from the underlying code, helping with ongoing management and integration of tests into CI-CD pipelines.
Pre-configured Kafka Service
We established a full Kafka runtime library in our test tool, Zerocode Samurai, allowing users to avoid the hassle of setting up the Kafka server configurations. This includes the support for all Kafka functional testing and security configurations.
JSON / YAML Declarative Framework
Samurai’s declarative JSON / YAML interface was introduced to allow users to simply establish test scenarios and payload. Using JSON / YAML enables rapid test scripting in a manner that is also more easily understood by other product stakeholders.
Re-usable Tests
This approach allows us to make tests reusable across test other test objectives and enables easier sharing of the tests across development and test teams. For example load tests can be defined using the functional tests.
Applying Samurai to a Real World Scenario
In the below Kafka setup we need to validate that the Application correctly enriches the message after consuming from the order-topic and writes (produces) the transformed message to the billing-topic .
We start by isolating the problems into independent user journeys (test scenarios). Then, single-node and multi-node Kafka brokers and Databases are spun up in docker via docker-compose to test the scenarios locally.
After this, we test and validate the data-ingestion flow from end-to-end i.e. starting from the source Database, passing through the Kafka topics, then the Micro-Services APIs, through to the target Hadoop BigData store. Once tests are passed locally, the code to trigger CI-CD pipeline is pushed, achieving a repeatable build.
Using the declarative approach in Samurai, it’s very easy to validate this as an integration test.
Step-1:
- Produce “Order No. 123“ to the order-topic
- Validate/assert the Acknowledgment from the Kafka broker
- That the message has landed in an Offset of a Partition
Step-2:
- Consume from the billing-topic
- Validate/assert that the message received was “Order No. 123, Card No. 456“
A sample scenario-step to produce to a Kafka topic looks like below in JSON format.
{
"name":"produce_message",
"url":"kafka-topic:order-topic",
"operation":"PRODUCE",
"request":{
"records":[
{
"key":"${RANDOM.NUMBER}",
"value":"Order No. 123"
}
]
}
}
Then to validate that the message has been successfully produced to the relevant topic, we add a verify section as below.
"verify":{
"status":"Ok",
"recordMetadata":{
"topicPartition":{
"partition":0,
"topic":"order-topic"
}
}
}
We can make this behaviour repeatable which seamlessly fits into the CI-CD pipeline
Step Definitions
We will switch the steps into YAML format to keep them concise.
Step-1:
Produce
name: produce_message
url: 'kafka-topic:order-topic'
operation: PRODUCE
request:
records:
- key: '${RANDOM.NUMBER}'
value: Order No. 123
verify:
status: Ok
recordMetadata:
topicPartition:
partition: 0
topic: order-topic
Step-2:
Consume
name: consume_message
url: 'kafka-topic:billing-topic'
operation: CONSUME
request: {}
assertions:
size: 1
records:
- key: $IS.NOTNULL
value: 'Order No. 123, Card No. 456'
We can further automate this as below if we need to read the message from a particular Partition and Offset of the Kafka topic. Below “billing-topic,0,0” means “billing-topic,partition,offset“.
request:
consumerLocalConfigs:
seek: "billing-topic,0,0"
End-to-end Testing
We have covered testing of the individual Kafka functions, but what about extending tests to incorporate Micro-Service(s) APIs as well as the records to-and-from Kafka topics? For example, to intercept the message from a particular partition and offset.
Again, this is achieved quite easily using Samurai.
To keep it simple, we break down the above scenario into 3 steps
Step-1:
- POST the message payload to Micro Service 1 /api/v1/orders REST API or SOAP
- Verify that we receive a 201 (relevant status code) and a resource ID (e.g. an orderId)
Step-2:
- Intercept the message from the Kafka billing-topic
- Verify that the message sent via the POST API has been enriched (e.g. with Credit card details)
Step-3
- GET the message payload from Micro Service 2 REST API /api/v1/invoices/orders/{orderId}
- Verify the invoice details to make sure the transaction completed (e.g. Invoice details)
Then we can connect or chain the steps to get an automated test scenario like the below and run it using Samurai runner.
scenarioName: Validate an order processing scenario with API and Kafka @@Dave
steps:
#Step -1
- name: place_order
url: /api/v1/orders
operation: POST
request:
headers:
Content-Type: application/json
body:
orderItem: Digital Piano
assertions:
status: 201
body:
orderId: 123
#Step -2
- name: consume_billing
url: 'kafka-topic:billing-topic'
operation: CONSUME
request: {}
assertions:
records:
- key: $IS.NOTNULL
value: 'Order No. 123, Card No. 456'
#Step -3
- name: get_invoice
url: '/api/v1/invoice/orders/${$.place_order.response.body.orderId}'
operation: GET
request:
headers:
Accept: application/json
assertions:
status: 200
body:
invoiceId: $IS.NOTNULL
orderId: $.place_order.response.body.orderId
orderItem: $.place_order.request.body.orderItem
Defect-free services?
Achieving defect-free services at optimal cost is the aim of product, test and IT management functions when it comes to quality assurance. As we have demonstrated above, using the right test approach, including process and tooling, can significantly impact on the ability to achieve increasingly defect-free services, speed up deployment and control the complexity of ongoing management.
Moreover, organisations that do step back to reconsider their approach to quality assurance are seeing the path to defect-free services through shift-left approaches to test, simplifying their operating models and ultimately spending more productive time on core quality assurance activity.
* The example we worked through in this article was achieved using the Zerocode Samurai tooling. Download a Free Trial of Samurai today.
Share this blog
