10 years ago
Quite a while ago, I gave a presentation on contract testing at an internal conference, using Spring Cloud Contract as an example. A colleague of mine had presented another tool in the area, Pact, at the previous internal conference.
Was it actually 10 years ago? Looking back at the GitHub repo, it turns out that it was 2017. Oh well, 10 is a nice round number.
Now, quite a few years later, I am about to start an assignment at a new company. When interviewing for the role, I found myself thinking that contract testing might be useful here. However, since this project uses Kafka, I wondered if there was support for it. And what about other asynchronous message-passing methods? When I did the original presentation, only synchronous REST-style communication was supported, as far as I remember.
A few days later, one of my new colleagues described a situation at his current assignment where they are trying to decompose their existing system into independent services. Their problem was that they did not feel confident deploying unless they tested end-to-end with several different scenarios. If I understood him correctly, this company had seven different testing environments—not counting QA, staging, and production. That is a record for me, at least. He also added that owners of downstream systems objected to upstream systems being updated and deployed without end-to-end testing.
All of this led me to take a renewed look at contract testing. Is it still a thing, and how has tooling evolved?
What is the problem?
It’s always a people problem — Gerald Weinberg
Let us start by taking one step back: what is the problem we are trying to solve?
Let us assume we have a service A, that is used by two clients, B and C, where the consumers use slightly different features in the API that A provides. These services are owned by different teams.
Now consider the situation where the team that owns Service A decides to release a new version to provide new capabilities to Service C. Imagine they accidentally remove a field in a message used by Service B. This might very well cause an unexpected error in Service B and lessen trust in the team. Other examples of changes that potentially break a consumer are fields becoming mandatory or changes in default values.
One solution to these kinds of problems is to do end-to-end testing. However, if we could confidently decouple the deployment of services, it would reduce the need for end-to-end testing that requires expensive and brittle testing environments and the need for synchronization between teams. We want to be able to change and deploy service A as long as it does not break its contract with the downstream service B.
Enter contract tests!
The copy+paste approach
One very pragmatic approach is to create consumer-specific test suites with the provider service. As a team owning a downstream service, you could donate tests and temporarily have a member work with the team owning the upstream service. The obvious drawback is that the tests need to be updated manually and could become outdated. Also this probably introduces some duplication between the consumer and provider. Still, this could be a good place to start; there is no need to bring in additional tools, and the conversation between the teams could be a way to build trust.
Bringing in a tool
Contract testing tools aim to fix the issue of keeping contract tests aligned between consumers and providers. For example, Pact uses mocking in the client service tests to produce and publish an artifact. This artifact captures the expectations of the consumer and is then used to test the provider.
Stubborn Contract uses a similar method to define the contract on one side only and then make it available to the other side.
The commercial version of Pact also introduces something they call bidirectional contract testing. The idea is to verify the API specification against the contract files when the provider service is updated.
Event-based communication
All of the tools I have looked at have added support for event-based communication, such as services communicating via a Kafka queue. The OpenAPI initiative has also added AsyncAPI specifications. I have not seen a lot of use of it in the wild, but OpenAPI specifications are certainly useful, so hopefully, they can facilitate documentation of event-based APIs.
Workflows
The Specmatic offering brings in support for Arazzo specifications in addition to OpenAPI and AsyncAPI. Arazzo specifications (https://www.openapis.org/arazzo-specification) from the OpenAPI initiative aim to capture workflows rather than isolated endpoints or events.
Non-functional testing
Contract testing has mostly been concerned with the functionality of the system. Specmatic claims to have features that could simplify testing non-functional aspects of your system.
Integration efforts
As always, when you bring in a new tool, some effort has to be put into integrating it into your environment and getting to grips with its quirks.
From what I have seen, automated tests can be a hurdle for many organizations, and having to maintain additional infrastructure adds to that burden.
Trade-offs
So what about draw-backs? For example, in order to run the tests in isolation they will require some scaffolding. The test scaffolding will make some assumptions about the environment so we need to put some effort into verifying that these assumptions still hold in some other type of tests.
Tools
Pact https://pact.io/
Open source. Features a proprietary contract broker for publishing contract files, Pact JSON. Supports event-based services such as Kafka, RabbitMQ, and ActiveMQ. Polyglot, with support for Java, Python, Go, TypeScript, and other languages.
Pactflow https://pactflow.io/
The commercial version of Pact adds support for bidirectional contract testing. They have made a series of short videos explaining the reasoning behind contract tests.
Stubborn Contract (formerly Spring Cloud Contract) https://stubborn.sh
Open source. Spring Cloud Contract has been transferred to the Stubborn Contract project. Originating from the Spring ecosystem, it is fairly tightly tied to Java. It has limited support for multiple languages but has added support for asynchronous contracts, such as Kafka and RabbitMQ.
Specmatic https://specmatic.io/
Specmatic is a freemium product. It uses a low code approach that puts the OpenAPI specification at the center. It also features support for Arazzo specifications in addition to OpenAPI and AsyncAPI.
In summary
Contract testing is still around. It enables a left-shift (or down-shift in terms of the test pyramid) of testing dependencies between services. We can catch a substantial part of problems with cheaper and less integrated tests reducing the need of synchronization and improving the feedback loop. The tooling I tried 10 years ago is alive and well and has been extended to support asynchronous communication mechanisms.
