Skip to main content

Bi-Directional Contract Testing Guide

Introduction

Bi-Directional Contract Testing is a type of static contract testing where two contracts - one representing consumer expectations, and another representing the provider's capability - are compared to ensure they are compatible.

Teams generate a consumer contract from a mocking tool (such as Pact or Wiremock) and API providers verify a provider contract (such as an OAS) using a functional API testing tool (such as ReadyAPI). PactFlow then statically compares the contracts down to the field level to ensure compatibility.

Bi-Directional Contract Testing (BDCT) allows you to "upgrade" your existing tools into a powerful contract-testing solution. It simplifies adoption and reduces contract testing implementation time across your architecture.

BDCT is a feature exclusive to PactFlow and is not available in the Pact OSS project.

info

A note for Pact users: when contract testing with Pact, you need to write and maintain a separate set of (Pact) tests responsible for ensuring systems are compatible. The tests on the consumer side produce a consumer contract containing example scenarios to be supported for the consumer to work. These scenarios are then replayed against an actual running provider in a record and replay style interaction. If the Provider responds correctly, the contract is valid.

With BDCT, the key difference is that a Provider uploads its own provider contract advertising its full capability which is statically compared to the consumer contract expectations. The consumer contract is never replayed against the provider code base. This creates a simpler and decoupled workflow. See the trade-offs for more information.

BDCT is not intended to replace Pact testing, but to provide an alternative in cases where Pact may not be best suited.

Use Cases

Use CaseDescriptionHow Bi-Directional Contract Testing Help
Retrofitting contract-testing onto an existing systemWriting many Pact tests to cover all scenarios can be time-consumingBi-Directional Contract Testing lets you re-use existing tools to get greater coverage faster
API GatewaysPass through systems can be cumbersome to test with Pact (particularly handling state and mocking systems)Using specifications as the contract simplifies the approach
Internal APIs with large numbers of consumersMany consumers can make testing with Pact difficult, due to challenges with provider states and release co-ordinationBi-Directional Contract Testing is ideally suited as it decouples teams, and is particularly useful with fairly stable APIs (for example, Auth)
Testing against 3rd party APIsPact is not ideally suited to 3rd party APIs because 3rd parties are unlikely to validate your PactsBy pulling in the third party's API specification (OAS) regularly, you can continually ensure your consumers are compatible
Internal microservices following a contract-first approach (OpenAPI Specification)Contract testing is ideally suited to internal service-to-service testingBi-Directional Contract Testing can speed this process up by re-using the OAS as the provider contract
Web-based contract tests (for example, Cypress, MSW)Web applications use mocking/stubbing heavily, which could result in getting out of sync with the Provider API implementation. Further, using Pact from these tools can lead to challenges if not carefully implementedBi-Directional Contract Testing removes the need for additional Pact tests and the problem associated with too many interactions in a contract
Monolith to microservice migrationsFor example, gradually splitting of microservices using the "strangler" pattern, or to a completely new designContract testing helps you migrate from legacy applications to a modernised architecture by preventing breaking changes. When existing contract tests pass, you know it's safe to cutover.

Comparison to Pact

When to use

Here is an at-a-glance view of the key differences in approaches to achieve the benefits of contract testing.

This is not intended as an overall guide to a comprehensive testing strategy and ignores the differences in value of each approach outside the scope of contract testing.

info

Remember: Contract tests focus on the messages that flow between a consumer and provider, while functional tests also ensure that the correct side effects have occurred. For example, imagine an endpoint for a collection of /orders that accepts a POST request to create a new order.

A contract test would ensure that the consumer and provider had a shared and accurate understanding of the request and response required to create an order.

A functional test for the provider would ensure that when a given request was made, an Order with the correct attributes was actually persisted to the underlying datastore. A contract test does not check for side effects.

Read more on the aim and scope of contract tests.

TraditionalConsumer Driven Contract TestingBi-Directional Contract Testing
ApproachE2E testingConsumer driven contract testing

(for example, Pact, Spring Cloud Contract)
Specification or code generated contract tests

(for example, OAS, Postman Collections, Wiremock, Mountebank, GraphQL, Protobuf)
SummaryThe highest cost and maintenance, with the best guaranteesStrongest contract-testing outcomes but comes with a learning curve.Weaker guarantees than CDC but decouples teams and can be used with third parties/unknown consumers.
When to useTo supplement CDC and functional tests.Use case: Green fields projects.

Author: Developer/SDET

Context: When there is good buy-in from teams.

When code is accessible (white-box testing mode).

When stronger guarantees are warranted.
Use case:

  • Retrofitting contract-tests onto existing systems
  • API Gateways
  • 3rd Party API Testing
  • Contract first / API first workflows

Author: Anyone (Developers, Testers)

Context: When you can’t do Pact or code-based contract tests for example, black-box testing required or no access to code.

When you can BYO tools and extract further value from existing investments

Trade-offs

TraditionalConsumer Driven Contract TestingBi-Directional Contract Testing
E2E TestingContract test

(for example, Pact, Spring Cloud Contract)
Specification or code generated contract tests

(for example, OAS, Postman Collections, Wiremock, Mountebank, GraphQL, Protobuf)
Outcomes
Guarantees++++++++
Flakiness---++++++
Cost of maintenance---+++++
Learning curve++++-+
Team coupling---++++
Testing Properties
Isolation-++++
Complexity-++++
Test data setup-+++++
Testing data semantics+--
Feedback time-+++++
Stability-+++++
Reveal unused interfaces / fields-++++
Well fittedness-+++-
Unknown consumers+-+
understanding the trade-off table

More + is better, more - is worse. For example, E2E testing has the highest Guarantee but the worst with regards to Flakiness

How it works

In the following animation, you can see a high-level overview of the process.

info

The testing process may be initiated from either the consumer side (consumer driven) or the provider side (specification first)

Steps

Provider

  1. Provider starts with its specification (for example, an OpenAPI specification) called the Provider Contract. This may be created by hand or generated by code (for example, swagger codegen).
  2. The Provider Contract is tested against the provider, using a functional API testing tool (such as ReadyAPI, SoapUI, RestAssured, Dredd, or Postman) or generated by code (such as via Swashbuckle, Spring Docs).
  3. The Provider Contract is uploaded to PactFlow.
  4. When we call can-i-deploy the cross-contract validation process in PactFlow generates a Verification Result ensuring the provider doesn't break any of its consumers.
  5. If that passes, we deploy the provider and record the deployment via the pact-broker record-deployment command.

Consumer

  1. Consumer tests its behaviour against a mock (such as Pact or Wiremock).
  2. The Consumer Contract is produced, in the form of a pact file, that captures only the actual interactions generated by the consumer code.
  3. The Consumer Contract is uploaded to PactFlow.
  4. When we call can-i-deploy the cross-contract validation process in PactFlow generates a Verification Result determining if the consumer consumes a valid subset of the provider contract.
  5. If that passes, we deploy the consumer and record the deployment via the pact-broker record-deployment command.

Terminology

  • Consumer: An application uses the functionality or data from another application to do its job. For applications that use HTTP, the consumer is always the application that initiates the HTTP request (for example, the web front end), regardless of the direction of data flow. For applications that use queues, the consumer is the application that reads the message from the queue.

  • Provider: An application (often called a service) that provides functionality or data for other applications to use, often via an API. For HTTP applications, the provider is the application that returns the response. For applications that use queues, the provider (also called the producer) is the application that writes the messages to the queue.

  • A consumer contract is a collection of interactions that describe how the Consumer expects the Provider to behave. Each Consumer will have its own unique consumer contract for each Provider.

  • A provider contract specifies the Provider's capability. It will take the form of an OpenAPI document.

  • Cross-contract validation or contract comparison: the process by which PactFlow confirms that the consumer contract is a valid subset of a provider contract. For example, it will ensure that all request/responses defined in a pact file and valid resources and match the schemas in a provider OAS file.

Contract support

Pact

Consumer contracts must currently be specified in a pact format. You may use any tool, including Pact, to generate a pact file.

Read the documentation on using Pact as a consumer contract for more on how to convert mocks into the Pact format.

OpenAPI Specification

Provider contracts may be specified using an OpenAPI Specification.

Read the documentation using OpenAPI Specification as provider contract for more.