Compatibility Checks (breaking change detection)
When contracts are published to PactFlow, the consumer contract (a pact file) will be verified against the Open API Specification (OAS). PactFlow ensures the consumer contract is a subset of what is in the OAS. In other words, check that all interactions in the Pact file are valid for that OAS.
PactFlow can only decide based on the information it receives. If the consumer contract does not contain all the interactions the consumer uses, then the resulting checks may say it is safe to deploy when it could not be. This is because the missing API calls may have breaking changes.
A tabbed detail screen displays the contracts involved and the verification status for the interaction.
There are three conditions that could result in an invalid (or failed) integration:
- The provider self-verification test results indicate failure. This means that the provider build ran some tests against the OAS and the result was a failure.
- The consumer Pact verification results failed. This means the Pact consumer tests failed, and the consumer is not compatible with the published Pact file.
- The consumer Pact file is not compatible with the OAS.
Compatibility checks
For the specific validations and checks performed, refer to OAS features and testing.
Interpreting verification result failures
Verification results are automatically pre-generated when a consumer contract is published against a number of common provider versions (such as deployed versions). They are also generated dynamically when can-i-deploy
is invoked for a given set of application versions and target environments.
Compatibility are visible from the user interface, in the API and in the output of can-i-deploy
.
User Interface
Compatibility results are listed on the detail view page, grouped by the relevant API resource in the OpenAPI document. They are also on the consumer contract tab, grouped by the interactions defined in the consumer contract.
Can I Deploy
When can-i-deploy
is called, you will get a table of results that shows which applications are compatible and the verification URL which explains the results.
Output from can-i-deploy
CONSUMER | C.VERSION | PROVIDER | P.VERSION | SUCCESS? | RESULT#
-------------------------------|--------------------|--------------------------------|--------------------|----------|--------
pactflow-example-consumer-w... | 5785fb8+1622544123 | pactflow-example-provider-r... | 7f3d83f+1622544125 | true | 1
VERIFICATION RESULTS
--------------------
1. https://testdemo.pactflow.io/hal-browser/browser.html#https://testdemo.pactflow.io/contracts/provider/pactflow-example-bi-directional-provider-restassured/version/7f3d83f%2B1622544125/consumer/pactflow-example-bi-directional-consumer-wiremock/pact-version/b421f8d1c0691e8304492c716e546427c4267c7f/verification-results (success)
Clicking on the verification results will take you to the (API) resource in PactFlow and show you the detailed analysis.
In some cases, results may not yet have been generated and entries in the above table will be unknown. To address this, you should consider polling via the --retry-while-unknown
and --retry-interval
flags. See this blog post for more.
API Resources
Response Object
summary
Whether or not the verification was successful
crossContractVerificationResults
This element contains the results of comparing the mock (pact contract) to the OpenAPI specification
providerContractVerificationResults
This contains the results of the provider verification, including the tool used to verify it, whether the test passed or failed and the base64 encoded OAS contract.
Successful result
{
"summary": {
"success": true
},
"crossContractVerificationResults": {
"success": true,
"results": {
"errors": [],
"warnings": []
},
"verificationDate": "2021-06-01T10:42:30.980+00:00",
"verifier": "pactflow-swagger-mock-validator",
"verifierVersion": "10.0.0"
},
"providerContractVerificationResults": {
"success": true,
"content": "dGVzdGVkIHZpYSBSZXN0QXNzdXJlZAo=",
"contentType": "text/plain",
"verifier": "verifier"
},
"_links": {
"self": {
"title": "Cross contract and Provider Contract verification results",
"href": "https://testdemo.pactflow.io/contracts/provider/pactflow-example-bi-directional-provider-restassured/version/7f3d83f%2B1622544125/consumer/pactflow-example-bi-directional-consumer-wiremock/pact-version/b421f8d1c0691e8304492c716e546427c4267c7f/verification-results"
}
}
}
Failure result
{
"summary": {
"success": false
},
"crossContractVerificationResults": {
"success": false,
"results": {
"errors": [{
"code": "request.body.incompatible",
"message": "Request body is incompatible with the request body schema in the spec file: should NOT have additional properties",
"mockDetails": {
"interactionDescription": "POST_/products_7436b06b-b387-4535-9d3a-da149d9826ba",
"interactionState": "[none]",
"location": "[root].interactions[0].request.body",
"mockFile": "pact",
"value": {
"id": "27",
"name": "pizza",
"type": "food",
"price": 27
}
},
"source": "spec-mock-validation",
"specDetails": {
"location": "[root].paths./products.post.requestBody.content.application/json.schema.additionalProperties",
"pathMethod": "post",
"pathName": "/products",
"specFile": "oas",
"value": false
},
"type": "error"
}, {
"code": "response.body.incompatible",
"message": "Response body is incompatible with the response body schema in the spec file: should NOT have additional properties - price",
"mockDetails": {
"interactionDescription": "GET_/products_646e1d83-da87-4155-9a43-3b24a2014cf3",
"interactionState": "[none]",
"location": "[root].interactions[1].response.body[0]",
"mockFile": "pact",
"value": {
"name": "pizza",
"id": "10",
"type": "food",
"price": 100
}
},
"source": "spec-mock-validation",
"specDetails": {
"location": "[root].paths./products.get.responses.200.content.application/json; charset=utf-8.schema.items.additionalProperties",
"pathMethod": "get",
"pathName": "/products",
"specFile": "oas",
"value": false
},
"type": "error"
}],
"failureReason": "Mock file \"pact\" is not compatible with spec file \"oas\"",
"warnings": []
},
"verificationDate": "2021-06-02T01:27:54.815+00:00",
"verifier": "pactflow-swagger-mock-validator",
"verifierVersion": "10.0.0"
},
"providerContractVerificationResults": {
"success": true,
"content": "dGVzdGVkIHZpYSBSZXN0QXNzdXJlZAo=",
"contentType": "text/plain",
"verifier": "verifier"
}
}
In the case of a failure, the following elements of the error
are most helpful in diagnosing the problem:
message
Summary of the problem. In most cases, you can understand the problem immediately. As above, you can see there are two errors - one for the request body, and another for the response body. In both cases (the request and response body), there is an additional unexpected property
price
expected by the consumer.mockDetails
Contains details of the Consumer Contract (mock) that are problematic, including the path to the interaction in the contract and the request/response details.
specDetails
Contains details of the Provider Contract (spec) that are problematic, including the path to the component of the resource in the OpenAPI specification the mock is incompatible with.
Contract Compatibility Errors
All errors and warnings are written from the consumer's perspective, referencing a "spec file" (the OpenAPI Document).
For example, if the consumer calls an unknown endpoint /products/10
, the error code
will be:
request.path-or-method.unknown`
and the corresponding message
:
Path or method not defined in spec file: GET /products/10
All errors contain 3 major components to aid with debugging:
Component | Description |
---|---|
Code | The category of error or warning discovered (see table). Warnings do not fail the compatibility check, but indicate a potential misunderstanding. Warnings are displayed alongside errors in the user interface and API resources. |
Message | A description of the violation, usually in the form of a JSON Schema validation error (see table) |
Interaction Path | A JSON-path like syntax that can help you reference the specific property in the interaction, within the consumer contract that is incompatible with the provider contract, including the errant value to review |
Error Codes
A code
describes the category of the problem, such as an incorrect header or an unexpected property.
This table describes error codes, descriptions and general advice to resolve them. Note that spec
is the terminology used to refer to the provider contract/OpenAPI Document. Interaction
is used to refer to the problematic interaction in the pact file.
Error Code | Type | Description | Fix |
---|---|---|---|
request.accept.incompatible | Error | The Accept header in the interaction does not match any of the mime-types in the OpenAPI document | Check your consumer test to ensure the expected mime type matches an acceptable mime type in the provider contract |
request.accept.unknown | Warning | There is an Accept header in your interaction, but the OpenAPI Document does not return any content. | This is a redundant header that should be removed. If there is an expected body, this is likely to be an error. |
request.authorization.missing | Error | The interaction lacks an Authorization query or header, but is required by the spec file. | Update the pact test to ensure it has an appropriate authorization scheme. |
request.body.incompatible | Error | The request body in the interaction is incompatible with the request body schema in the spec. | Review the JSON schema validation message. |
request.body.unknown | Warning | No matching schema could be found for the request body | Your API Provider does not expect a request body, or the request body does not match one of the allowed schemas (for example, in a oneOf clause. Check if a request body is required) |
request.content-type.incompatible | Error | Request Content-Type header is incompatible with the mime-types the spec accepts to consume | Confirm that your pact test is sending the correct content type |
request.content-type.missing | Warning | Request content type header is not defined but spec specifies mime-types to consume | It's possible your provider will send a mime type your consumer doesn't expect if it produces more than one mime type. You should explicitly set the Accept header in your pact test. |
request.content-type.unknown | Warning | Request content-type header is defined but the spec does not specify any mime-types to consume | The request body is redundant. Check if the spec should be accepting one, or remove it from your Pact test |
request.header.incompatible | Error | The interaction request header is incompatible with the spec | Review the JSON Schema validation message |
request.header.unknown | Warning | Request header is not defined in the spec file | Remove the redundant header or ensure it is defined in the spec |
request.path-or-method.unknown | Error | The Path or method used in the interaction is not defined in spec file | Check the correct resource is being used |
request.query.incompatible | Error | The request query in the interaction is incompatible with the spec | Review the JSON Schema validation message |
request.query.unknown | Warning | The query parameter in the interaction is not defined in the spec file | Review if the query parameter is valid. |
response.body.incompatible | Error | The response body in the interaction is incompatible with the spec | Review the JSON Schema validation message |
response.body.unknown | Warning | No matching schema was found for response body | Your API Provider does not return a response body, or the response body does not match one of the allowed schemas (e.g. in a oneOf clause. Check if a response body is required) |
response.content-type.incompatible | Error | Response Content-Type header is incompatible with the mime-types the spec defines to produce | Confirm that your pact test is consuming the correct content type |
response.content-type.unknown | Warning | Response content-type header is defined but the spec does not specify any mime-types to produce | The response body is redundant. Check if the spec should be returning one, or remove it from your Pact test |
response.header.incompatible | Error | The response header in the interaction is incompatible with the spec | Review the JSON Schema validation message |
response.header.unknown | Warning | Response header is not defined in the spec file | Remove the redundant header or ensure it is defined in the spec |
response.status.default | Warning | The interaction is using a response status code that is a default response in the spec file | Avoid default responses as they increase the ambiguity of possible valid response types |
response.status.unknown | Error | The expected response status code is not defined in the spec | Check the spec to see what valid status codes are returned. |
Error Messages
There are broadly three categories of error:
- Unknown - values that aren't defined in the spec, but referenced in the interaction.
- Missing - values that are required in the spec, but are missing in the interaction.
- Incompatible - values that are both defined in the spec and referenced in the interaction, but are incompatible.
For the ones deemed "incompatible", they will usually be accompanied by a detailed error message explaining the mismatch, which will be JSON Schema errors.
Error Path
Error paths use a JSONPath-like syntax with an associated value, to help you navigate from the error to the specific problematic component within the Pact interaction. The syntax is straightforward. Here are a few examples:
[root].interactions[0].response.body = {"id":"09","type":"CREDIT_CARD","name":"Gem Visa","price":99.99}
- the response body (with value{"id":"09","type":"CREDIT_CARD","name":"Gem Visa","price":99.99}
) expected in the first pact interaction is incompatible
- the
Access-Control-Allow-Origin
header (with a value of"*"
) in the first pact interaction
[root].interactions[0].response.headers.access-control-allow-origin = "*"
- theAccess-Control-Allow-Origin
header (with a value of"*"
) in the first pact interaction[root].interactions[1].request.query.id = "2"
- theid
query parameter (with a value of2
) in the 2nd pact interaction[root].interactions[3].request.path
- the request path in the third pact interaction
Common Error types
additionalProperties
When a pact test expects a response body, it may ask for a subset of what the provider can provide - this is perfectly acceptable. However it cannot ask for a property not present in the spec - this will cause failure - and is the most common error of this kind.
For example, if there is an expected property foo
in your pact file that does not match a schema in the OpenAPI Document, the following error will be displayed:
Response body is incompatible with the response body schema in the spec file: must NOT have additional properties - foo
In JSON Schema terminology, foo
is an "additional property" not defined in the schema. For a detailed tour of this topic, refer to this guide
The error you will receive will be: Request body is incompatible with the request body schema in the spec file: must NOT have additional properties - <property>
. The problematic property should be identified by name.
To correct this, the problematic property should be removed from the pact interaction, or supplied by the spec.
unevaluatedProperties
(allOf
)
When a pact test expects to send a request body or to receive a response body, the body must match any defined schemas.
If the allOf
keyword is used, we must treat all the schemas as a single composite schema. As per the additionalProperties
checks, if a property is expected that is not part of this composite schema, a similar error will be returned:
Response body is incompatible with the response body schema in the spec file: must NOT have unevaluated properties
To correct this, the problematic property should be removed from the pact interaction, or supplied by the spec.
Refer to the unevaluatedProperties
documentation of JSON Schema for more.