Your OpenAPI document says POST /v1/orders returns 201 with an order_id string. Your server returns 200 with orderId as an integer. The spec is published. The SDK is generated. The docs are live. And every consumer that trusted the contract is now wrong.
This is OpenAPI drift: the gap between the interface you publish and the interface you actually serve. It is not a documentation problem. It is a contract problem, and it belongs in the build.
What Drift Actually Looks Like
Drift is rarely a dramatic rewrite. It arrives in small, plausible changes that pass code review because they look harmless in isolation.
- Undocumented endpoints. A new route ships without a spec entry. Consumers cannot discover it, and SDK generation silently omits it.
- Missing or renamed fields. A response field is dropped or renamed during a refactor. The spec still advertises the old name.
- Type mismatches. A field declared as
stringstarts returning a number, or anintegerbecomes a string. The OpenAPI Specification v3.1.0 defines data types based on JSON Schema Draft 2020-12, so a type change is a schema change, not a cosmetic one. - Status code changes. A handler starts returning
200where the spec documents201, or a new422appears without a documented response. - Error shape drift. If your spec documents RFC 9457 problem details, a handler that returns a bare
{"error": "..."}is drift. RFC 9457 defines theapplication/problem+jsonmedia type and thetype,title,status,detail, andinstancemembers; a response that omits them is not the contract you published.
The OpenAPI Specification exists precisely to remove this guesswork. As the spec states, an OpenAPI definition “allows both humans and computers to discover and understand the capabilities of a service without access to source code, documentation, or through network traffic inspection.” When the server disagrees with the document, that promise breaks.
Two Kinds of Contract Testing, One Confusion
Teams often say “contract testing” and mean two different things. Naming the difference prevents a lot of wasted argument.
Provider contract testing checks that a provider’s actual behavior conforms to its documented contract, such as an OpenAPI document. This is the drift check. It answers: does the server do what the spec says?
Consumer-driven contract testing, as implemented by tools like Pact, checks that two applications share a documented understanding of the messages they exchange. Pact’s documentation describes it as “contract by example” — a collection of concrete request/response pairs generated from consumer tests, not a static schema describing all possible states.
These are complementary, not competing. Consumer-driven contracts catch the case where the provider is internally consistent but no longer meets a real consumer’s expectations. Provider contract testing catches the case where the provider has quietly stopped matching its own published spec. If your OpenAPI document is the source of truth for SDK generation and external documentation, you need the second kind. Most teams that complain about “drift” have only the first.
Diff the Spec, Then Diff the Server
There are two distinct checks, and conflating them is why drift detection feels unreliable.
Check 1: Spec-to-spec diff
Compare the OpenAPI document in the pull request against the one on your main branch. This catches intentional contract changes before they merge. OpenAPITools/openapi-diff compares two OpenAPI 3.x specifications and renders the difference as HTML, Markdown, AsciiDoc, or JSON. Its CLI supports --fail-on-incompatible to fail only when changes break backward compatibility, and --fail-on-changed to fail on any change at all.
That distinction matters. A new optional response field is compatible. A removed required request field is not. A changed enum value is usually not. You want the build to fail on the second category and warn on the first.
openapi-diff \
https://api.example.com/openapi.json \
./openapi.yaml \
--fail-on-incompatible \
--markdown diff.md
The --fail-on-incompatible flag is the one that belongs in CI. --fail-on-changed is useful for a stricter service where every contract change requires an explicit version bump, but it will generate noise on teams that iterate quickly.
Check 2: Spec-to-server diff
Spec-to-spec diffing cannot catch a handler that was changed without touching the spec. For that, you need to exercise the running server and compare its actual responses against the document.
Schemathesis generates property-based tests from an OpenAPI or GraphQL schema and runs them against a live server. It supports OpenAPI 2.0, 3.0, 3.1, and 3.2, chains operations into multi-step workflows, and produces a minimal curl reproducer for each finding. It integrates with pytest, GitHub Actions, and standard CI runners.
uvx schemathesis run https://staging.example.com/openapi.json \
--checks all \
--report junit.xml
Run it against a staging environment, not production. The point is to catch drift before the contract reaches consumers, not to generate load on live traffic.
What Should Fail the Build
Not every difference deserves a red build. A build that fails on cosmetic changes gets disabled within a month. Classify drift into three buckets.
Fail the build
- A documented endpoint is missing from the server.
- A required request field is removed or its type changes.
- A documented response status code is no longer returned.
- A response field documented as required is absent.
- An error response no longer conforms to the documented problem detail shape.
Warn, do not fail
- A new optional response field appears.
- A new endpoint is added without a spec entry.
- A description or summary changes.
- A deprecated field is still present.
Ignore
- Formatting, ordering, or whitespace in the spec file.
- Example values that differ from live data.
- Server URLs that vary by environment.
Write these rules down. Put them in the repository next to the spec. The classification is a team decision, not a tool default, and it should be reviewable when someone disagrees.
Keeping the Check Fast and Non-Flaky
The fastest way to kill a contract test is to make it slow or intermittent. Two rules prevent that.
Run spec-to-spec diff on every pull request. It is a file comparison. It takes seconds. There is no excuse for skipping it.
Run spec-to-server checks on a schedule and on merge to main, not on every commit. Schemathesis and similar tools need a running server. Spinning one up per pull request is possible but adds minutes and a new class of infrastructure flakiness. A nightly run against staging, plus a run on merge to main, catches drift within a day without slowing down the inner loop.
If you do run server checks per pull request, scope them. Hit a representative subset of endpoints, use a dedicated test environment with seeded data, and set a hard timeout. A contract test that hangs is worse than no contract test.
The Artifact: A Drift Gate You Can Adapt
Here is a minimal CI configuration that implements the two-check approach. It assumes a Node.js service, but the structure translates to any stack.
# .github/workflows/contract-drift.yml
name: Contract drift
on:
pull_request:
paths:
- 'openapi.yaml'
- 'src/**'
schedule:
- cron: '0 6 * * *'
jobs:
spec-diff:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Diff spec against main
run: |
git fetch origin main --depth=1
git show origin/main:openapi.yaml > /tmp/openapi-main.yaml
npx @openapitools/openapi-diff \
/tmp/openapi-main.yaml openapi.yaml \
--fail-on-incompatible \
--markdown /tmp/diff.md
- name: Comment diff on PR
if: always()
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const body = fs.readFileSync('/tmp/diff.md', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '### OpenAPI diff\n' + body
});
server-check:
if: github.event_name == 'schedule' || github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Schemathesis against staging
run: |
uvx schemathesis run \
https://staging.example.com/openapi.json \
--checks all \
--report junit.xml
- name: Upload report
if: always()
uses: actions/upload-artifact@v4
with:
name: schemathesis-report
path: junit.xml
Adapt the paths, the staging URL, and the failure thresholds to your service. The structure is the point: a fast spec diff on every pull request, a slower server check on a schedule, and a visible artifact when either fails.
What This Does Not Solve
Contract tests catch drift between the spec and the server. They do not catch drift between the spec and what consumers actually need. A perfectly enforced contract can still be the wrong contract.
They also do not replace integration tests. A contract test confirms that a response matches a schema. It does not confirm that the response contains the right data for a given business scenario. Use both.
And they do not fix the underlying cause of drift: a spec that is maintained separately from the code. The most durable fix is to generate the spec from the code, or generate the code from the spec, so that there is only one artifact to keep in sync. Contract tests are the safety net for teams that cannot yet do that. They are not a substitute for it.
FAQ
Do I need both spec-to-spec diffing and server checks?
Yes, if you want to catch both intentional and accidental drift. Spec-to-spec diffing catches deliberate contract changes before they merge. Server checks catch handlers that changed without a spec update. Neither replaces the other.
Can I run Schemathesis against production?
You can, but you should not. Schemathesis generates edge-case requests, including malformed inputs. Run it against a staging environment that mirrors production. The goal is to catch drift before it reaches consumers, not to stress live traffic.
What if my spec is generated from code?
Then spec-to-spec diffing still works, and it becomes a check on the generated output. Server checks are still useful because they verify that the running server matches the generated spec, which catches deployment and configuration drift that a code-level diff would miss.
How do I handle a legitimate breaking change?
Version the endpoint. Add the new behavior under a new path or a new version prefix, keep the old one working for a deprecation window, and update the spec to document both. The build should fail on an unversioned breaking change, not on a versioned one.
Does this work with GraphQL?
Schemathesis supports GraphQL schemas, but the drift problem is different. GraphQL has a single endpoint and a typed schema, so drift usually appears as a resolver that returns a shape the schema does not describe. The same principle applies — compare the running server against the published schema — but the tooling and the failure modes are not identical to REST.
Sources
- OpenAPI Specification v3.1.0 — definition of the OpenAPI document, data types, and the purpose of the specification.
- RFC 9457: Problem Details for HTTP APIs — the
application/problem+jsonmedia type and the problem detail object members. - Pact documentation — definitions of consumer-driven and provider contract testing.
- OpenAPITools/openapi-diff — CLI flags and Maven plugin configuration for spec comparison.
- Schemathesis documentation — supported specifications, CI integration, and property-based test generation.













