How to Test Your Documentation Like You Test Your Code

Your code has unit tests. Integration tests. End-to-end tests. A CI pipeline that runs them on every commit.

Your documentation has none of that.

It’s written in a text editor, reviewed by a human (maybe), merged, and left to drift. There’s no test that catches when an endpoint description no longer matches the code. No lint that flags a broken code example. No CI check that fails when a documented parameter gets renamed.

This is the testing gap. And it’s the reason documentation is always the least reliable part of any software product.

Let’s fix that. Not with process — with tests.

The Testing Pyramid for Documentation

Code testing has a pyramid: unit tests at the base, integration tests in the middle, end-to-end tests at the top. Documentation testing has a similar structure.

Level 1: Linting

The foundation. Fast, local, catches the obvious stuff.

Documentation linting checks for:

Formatting consistency. Are all code blocks properly fenced? Do all headings follow the same style? Are links properly formed?
Spelling and grammar. Not because your docs need to be literary, but because a typo in a field name (user_id vs usr_id) creates a broken code example.
Structure. Does every endpoint have a description? Does every parameter have a type? Are required fields marked?

Tools like markdownlint, vale, and write-good handle the basics. They won’t catch semantic drift, but they’ll catch the noise that makes docs look unprofessional.

# Example: markdownlint in CI
npx markdownlint-cli 'docs/**/*.md'

Run linting in your editor on save. Make it part of the pre-commit hook. It should be as automatic as formatting.

Level 2: Structural Validation

This is where documentation testing gets interesting. Structural validation checks that your docs are complete — that they cover everything they should.

For API documentation, this means:

Every endpoint has documentation. No undocumented routes.
Every parameter is documented. If the function takes three arguments, the docs describe three arguments.
Every response schema is documented. Not just the happy path — error responses too.
No orphaned documentation. Every doc section maps to an actual code construct. No docs for deleted endpoints.

This is where most documentation breaks down. A team adds a new required parameter. The code works. The tests pass. But the docs still show the old signature, and the first time anyone notices is when a user gets a 400 error.

Structural validation catches this automatically:

# boringdocs.yaml - structural rules
rules:
  - endpoint-must-have-description: error
  - param-must-match-signature: error
  - no-orphaned-docs: warning
  - no-orphaned-code: warning

Level 3: Semantic Validation

Structural validation checks that docs exist. Semantic validation checks that they’re right.

This is the hard part. It requires comparing documentation to code — not just checking that a description exists, but checking that the description matches reality.

Examples of semantic checks:

Type matching. The docs say user_id is an integer. The code expects a string. That’s a bug — in the docs.
Example accuracy. The code example shows POST /users with a name field. The actual API requires full_name. The example is wrong.
Response accuracy. The documented response includes a created_at timestamp. The actual response uses createdAt. The docs are lying.
Constraint documentation. The API requires amount to be a positive integer. The docs don’t mention this. Code generated from the docs will fail.

This is the layer that boringdocs operates at. It’s the layer that matters most, because semantic drift is the kind of drift that breaks integrations.

Level 4: Integration Testing

The top of the pyramid. Actually running the code examples in your documentation against the real API.

This sounds extreme. It’s not.

Tools like dredd, schemathesis, and optic can take your API documentation and automatically generate test requests. They hit your actual endpoints with the parameters described in your docs and check that the responses match.

# Dredd: test your API against your docs
dredd docs/api.apib http://localhost:3000

If your docs say GET /users/:id returns a name field, Dredd will call that endpoint and verify the response actually contains name. If the field was renamed to full_name, the test fails.

This is documentation testing at its most literal: does the documentation accurately describe what the code does?

Level 5: Continuous Validation

Linting is a one-time check. Structural validation runs on commit. Integration tests run in CI. But documentation drift is continuous — it happens every time code changes without docs changing with it.

Continuous validation means running documentation checks on every commit, in every environment, forever. Not as a separate step. Not as a quarterly audit. As infrastructure.

The workflow looks like this:

Developer pushes a commit that changes an API endpoint.
CI runs code tests. They pass.
CI runs documentation validation. It fails — the endpoint signature changed but the docs weren’t updated.
The developer sees the failure in the PR. They update the docs. CI passes. The merge goes through.

This is the same feedback loop code has had for years. Documentation just hasn’t had it until now.

What to Test First

If you’re starting from zero — no documentation tests at all — here’s the priority order:

1. Code examples. These are the highest-value target. Broken code examples directly cause integration failures. If you can only test one thing, test that your code examples actually work.

2. Endpoint coverage. Make sure every API endpoint has documentation. Undocumented endpoints are worse than bad documentation — they’re invisible.

3. Parameter accuracy. Check that documented parameters match the actual function signatures. This catches the most common form of drift.

4. Response schemas. Verify that documented response shapes match actual responses. This is especially important for AI agents that generate code based on your response examples.

5. Error documentation. Check that error codes and error response formats are documented. This is the most neglected area of API documentation and the one that causes the most debugging pain.

The CI Pipeline

Here’s what a documentation testing pipeline looks like in practice:

# .github/workflows/docs.yml
name: Documentation Tests
on: [push, pull_request]

jobs:
  docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Lint docs
        run: npx markdownlint-cli 'docs/**/*.md'

      - name: Validate structure
        run: boringdocs check --format github

      - name: Test code examples
        run: dredd docs/api.apib http://localhost:3000

Three steps. Minutes to run. Catches drift before it reaches your users.

The Mindshift

The biggest barrier to documentation testing isn’t tooling. It’s the belief that documentation is content — something you write, not something you test.

Code isn’t “content.” It’s a system with inputs, outputs, and behaviors. It has tests because it needs to work.

Documentation is the same. It has inputs (code changes), outputs (published docs), and behaviors (describing the system accurately). It needs tests for the same reason code does: because untested systems drift.

The teams that treat documentation as infrastructure — testable, versionable, continuously validated — will have more accurate docs, faster integrations, and fewer support tickets. The teams that treat it as content will keep wondering why their docs are always outdated.

The Bottom Line

Documentation doesn’t have a compiler. But it can have a test suite.

Start with linting. Add structural validation. Work toward semantic checks and integration tests. Run them all in CI. Make documentation accuracy a build requirement, not a nice-to-have.

Your docs are part of your product. Test them like one.

Documentation accuracy is code quality. Join the waitlist — boringdocs is the validation layer that keeps your docs in sync with your code, continuously. Because your docs deserve a test suite too.