Benchmarks & methodology

Where the numbers come from

Every figure we publish is either a CI-enforced floor or a measurement with a versioned harness behind it. Nothing on this site is an adoption claim, an estimate, or a peak cherry-picked from a lucky run. Each number below names its gate and links to the live source.

25,000+

automated tests

What it measures: The number of tests pytest collects across the full suite — unit, integration, security, workflow, and gate tests. We publish a round FLOOR, not a live count, so the number can only understate.
How it's produced: Reproduce with: pip install -e ".[dev]" && pytest --collect-only -q. The final line reports the collected total.
The gate that keeps it honest: scripts/check_badge_freshness.py runs inside the CI coverage job and fails the build in both directions: if the floor ever overstates the real count, or if the real count exceeds the floor by more than 5,000 (a stale floor is treated as a bug too).

Freshness guard script →Test suite →

85%

coverage floor, CI-enforced

What it measures: The minimum line coverage the project accepts — a floor the build enforces, not a marketing average. Actual coverage sits at or above it by construction.
How it's produced: Enforced identically in three places: pytest runs with --cov-fail-under=85 (pyproject.toml), and Codecov gates both the project total and every patch at 85%. One number everywhere — a PR that drops changed-code coverage below the floor cannot merge.
The gate that keeps it honest: The threshold itself is drift-guarded: a unit test fails CI if the configured coverage gate is ever lowered.

pyproject.toml →codecov.yml →Live coverage on Codecov →

0.97

mean RAG faithfulness

What it measures: How well answers produced through attune-rag (the retrieval engine inside Attune) stay grounded in the retrieved source, scored on a 40-query golden set over N=20 runs. The published number is the measured mean: 0.97.
How it's produced: The evaluation harness, golden set, and scoring live in the attune-rag repository and run in its CI — the benchmark is versioned with the code it measures.
The gate that keeps it honest: The CI regression gate is locked at mean faithfulness ≥ 0.9686: a PR that drops the mean below that threshold fails the build. Note the distinction — 0.97 is the measured mean, 0.9686 is the locked floor beneath it.

Methodology decision record →attune-rag repository →

Found a number on this site that doesn't match the code? That's a bug — the repository is the source of truth.

← Back to overview