Benchmarks & methodology

Where the numbers come from

Every figure we publish is either a CI-enforced floor or a measurement with a versioned harness behind it. Nothing on this site is an adoption claim, an estimate, or a peak cherry-picked from a lucky run. Each number below names its gate and links to the live source.

25,000+

automated tests

What it measures: The number of tests pytest collects across the full suite — unit, integration, security, workflow, and gate tests. We publish a round FLOOR, not a live count, so the number can only understate.
How it's produced: Reproduce with: pip install -e ".[dev]" && pytest --collect-only -q. The final line reports the collected total.
The gate that keeps it honest: scripts/check_badge_freshness.py runs inside the CI coverage job and fails the build in both directions: if the floor ever overstates the real count, or if the real count exceeds the floor by more than 5,000 (a stale floor is treated as a bug too).

Freshness guard script →Test suite →

94%+

test coverage, CI-gated

What it measures: The line coverage CI enforces on every merge — a raise-only ratchet, not a marketing average. Actual coverage sits at or above it by construction: a run below this number fails the build.
How it's produced: The coverage job runs pytest with --cov-fail-under=94 (a one-way valve: raised when main climbs, never lowered to make a red PR pass). Codecov additionally gates the project total and every patch at 85%. The live figure is on Codecov.
The gate that keeps it honest: Drift-guarded twice: a unit test fails CI if the configured gate is ever lowered, and another fails if this displayed number diverges from the gate the workflow actually enforces.

CI coverage gate (tests.yml) →codecov.yml →Live coverage on Codecov →

0.97

mean RAG faithfulness

What it measures: How well answers produced through attune-rag (the retrieval engine inside Attune) stay grounded in the retrieved source, scored on a 40-query golden set over N=20 runs. The published number is the measured mean: 0.97.
How it's produced: The evaluation harness, golden set, and scoring live in the attune-rag repository and run in its CI — the benchmark is versioned with the code it measures.
The gate that keeps it honest: The CI regression gate is locked at mean faithfulness ≥ 0.9686: a PR that drops the mean below that threshold fails the build. Note the distinction — 0.97 is the measured mean, 0.9686 is the locked floor beneath it.

Methodology decision record →attune-rag repository →

Found a number on this site that doesn't match the code? That's a bug — the repository is the source of truth.

← Back to overview