[ { "variant": "baseline", "total_queries": 15, "precision_at_1": 0.7333333333333333, "recall_at_k": 0.8666666666666667, "mean_faithfulness": 0.938035113035113, "refusal_rate": 0.0, "hallucination_rate": 0.4666666666666667, "mean_generate_ms": 7756.4924997355165, "mean_judge_ms": 9451.551972275289, "runs": [ { "id": "gq-001", "query": "how do I run a security audit", "difficulty": "easy", "variant": "baseline", "answer": "## Running a Security Audit\n\nYou have a few ways to do this, depending on your setup:\n\n---\n\n### Option 1: Plain language (simplest)\n\nJust describe what you want:\n\n```\nscan src/ for security issues\n```\n\n---\n\n### Option 2: Use the skill directly\n\n```\n/security-audit \n```\n\nExamples:\n\n| Command | What it scans |\n|---------|---------------|\n| `/security-audit src/auth.py` | One file |\n| `/security-audit src/` | A directory tree |\n| `/security-audit .` | The whole project |\n\n---\n\n### Option 3: Use the CLI workflow\n\n```\nattune workflow run security-audit --path \"src/\"\n```\n\n---\n\n### What you get back\n\nResults are grouped by severity with file locations, line numbers, and CWE identifiers \u2014 for example:\n\n```\nCritical\n src/hooks/executor.py:89 \u2014 eval() on user input (CWE-95)\n\nHigh\n src/config.py:203 \u2014 Path not validated (CWE-22)\n```\n\n---\n\n### Narrowing the scope\n\nIf you want to focus on something specific rather than a full sweep, you can say things like:\n- *\"check for hardcoded secrets\"*\n- *\"look for injection vulnerabilities\"*\n- *\"scan dependencies for known CVEs\"*\n\n---\n\n### After the audit\n\n- Ask **\"fix the critical findings\"** to generate patches\n- Ask **\"write tests for the flagged files\"** to prevent regressions\n- Ask **\"deep scan src/auth/\"** for a more thorough review of a specific area\n\n*(Sources: `tasks/use-security-audit.md`, `quickstarts/skill-security-audit.md`, `quickstarts/run-security-audit.md`)*", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 18, "unsupported_claims": 0, "total_claims": 18, "generate_ms": 6469.77870806586, "judge_ms": 7677.700416999869 }, { "id": "gq-002", "query": "generate tests for my code", "difficulty": "easy", "variant": "baseline", "answer": "I'll help you generate tests! To get started quickly, I just need two things:\n\n## 1. Which file or module do you want to target?\n\nFor example:\n- A single file: `src/auth/login.py`\n- A whole directory: `src/auth/`\n- Your whole project: just say \"everything\" or point me to your root `src/` folder\n\n## 2. What approach do you want?\n\n| Approach | What you get | Time |\n|----------|-------------|------|\n| **Gap analysis** | List of untested functions ranked by risk | Seconds |\n| **Generate tests** | Working pytest files with edge cases | ~1\u20132 min per module |\n| **Both** | Gaps first, then tests for the top gaps | ~2\u20133 min |\n\n---\n\n**Tip:** If you give me both upfront, we skip the questions entirely and run immediately. For example:\n\n```\ngenerate tests for src/auth/\n```\nor\n```\n/smart-test src/auth/login.py\n```\n\n---\n\nOnce I know the target, I'll produce:\n- \u2705 A **coverage gap report** ranked by risk (high/medium/low)\n- \u2705 **Working pytest functions** with assertions\n- \u2705 **Edge cases** (empty inputs, `None`, boundary values)\n- \u2705 **Error path tests** for exception handling\n- \u2705 **Parametrized tests** where appropriate\n\n**What's the path to your code?**", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.9444444444444444, "supported_claims": 17, "unsupported_claims": 1, "total_claims": 18, "generate_ms": 6522.384957992472, "judge_ms": 9893.34512501955 }, { "id": "gq-003", "query": "fix failing tests", "difficulty": "easy", "variant": "baseline", "answer": "I can help you fix your failing tests. Based on the fix-test skill documentation, here's how to get started:\n\n## Quickest Path\n\nIf you know which test is failing, name it directly \u2014 this skips the setup questions and jumps straight to diagnosis:\n\n```\nfix tests/your_test_file.py\n```\n\nOr target a specific test:\n\n```\n/fix-test tests/your_test_file.py::your_test_name\n```\n\n---\n\n## To Help You Further, I Have Two Questions\n\nSince you didn't specify a target, the guided flow asks:\n\n1. **Which test is failing?**\n - A specific file (e.g., `tests/unit/test_auth.py`)\n - A specific test name\n - Or should it find all failures automatically?\n\n2. **When did it start failing?**\n - After a recent change (refactor, dependency upgrade, migration)?\n - Or has it been broken for a while?\n\nYour answers matter \u2014 for example, if it broke after a rename, the skill will prioritize **import errors** and **mock mismatches** first.\n\n---\n\n## What to Expect\n\nOnce a target is identified, the process runs automatically:\n\n1. **Diagnose** \u2014 runs the test, reads the traceback, classifies the root cause\n2. **Fix & retry** \u2014 applies a fix and re-runs, up to **3 attempts**\n3. **Report** \u2014 tells you what was fixed, what still fails, and what to try next\n\nThe root causes it can handle automatically include import errors, stale mocks, assertion drift, type errors, and missing fixtures *(per `concepts/tool-fix-test.md`)*.\n\n---\n\n**Which test should I look at?**", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 16, "unsupported_claims": 0, "total_claims": 16, "generate_ms": 8486.332791973837, "judge_ms": 7220.624000066891 }, { "id": "gq-004", "query": "review code quality", "difficulty": "easy", "variant": "baseline", "answer": "I'll help you run a code quality review. Before starting, I need two quick things:\n\n## 1. What should I review?\n\n| Option | Example |\n|--------|---------|\n| A specific file | `src/config.py` |\n| A directory | `src/auth/` |\n| The whole project | `.` |\n\n## 2. How deep should I go?\n\n| Depth | What it covers | Time |\n|-------|---------------|------|\n| **Quick** | Style and formatting only | Seconds |\n| **Standard** | + Logic errors, likely bugs | ~1 min |\n| **Deep** | + Security, architecture, test gaps | ~3 min |\n\nThe default is **standard** if you don't have a preference.\n\n---\n\n**Just tell me something like:**\n- *\"review src/auth/\"* \u2014 standard depth on a directory\n- *\"deep review of src/config.py\"* \u2014 full analysis on one file\n- *\"quick review of .\"* \u2014 fast style check on everything\n\nIf you give me both the path and depth upfront, I'll skip the questions and run immediately.", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 14, "unsupported_claims": 0, "total_claims": 14, "generate_ms": 4586.556916008703, "judge_ms": 7962.03841699753 }, { "id": "gq-005", "query": "deep review my PR", "difficulty": "easy", "variant": "baseline", "answer": "I can help you run a deep review on your PR! Based on the context, here are your options:\n\n## Run a Deep Review\n\n### Option 1 \u2014 Natural language (simplest)\nJust tell me which files or directory your PR touches:\n\n```\ndeep review of src/auth/\n```\n\nIf you give me the path upfront, it runs immediately without any follow-up questions.\n\n---\n\n### Option 2 \u2014 Use the skill directly\n```\n/code-quality src/your-pr-directory/\n```\n\nWhen prompted for depth, choose **Deep**.\n\n---\n\n### Option 3 \u2014 Use the `deep_review` tool directly\n*(from `references/tool-deep-review.md`)*\n\n```\ndeep_review(path=\"src/your-pr-directory/\")\n```\n\n---\n\n## What \"Deep\" includes\n*(from `tasks/use-code-quality.md`)*\n\n| What runs | Time |\n|-----------|------|\n| Style and formatting | |\n| Logic errors and likely bugs | |\n| **+ Security, architecture, test gaps** | ~3 min |\n\nDeep is the right choice for PR review of critical or pre-release code.\n\n---\n\n## What I need from you\n\n**Which files or directory does your PR change?**\n\nFor example:\n- A specific file: `src/auth/login.py`\n- A directory: `src/auth/`\n- The whole project: `.`\n\nGive me the path and I'll kick off the deep review right away.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.8571428571428571, "supported_claims": 12, "unsupported_claims": 2, "total_claims": 14, "generate_ms": 7638.177665998228, "judge_ms": 11818.90137505252 }, { "id": "gq-006", "query": "predict bugs before they happen", "difficulty": "medium", "variant": "baseline", "answer": "# Bug Prediction\n\nYou can scan your code for likely bug locations before they cause problems in production.\n\n## Quickstart\n\n```\n/bug-predict src/\n```\n\nOr just describe what you want in plain language:\n\n```\npredict bugs in src/\nwhere are bugs most likely in the workflows module?\nfind risky code in the auth package\n```\n\n---\n\n## What gets scanned\n\nThe scanner looks for three categories of risk (from `concepts/tool-bug-predict.md`):\n\n| Pattern | Severity | What it catches |\n|---|---|---|\n| `dangerous_eval` | **HIGH** | `eval()`, `exec()`, `compile()` on input \u2014 code injection vectors |\n| `broad_exception` | **MEDIUM** | Bare `except:`, unlogged `except Exception:` \u2014 silently swallowed errors |\n| `incomplete_code` | **LOW** | TODO, FIXME, HACK, XXX comments \u2014 unfinished paths that break under edge cases |\n\nBeyond patterns, it also weighs **cyclomatic complexity**, **change frequency** (hot files), and **code smells** like functions over 50 lines.\n\n---\n\n## Choose your scope\n\n| Command | What it scans |\n|---|---|\n| `/bug-predict src/auth.py` | One file |\n| `/bug-predict src/` | A directory tree |\n| `/bug-predict .` | The whole project |\n\n---\n\n## Example output\n\n```\nBug Prediction Report\nRisk Score: 73/100 | Files: 34 | Findings: 8\n\nHIGH (2 findings)\n src/hooks/executor.py:89 dangerous_eval eval() on user input\n src/plugins/loader.py:142 dangerous_eval exec() in plugin loader\n\nMEDIUM (3 findings)\n src/api/webhook.py:67 broad_exception bare except: masks errors\n ...\n```\n\nResults are sorted by severity with clickable file links to jump directly to each issue.\n\n---\n\n## Good next steps\n\n- **Fix HIGHs first** \u2014 ask *\"fix the dangerous_eval in executor.py\"*\n- **Generate tests** \u2014 ask *\"write tests for the flagged files\"*\n- **Go deeper** \u2014 `/bug-predict src/auth/` for a focused scan\n- **Track trends** \u2014 run weekly to see if risk scores improve or drift\n\nWant the full reference? Run `attune help-docs ref-skill-bug-predict`.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 30, "unsupported_claims": 0, "total_claims": 30, "generate_ms": 9266.656665946357, "judge_ms": 9977.185708004981 }, { "id": "gq-007", "query": "refactor my code", "difficulty": "medium", "variant": "baseline", "answer": "## Getting Started with Refactoring\n\nHappy to help! \"Refactor my code\" can mean a few different things \u2014 here's how to point me in the right direction.\n\n---\n\n### Option 1: Analyze a specific path for problems\n\nIf you want a structured analysis of what's wrong and a prioritized roadmap to fix it, run:\n\n```\n/refactor-plan \n```\n\nThis scans for issues across several categories (from `concepts/tool-refactor-plan.md`):\n\n| Category | Examples |\n|---|---|\n| **Code smells** | Long methods, god classes, feature envy |\n| **Duplication** | Copy-pasted blocks, DRY violations |\n| **Complexity** | Deep nesting, high cyclomatic complexity |\n| **Coupling** | Circular imports, tight dependencies |\n| **Naming** | Abbreviations, generic or inconsistent names |\n| **Dead code** | Unreachable branches, unused parameters |\n\nResults are **prioritized** \u2014 high-severity, low-effort, high-impact fixes rise to the top. Risky changes get flagged so you can plan extra testing.\n\n---\n\n### Option 2: Share your code directly\n\nPaste the code (or describe the problem) here and I can:\n\n- Identify specific issues\n- Suggest concrete improvements\n- Help you work through the changes\n\n---\n\n### Option 3: Planning a larger migration?\n\nIf your refactor involves upgrading a language version, framework, or library, that's a **migration** \u2014 a different scope and risk profile. Let me know and I can help you plan an incremental approach with compatibility layers and rollback options.\n\n---\n\n**What would help most:** share the code you're working with, or tell me what's feeling painful about it (hard to test? too complex? duplicate logic?), and we'll go from there.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.8571428571428571, "supported_claims": 12, "unsupported_claims": 2, "total_claims": 14, "generate_ms": 8154.8367500072345, "judge_ms": 8339.453959022649 }, { "id": "gq-008", "query": "prepare a release", "difficulty": "medium", "variant": "baseline", "answer": "# Preparing a Release\n\nI can walk you through this using the release prep skill documented in the sources above.\n\n---\n\n## The fastest way to start\n\nJust run the full preflight check:\n\n```\n/release-prep check\n```\n\nOr, if you already know your target version:\n\n```\n/release-prep 5.6.0 # substitute your actual version\n```\n\n*(Source: `tasks/use-release-prep.md`)*\n\n---\n\n## What happens next\n\nBefore running, the skill will ask you two quick questions:\n\n1. **What version are you releasing?** (or should it check and suggest the next bump?)\n2. **What scope?** \u2014 full release prep, or a specific check only?\n\nYou can answer in plain language \u2014 e.g., *\"just check everything\"* runs the full assessment.\n\n---\n\n## What gets checked\n\nThe full preflight covers five areas (`concepts/tool-release-prep.md`):\n\n| Area | What it verifies | Blocks release? |\n|---|---|---|\n| **Health** | Tests pass, lint clean, coverage threshold met | Yes, if tests fail |\n| **Security** | No new CVEs, no `eval`/`exec`, secrets scan clean | Yes, if critical findings |\n| **Changelog** | Entry exists for this version, date is current | Yes, if missing |\n| **Dependencies** | Pinned versions, no known vulnerabilities | Yes, if vulnerable dep found |\n| **Version** | Semver bump matches changes, `pyproject.toml` updated, dist builds cleanly | Yes, if version mismatch |\n\n---\n\n## Reading the result\n\nYou'll get a **GO** or **NO-GO** verdict with a structured report. Example from `tasks/use-release-prep.md`:\n\n```\nRelease Readiness Assessment\nVerdict: NO-GO\n\nHealth PASS Tests: 15,482 passing, coverage 87%\nSecurity PASS No critical findings, 2 advisory\nChangelog FAIL No entry for v5.6.0\nDependencies PASS All pinned, no known CVEs\nVersion PASS Semver valid, dist builds clean\n\nBlockers (1)\n Changelog \u2014 No entry for v5.6.0 in CHANGELOG.md\n```\n\n---\n\n## If there are blockers\n\nAfter reviewing the report, you can say:\n\n- **\"Fix the blockers\"** \u2014 the skill attempts to resolve each issue\n- **\"Update the changelog\"** \u2014 generates a changelog entry from recent commits\n- **\"Re-run the check\"** \u2014 re-runs the assessment after your fixes\n- **\"Tag and publish\"** \u2014 proceeds to tag and upload once the verdict is **GO**\n\n---\n\n**Ready to go?** Start with `/release-prep check` and share the results \u2014 I can help you work through any blockers from there.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 24, "unsupported_claims": 0, "total_claims": 24, "generate_ms": 10854.754750034772, "judge_ms": 11877.673582988791 }, { "id": "gq-009", "query": "audit documentation for staleness", "difficulty": "medium", "variant": "baseline", "answer": "## Auditing Documentation for Staleness\n\nBased on the context, there's a dedicated tool for exactly this: **Doc Audit**.\n\n### What it does\n\n`doc_audit()` audits existing documentation for:\n- **Staleness** \u2014 content that's drifted out of date\n- **Broken links** \u2014 references that no longer resolve\n- **Drift from source code** \u2014 docs that no longer match the codebase they describe\n\n*(Source: `references/tool-doc-audit.md`)*\n\n---\n\n### How to run it\n\nThe tool accepts a single parameter:\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `path` | string | `.` | Project root path |\n\n**Basic usage** (runs against current directory):\n```\ndoc_audit()\n```\n\n**Against a specific path**, you'd pass the `path` parameter pointing to your project root.\n\n---\n\n### What to be aware of\n\nThe reference entry notes that `doc_audit` belongs to the **workflow** group and is sourced from `src/attune/mcp/tool_schemas.py`. The documentation for it is relatively sparse \u2014 notably, there's no CLI equivalent shown (unlike security audit, which has `attune workflow run security-audit`), and no description of output format or depth options. That itself may be worth flagging as a gap in the docs.\n\n---\n\n### Related tools you might also want\n\nThe doc audit reference points to these related workflow tools if you're doing a broader quality pass:\n- **Security Audit** \u2014 vulnerability scanning\n- **Bug Predict** \u2014 predicts failure-prone areas\n- **Code Review** \u2014 general code quality", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.8181818181818182, "supported_claims": 18, "unsupported_claims": 4, "total_claims": 22, "generate_ms": 8065.830999985337, "judge_ms": 9444.114375044592 }, { "id": "gq-010", "query": "plan a new feature", "difficulty": "hard", "variant": "baseline", "answer": "## Planning a New Feature\n\nBased on the context, you can use the **`/planning` skill** to plan a new feature in Claude Code.\n\n### Quick Start\n\n```\n/planning \n```\n\n**Example:**\n```\n/planning feature: user authentication with OAuth\n```\n\n*(Source: `quickstarts/skill-planning.md`)*\n\n---\n\n### What It Does\n\nThe `/planning` skill handles high-level development planning, including:\n\n- **Feature planning** \u2014 what you're asking about\n- **TDD planning** \u2014 test-driven development structure\n- **Architecture review** \u2014 structural design decisions\n\nResults are returned as structured output directly in your Claude Code conversation.\n\n---\n\n### Want the Full Reference?\n\nRun this in Claude Code for complete documentation:\n\n```\nattune help-docs ref-skill-planning\n```\n\n---\n\n### Is Your Target Area Already Complex?\n\nIf the part of the codebase where you're adding the feature is already tangled or hard to change, you might want to run a **`/refactor-plan`** on it first before adding new functionality:\n\n```\n/refactor-plan \n```\n\nThis helps you avoid building on a shaky foundation. *(Source: `concepts/tool-refactor-plan.md`)*", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.9090909090909091, "supported_claims": 10, "unsupported_claims": 1, "total_claims": 11, "generate_ms": 6351.240082993172, "judge_ms": 7125.28400006704 }, { "id": "gq-011", "query": "vulnerability scan", "difficulty": "hard", "variant": "baseline", "answer": "## Running a Vulnerability Scan\n\nYou have two ways to run a security vulnerability scan, depending on your setup:\n\n---\n\n### Option 1: CLI Workflow\n\n```\nattune workflow run security-audit --path \"src/\"\n```\n\nThis gives you **severity-grouped findings with CWE identifiers**. After fixing critical issues, the suggested next step is:\n\n```\nattune workflow run test-gen\n```\n\n*(Source: quickstarts/run-security-audit.md)*\n\n---\n\n### Option 2: Claude Code Skill\n\n```\n/security-audit \n```\n\nThis surfaces structured results directly in your Claude Code conversation. For the full reference:\n\n```\nattune help-docs ref-skill-security-audit\n```\n\n*(Source: quickstarts/skill-security-audit.md)*\n\n---\n\n### What Gets Scanned\n\nThe audit checks for these vulnerability categories:\n\n| Category | Examples |\n|----------|----------|\n| **Code injection** | `eval()`, `exec()`, `compile()` on untrusted input |\n| **Path traversal** | File ops without path validation |\n| **Hardcoded secrets** | API keys, tokens, passwords in source |\n| **SQL/command injection** | String concatenation in queries or shell commands |\n| **SSRF** | HTTP requests to user-controlled URLs |\n| **Weak cryptography** | MD5/SHA1 for security, hardcoded IVs |\n\n### How Deep?\n\n| Depth | Time | Coverage |\n|-------|------|----------|\n| Quick | ~30s | Surface scan \u2014 eval/exec, obvious secrets |\n| Standard | ~2 min | Full pattern matching with severity ratings |\n| Deep | ~5 min | Multi-pass with OWASP mapping and fix suggestions |\n\n*(Source: concepts/tool-security-audit.md)*\n\n---\n\n**Good times to run this:** before a release, after adding file-handling or user-input code, when adding a new dependency, or as a CI gate on pull requests.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 19, "unsupported_claims": 0, "total_claims": 19, "generate_ms": 6798.211542074569, "judge_ms": 8549.02824992314 }, { "id": "gq-012", "query": "write documentation for my module", "difficulty": "hard", "variant": "baseline", "answer": "I can help you generate documentation for your module. Based on the doc-gen skill (from `references/skill-doc-gen.md`), here's how to get started:\n\n## Invoke the skill\n\n```\n/doc-gen \n```\n\nOr use natural language, for example:\n\n```\ndocument src/models/\nadd docstrings to auth.py\ngenerate a README for the cli module\n```\n\n---\n\n## What happens next\n\nBefore generating anything, the skill will ask you **two scoping questions**:\n\n1. **What to document** \u2014 a single file, a module, or a directory tree\n2. **What format** \u2014 choose from:\n\n| Doc type | What you get |\n|----------|-------------|\n| **Docstrings** | Google-style per function/class/method |\n| **README** | Feature list, usage examples, install instructions |\n| **API reference** | Full signatures with types and descriptions |\n| **Module overview** | Architecture summary, dependency map |\n| **Full pipeline** | Audit gaps + generate all of the above |\n\n---\n\n## What it reads from your code\n\nDoc-gen analyzes your actual source \u2014 function signatures, type hints, class hierarchies, `__all__` exports, decorators, and exception handlers \u2014 so the generated docs reflect your real API rather than drifting from it.\n\n---\n\n## To get started\n\nJust tell me:\n\n1. **Which file or module** do you want documented? (e.g., `src/auth/`, `models.py`)\n2. **What kind of docs** do you need \u2014 docstrings, a README, API reference, or a full audit?", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 19, "unsupported_claims": 0, "total_claims": 19, "generate_ms": 8395.555416005664, "judge_ms": 8836.404333938845 }, { "id": "gq-013", "query": "orchestrate documentation workflow", "difficulty": "hard", "variant": "baseline", "answer": "## Running a Documentation Workflow via Orchestration\n\nBased on the context (from `concepts/tool-workflow-orchestration.md` and `tasks/use-workflow-orchestration.md`), there are **two documentation-related workflows** you can orchestrate:\n\n| Workflow | What it does | Typical time |\n|----------|-------------|-------------|\n| **Doc Audit** | Documentation freshness and gap analysis | ~1 min |\n| **Doc Generation** | Generates docs from source code | ~2 min |\n\n---\n\n### How to run it\n\nYou can trigger this in plain English or with the explicit command:\n\n**Plain English:**\n```\nrun doc audit and doc generation on src/\n```\n\n**Explicit command** (from `quickstarts/skill-workflow-orchestration.md`):\n```\n/workflow-orchestration docs\n```\n\n---\n\n### What to expect\n\n1. The skill will ask **which path or files to analyze** if you haven't specified one.\n2. It runs each documentation workflow in sequence.\n3. You get a **combined report** with findings grouped by severity, with each finding tagged by which workflow flagged it (e.g., `[doc-audit]`).\n\n---\n\n### What to do after\n\nFrom `tasks/use-workflow-orchestration.md`, after reviewing results you can:\n\n- **Go deeper** \u2014 *\"deep review src/docs/\"*\n- **Add more workflows** \u2014 *\"add code-review to the results\"*\n- **Export** \u2014 *\"export the report as JSON\"*\n\n---\n\n### Want more detail?\n\n- Say **\"tell me more\"** for the full reference: `attune help-docs ref-skill-workflow-orchestration`\n- If you only need one doc workflow, you can call it directly (e.g., `doc-audit src/`) \u2014 orchestration is most valuable when combining two or more workflows.", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 0.8095238095238095, "supported_claims": 17, "unsupported_claims": 4, "total_claims": 21, "generate_ms": 7496.538207982667, "judge_ms": 11685.944041004404 }, { "id": "gq-014", "query": "look for dangerous eval calls", "difficulty": "hard", "variant": "baseline", "answer": "I can help you scan for dangerous `eval()` calls. Based on the context, here's how to run that scan:\n\n## Running the Scan\n\nThe quickest way:\n\n```\n/bug-predict src/\n```\n\nOr using natural language:\n\n```\npredict bugs in src/\n```\n\nThe scanner will flag dangerous `eval()` and `exec()` usage under the **`dangerous_eval`** pattern type, which shows up in the HIGH severity section of results.\n\n---\n\n## \u26a0\ufe0f Important: Known False Positive\n\nBefore you act on results, there's a **verified warning** you should know about (from `warnings/bug-predict-dangerous-eval-flags-subprocess-exec.md`):\n\n> The scanner's regex matches `create_subprocess_exec` as containing `exec`, producing a **false positive** for `dangerous_eval` in `hooks/executor.py`.\n\n**What this means:** If you see a `dangerous_eval` finding in `hooks/executor.py`, it may not be a real vulnerability \u2014 the pattern match is catching `subprocess_exec` in the function name, not an actual `eval()` call.\n\n**Mitigation:** Always verify HIGH severity `dangerous_eval` findings against the actual source code before treating them as real vulnerabilities.\n\n---\n\n## Reading the Results\n\nGenuine `dangerous_eval` findings will look like this in the report:\n\n```\nHIGH (findings)\n src/hooks/executor.py:89 dangerous_eval eval() on user input\n src/plugins/loader.py:142 dangerous_eval exec() in plugin loader\n```\n\nEach finding includes the **file path**, **line number**, and a plain-English description. Links are clickable to jump directly to the code.\n\nOnce you have results, you can ask **\"fix the dangerous_eval in [file]\"** for a guided remediation.", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 1.0, "supported_claims": 14, "unsupported_claims": 0, "total_claims": 14, "generate_ms": 7771.835624938831, "judge_ms": 7819.358375039883 }, { "id": "gq-015", "query": "sniff out hard-to-catch bugs", "difficulty": "hard", "variant": "baseline", "answer": "## Sniffing Out Hard-to-Catch Bugs\n\nBased on the grounding context, here's what's available for proactive bug detection:\n\n---\n\n### `/bug-predict` \u2014 Scan for Likely Bug Locations\n\nThe **bug prediction scanner** (from `concepts/tool-bug-predict.md`) is designed exactly for this. It analyzes code *before* bugs surface rather than waiting for production incidents.\n\n**Run it with:**\n```\n/bug-predict \n```\n\n#### What it catches\n\n| Pattern | Severity | Why it's sneaky |\n|---|---|---|\n| `eval()`, `exec()`, `compile()` on input | HIGH | Code injection vectors that may only trigger on specific inputs |\n| Bare `except:` or silent `except Exception:` | MEDIUM | Errors get swallowed \u2014 failures disappear without a trace |\n| TODO / FIXME / HACK / XXX comments | LOW | Unfinished code paths that break under edge cases |\n\n#### Beyond simple pattern matching\n\nIt also weighs harder-to-spot signals:\n\n- **Cyclomatic complexity** \u2014 deeply nested conditionals that are hard to reason about\n- **Change frequency** \u2014 \"hot\" files modified often are regression magnets\n- **Code smells** \u2014 functions over 50 lines, oversized classes, duplicated logic\n\n#### Smart false-positive filtering\n\nIt suppresses known-safe matches (e.g., `eval()` in test fixture strings, JavaScript's `regex.exec()`), so results focus on *real* risks.\n\n---\n\n### Good Moments to Run It\n\n- Before merging a large PR\n- After inheriting unfamiliar code\n- Before a release, to verify no new high-severity patterns crept in\n- As a periodic health check on high-churn modules\n\n---\n\n### Also Worth Knowing\n\nIf you want bugs caught *automatically on every push*, the CI/CD pipeline concept (`concepts/task-ci-cd-pipeline.md`) covers integrating security scanning stages \u2014 including tools like **bandit** and **pip-audit** \u2014 directly into GitHub Actions so nothing slips through at merge time.\n\nWant the deeper reference for `/bug-predict`, or help wiring scanning into a CI pipeline?", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 0.875, "supported_claims": 21, "unsupported_claims": 3, "total_claims": 24, "generate_ms": 9488.69641602505, "judge_ms": 13546.223624958657 } ] }, { "variant": "strict", "total_queries": 15, "precision_at_1": 0.7333333333333333, "recall_at_k": 0.8666666666666667, "mean_faithfulness": 0.9683220211161389, "refusal_rate": 0.0, "hallucination_rate": 0.26666666666666666, "mean_generate_ms": 6472.321252718878, "mean_judge_ms": 8339.42632500548, "runs": [ { "id": "gq-001", "query": "how do I run a security audit", "difficulty": "easy", "variant": "strict", "answer": "## Running a Security Audit\n\nYou have a few ways to run a security audit:\n\n### Option 1: Natural language\n```\nscan src/ for security issues\n```\n\n### Option 2: Direct skill command\n```\n/security-audit \n```\n\n### Option 3: Workflow command\n```\nattune workflow run security-audit --path \"src/\"\n```\n\n---\n\n### What you can scan\n\n| Command | What it scans |\n|---------|---------------|\n| `/security-audit src/auth.py` | One file |\n| `/security-audit src/` | A directory tree |\n| `/security-audit .` | The whole project |\n\n---\n\n### What you get back\n\nResults are returned as a severity-grouped table with CWE identifiers, file links, and line numbers \u2014 for example:\n\n- **Critical** \u2014 `eval()` on user input (CWE-95)\n- **High** \u2014 path traversal, SSRF risks\n- **Medium** \u2014 hardcoded credentials, broad exception handling\n\n---\n\n### What to do next\n\nAfter reviewing results you can:\n- Ask **\"fix the critical findings\"** to generate patches\n- Ask **\"write tests for the flagged files\"** to prevent regressions\n- Run `attune workflow run test-gen` to follow up with test generation", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 14, "unsupported_claims": 0, "total_claims": 14, "generate_ms": 5189.534790930338, "judge_ms": 7841.101667028852 }, { "id": "gq-002", "query": "generate tests for my code", "difficulty": "easy", "variant": "strict", "answer": "Based on the context provided, here is what will happen when you generate tests for your code:\n\n## Getting Started\n\nSince you haven't specified a target, you'll be guided through **two quick questions** before anything runs:\n\n| Step | What you'll be asked | Why |\n|------|---------------------|-----|\n| 1. **Target** | \"Which file or module needs tests?\" | Focuses on one area instead of scanning everything |\n| 2. **Approach** | \"Gap analysis, generate tests, or both?\" | Gap analysis is fast; generation takes longer but gives you runnable code |\n\n> **Tip:** You can skip these questions entirely by providing both details upfront, for example:\n> ```\n> generate tests for src/auth/\n> ```\n\n---\n\n## What You'll Get\n\nOnce a target is confirmed, the **generate tests** approach produces:\n\n- Working **pytest files** with assertions\n- **Edge cases** \u2014 boundary values, empty inputs, None handling\n- **Error path tests** \u2014 tests for expected exceptions and failures\n- **Parametrized tests** \u2014 `@pytest.mark.parametrize` for input combinations\n\nGeneration typically takes **~1\u20132 minutes per module**.\n\n---\n\n## What to Do Next\n\nAfter tests are generated, you can:\n\n- **\"run the tests to verify\"** \u2014 check they pass\n- **\"write tests for those gaps\"** \u2014 target remaining untested functions\n- **\"now check src/api/\"** \u2014 move to another module\n- **\"show me coverage now\"** \u2014 see updated coverage", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 14, "unsupported_claims": 0, "total_claims": 14, "generate_ms": 7211.690749973059, "judge_ms": 7781.545958016068 }, { "id": "gq-003", "query": "fix failing tests", "difficulty": "easy", "variant": "strict", "answer": "Based on the context provided, here is how to fix your failing tests:\n\n## Getting Started\n\nSince you didn't specify a target, the skill will ask you two questions before running:\n\n1. **Which test is failing?** \u2014 Provide a file, a test name, or ask it to find failures automatically.\n2. **When did it break?** \u2014 Did it start failing after a recent change, or has it been broken for a while?\n\n---\n\n## Fastest Option: Name the Test Directly\n\nIf you know which test is failing, you can skip those questions entirely by naming it inline:\n\n```\nfix tests/unit/test_auth.py\n```\nor\n```\n/fix-test tests/unit/test_auth.py::test_login_expired\n```\n\n---\n\n## What Will Happen\n\nOnce a target is identified, the skill will:\n\n1. **Diagnose** \u2014 Run the test, read the traceback, and classify the root cause\n2. **Repair & Retry** \u2014 Apply a fix and re-run, up to **3 attempts total**\n3. **Report** \u2014 Show which tests were fixed, which still fail, how many attempts were used, and what to try next\n\n---\n\n## If It Can't Fix Everything\n\nAfter 3 attempts, any remaining failures will include a clear explanation of what was tried and a **suggestion for manual follow-up**.\n\n---\n\n**To proceed:** Which test is failing, and did it break after a recent change?", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 11, "unsupported_claims": 0, "total_claims": 11, "generate_ms": 6767.906291992404, "judge_ms": 11393.09695805423 }, { "id": "gq-004", "query": "review code quality", "difficulty": "easy", "variant": "strict", "answer": "# Code Quality Review\n\nTo get started, I need two quick things before running the review:\n\n## Step 1: Scope\n**Which files or directory should I review?**\n\n| Option | Example | When to use |\n|--------|---------|-------------|\n| Single file | `src/config.py` | Quick check on a specific change |\n| Directory | `src/auth/` | Review a module before PR |\n| Whole project | `.` | Baseline health assessment |\n\n---\n\n## Step 2: Depth\n**How thorough should the review be?**\n\n| Depth | What runs | Time | Best for |\n|-------|-----------|------|----------|\n| **Quick** | Style and formatting only | Seconds | Pre-commit, fast feedback |\n| **Standard** | + Logic errors, likely bugs | ~1 min | Pull requests *(default)* |\n| **Deep** | + Security, architecture, test gaps | ~3 min | Pre-release, critical modules |\n\n---\n\n**You can skip these questions** by providing both upfront, for example:\n\n```\ndeep review of src/auth/\n```\n\n```\n/code-quality src/auth/\n```\n\nWhat would you like to review, and at what depth?", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 17, "unsupported_claims": 0, "total_claims": 17, "generate_ms": 4262.414541095495, "judge_ms": 7913.36170793511 }, { "id": "gq-005", "query": "deep review my PR", "difficulty": "easy", "variant": "strict", "answer": "Based on the context provided, here is how to run a deep review:\n\n## Running a Deep Review\n\nYou can trigger a deep review in a few ways:\n\n**Natural language:**\n```\ndeep review of src/auth/\n```\n\n**Skill command:**\n```\n/code-quality src/auth/\n```\n\n**CLI:**\n```\nattune workflow run code-review --path \"src/\"\n```\n\n**Directly via tool:**\n```\ndeep_review(path=\"...\")\n```\n\n---\n\nHowever, your request \u2014 **\"deep review my PR\"** \u2014 does not specify which files or directory your PR covers. Based on the guided flow described in the context, you would be asked:\n\n> **\"Which files or directory should I review?\"**\n\n**Please provide the path** (e.g., `src/auth/`, a specific file, or `.` for the whole project) and the deep review will run immediately without further questions, since you've already specified the depth.\n\n---\n\n**What a deep review includes:**\n\n| What runs | Time |\n|-----------|------|\n| Style and formatting | |\n| Logic errors and likely bugs | |\n| **+ Security, architecture, and test gaps** | ~3 min |\n\nDeep depth is described as best for **pre-release and critical modules** \u2014 which aligns well with a PR review.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.8461538461538461, "supported_claims": 11, "unsupported_claims": 2, "total_claims": 13, "generate_ms": 7864.924291963689, "judge_ms": 9824.530750047415 }, { "id": "gq-006", "query": "predict bugs before they happen", "difficulty": "medium", "variant": "strict", "answer": "# Bug Prediction\n\nBug prediction scans your codebase for patterns that historically cause production incidents, analyzing code structure, complexity, and known anti-patterns to identify where failures are most likely to happen.\n\n---\n\n## Get Started Immediately\n\n```\n/bug-predict src/\n```\n\nOr just describe what you want:\n\n```\npredict bugs in src/\nwhere are bugs most likely in the workflows module?\nfind risky code in the auth package\n```\n\n---\n\n## What It Detects\n\n| Pattern | Severity | What It Flags |\n|---------|----------|---------------|\n| **dangerous_eval** | HIGH | `eval()`, `exec()`, `compile()` on any input \u2014 code injection vectors |\n| **broad_exception** | MEDIUM | Bare `except:`, unlogged `except Exception:` \u2014 silently swallowed errors |\n| **incomplete_code** | LOW | TODO, FIXME, HACK, XXX comments \u2014 unfinished code paths |\n\nBeyond patterns, the scanner also weighs **cyclomatic complexity**, **change frequency** (hot files), and **code smells** like functions over 50 lines or duplicated logic.\n\n---\n\n## Example Output\n\n```\nBug Prediction Report\nRisk Score: 73/100 | Files: 34 | Findings: 8\n\nHIGH (2 findings)\n src/hooks/executor.py:89 dangerous_eval eval() on user input\n src/plugins/loader.py:142 dangerous_eval exec() in plugin loader\n\nMEDIUM (3 findings)\n src/api/webhook.py:67 broad_exception bare except: masks errors\n ...\n```\n\nResults are sorted by severity with clickable file links to jump directly to each issue.\n\n---\n\n## Choosing Your Scope\n\n| Command | Scans |\n|---------|-------|\n| `/bug-predict src/auth.py` | One file |\n| `/bug-predict src/` | A directory tree |\n| `/bug-predict .` | The whole project |\n\n---\n\n## After the Report\n\n- **Fix HIGHs first** \u2014 ask *\"fix the dangerous_eval in executor.py\"* for a guided fix\n- **Generate tests** \u2014 ask *\"write tests for the flagged files\"*\n- **Track trends** \u2014 run weekly to see if risk scores improve or drift\n\n---\n\n## Learn More\n\n- Say **\"what is bug prediction?\"** for a full concept overview\n- Say **\"scan for security issues\"** to run a security audit instead", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 28, "unsupported_claims": 0, "total_claims": 28, "generate_ms": 9341.828624950722, "judge_ms": 10012.80908298213 }, { "id": "gq-007", "query": "refactor my code", "difficulty": "medium", "variant": "strict", "answer": "Based on the context provided, here is how you can get a refactor plan for your code:\n\n## Run the Refactor Plan Skill\n\n```\n/refactor-plan \n```\n\nReplace `` with the path to the code you want refactored. This will produce structured results in your Claude Code conversation.\n\n---\n\n## What It Will Analyze\n\nThe refactor plan scans for:\n\n| Category | What It Finds |\n|----------|---------------|\n| **Code smells** | Long methods, god classes, feature envy, data clumps |\n| **Duplication** | Copy-pasted blocks, near-duplicates, DRY violations |\n| **Complexity** | High cyclomatic complexity, deep nesting, long chains |\n| **Coupling** | Circular imports, tight dependencies, shotgun surgery |\n| **Naming** | Abbreviations, generic names, inconsistent conventions |\n| **Dead code** | Unreachable branches, unused params, vestigial modules |\n\n---\n\n## How Issues Are Prioritized\n\nIssues are ranked by four factors \u2014 **severity**, **effort**, **impact**, and **risk**. Items that are high-severity, low-effort, and high-impact rise to the top. Risky changes are flagged so you can plan extra testing.\n\n---\n\n## To Get More Detail\n\n- Say **\"tell me more\"** for step-by-step instructions\n- Say **\"what is code quality?\"** for a broader code review\n- Say **\"tell me about security audit\"** to scan for vulnerabilities instead", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 14, "unsupported_claims": 0, "total_claims": 14, "generate_ms": 7857.738833990879, "judge_ms": 6892.420749994926 }, { "id": "gq-008", "query": "prepare a release", "difficulty": "medium", "variant": "strict", "answer": "# Preparing a Release\n\nHere's how to get started with release prep:\n\n## Quickest way to start\n\nJust say:\n```\nis this project ready to release?\n```\nOr invoke the skill directly:\n```\n/release-prep check\n```\n\n---\n\n## What happens next\n\nRelease prep will ask you **two questions** before running:\n\n1. **What version?** \u2014 What version are you releasing, or should it check the current version and suggest the next bump?\n2. **What stage?** \u2014 Full release prep, or a specific check?\n - **Full** \u2014 health, security, changelog, deps, version\n - **Prep check** \u2014 verify readiness without a version bump\n - **Changelog only** \u2014 validate the changelog entry\n - **Security only** \u2014 run the security gate\n\nYou can answer in natural language (e.g., *\"just check everything\"*).\n\n---\n\n## What gets checked\n\n| Check Area | What It Verifies | Blocks Release? |\n|---|---|---|\n| **Health** | Tests pass, lint clean, coverage above threshold | Yes, if tests fail |\n| **Security** | No new CVEs, no eval/exec, secrets scan clean | Yes, if critical findings |\n| **Changelog** | Entry exists for this version, date is current | Yes, if missing |\n| **Dependencies** | Pinned versions, no vulnerabilities, compatible ranges | Yes, if vulnerable dep |\n| **Version** | Semver bump matches changes, pyproject.toml updated, dist builds cleanly | Yes, if version mismatch |\n\n---\n\n## The verdict\n\nYou'll get either:\n- \u2705 **GO** \u2014 every check passed or has only advisory warnings. Safe to tag, build, and publish.\n- \u274c **NO-GO** \u2014 one or more blocking issues found, with a list of exactly what to fix.\n\n---\n\n## After the report\n\n- Say **\"fix the blockers\"** to have the skill attempt to resolve issues\n- Say **\"re-run the check\"** after making fixes\n- Say **\"tag and publish\"** if the verdict is GO", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 25, "unsupported_claims": 0, "total_claims": 25, "generate_ms": 8599.437083001249, "judge_ms": 9321.286750026047 }, { "id": "gq-009", "query": "audit documentation for staleness", "difficulty": "medium", "variant": "strict", "answer": "## Doc Audit Tool\n\nBased on the provided context, here is what is documented about auditing documentation for staleness:\n\n### What it does\nThe **doc audit** tool audits existing documentation for:\n- Staleness\n- Broken links\n- Drift from source code\n\n### How to use it\n\n**Tool call:**\n```\ndoc_audit()\n```\n\n### Parameters\n\n| Parameter | Type | Description | Default |\n|-----------|------|-------------|---------|\n| `path` | string | Project root path | `.` |\n\nThe `path` parameter is optional \u2014 it defaults to the current directory (`.`) if not specified.\n\n### Additional details\n- **Group:** workflow\n- **Source:** `src/attune/mcp/tool_schemas.py`\n\n---\n\nThe context does not contain further details about how the doc audit tool works internally, what its output looks like, or how deep its analysis goes beyond the three stated checks (staleness, broken links, and source drift).", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 10, "unsupported_claims": 0, "total_claims": 10, "generate_ms": 4647.93304190971, "judge_ms": 6020.579959033057 }, { "id": "gq-010", "query": "plan a new feature", "difficulty": "hard", "variant": "strict", "answer": "Based on the context provided, you can plan a new feature using the `/planning` skill:\n\n```\n/planning \n```\n\nFor a new feature specifically, you would run:\n\n```\n/planning feature\n```\n\n**Result:** Structured results in your Claude Code conversation.\n\n**Next:** For more details, see the full reference by running:\n```\nattune help-docs ref-skill-planning\n```", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.8, "supported_claims": 4, "unsupported_claims": 1, "total_claims": 5, "generate_ms": 2436.9569580303505, "judge_ms": 5525.078792008571 }, { "id": "gq-011", "query": "vulnerability scan", "difficulty": "hard", "variant": "strict", "answer": "## Vulnerability Scan\n\nBased on the context, there are two ways to run a vulnerability scan:\n\n---\n\n### Option 1: CLI Workflow\n\n```\nattune workflow run security-audit --path \"src/\"\n```\n\n**Result:** Severity-grouped findings with CWE identifiers.\n**Next step:** Fix critical issues, then run `attune workflow run test-gen`.\n\n---\n\n### Option 2: Claude Code Skill\n\n```\n/security-audit \n```\n\n**Result:** Structured results in your Claude Code conversation.\n**Next step:** See full reference with `attune help-docs ref-skill-security-audit`.\n\n---\n\n### What It Scans For\n\n| Category | Details |\n|----------|---------|\n| **Code injection** | `eval()`, `exec()`, `compile()` on untrusted input |\n| **Path traversal** | File operations without path validation |\n| **Hardcoded secrets** | API keys, tokens, passwords in source |\n| **SQL/command injection** | String concatenation in queries or shell commands |\n| **SSRF** | HTTP requests to user-controlled URLs |\n| **Weak cryptography** | MD5/SHA1 for security, hardcoded IVs |\n\n### Scan Depth Options\n\n| Depth | Time | Output |\n|-------|------|--------|\n| **Quick** | ~30s | Surface scan \u2014 eval/exec, obvious secrets |\n| **Standard** | ~2 min | Full pattern matching with severity ratings |\n| **Deep** | ~5 min | Multi-pass review with OWASP mapping and fix suggestions |", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 16, "unsupported_claims": 0, "total_claims": 16, "generate_ms": 5118.200332974084, "judge_ms": 7067.97587510664 }, { "id": "gq-012", "query": "write documentation for my module", "difficulty": "hard", "variant": "strict", "answer": "I can help you generate documentation for your module. Based on the doc-gen skill, here's how to get started:\n\n## Invoke the skill\n\n```\n/doc-gen \n```\n\nOr use natural language, for example:\n```\ndocument src/models/\nadd docstrings to auth.py\ngenerate a README for the cli module\n```\n\n---\n\n## What happens next\n\nBefore generating anything, the skill will ask you **two questions**:\n\n1. **What to document** \u2014 a single file, a module, or a directory tree\n2. **What format** \u2014 choose from:\n\n| Doc type | Best for |\n|----------|----------|\n| **Docstrings** | Keeping code self-documenting |\n| **README** | Module or project overviews |\n| **API reference** | Library consumers |\n| **Module overview** | Onboarding and navigation |\n| **Full pipeline** | Pre-release documentation sweep (audits gaps + generates all doc types) |\n\n---\n\n## What it reads from your code\n\nDoc-gen analyzes your actual source to produce accurate docs, including:\n- Function signatures, parameter names, types, and defaults\n- Type hints and return types\n- Class hierarchies and abstract methods\n- Exception handlers\n- Existing docstrings (preserving manual docs and filling gaps)\n\n---\n\n## After generation\n\nOnce docs are produced, you can follow up with commands like:\n- **\"apply these to the files\"** to write the changes\n- **\"audit the whole project for gaps\"** to check coverage\n- **\"generate a README for this module\"** to add a README\n- **\"export the docs as a markdown file\"** to save the output\n\nTo proceed, share the path or module you want documented and your preferred format.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 21, "unsupported_claims": 0, "total_claims": 21, "generate_ms": 6407.239749911241, "judge_ms": 8950.196665944532 }, { "id": "gq-013", "query": "orchestrate documentation workflow", "difficulty": "hard", "variant": "strict", "answer": "## Running the Documentation Workflow\n\nBased on the context, here is how to orchestrate the documentation workflow:\n\n---\n\n### Command\n\n```\n/workflow-orchestration docs\n```\n\nOr in plain English:\n```\nrun doc audit on \n```\n\n---\n\n### What the Documentation Workflows Do\n\nThere are **two documentation-related workflows** available:\n\n| Workflow | What it does | Typical time |\n|----------|-------------|-------------|\n| **Doc Audit** | Documentation freshness and gap analysis | ~1 min |\n| **Doc Generation** | Generates docs from source code | ~2 min |\n\n---\n\n### How It Will Proceed\n\n1. The skill will ask which **path or files** to analyze if you haven't specified one:\n > *\"Which path or files should I analyze?\"*\n2. It runs the workflow(s) in sequence on that path\n3. It delivers a **combined scored report** with findings grouped by severity, showing which workflow flagged each item\n\n---\n\n### After the Report\n\nOnce you have results, you can:\n\n- **Go deeper** \u2014 \"deep review ``\"\n- **Generate tests** \u2014 \"write tests for the flagged files\"\n- **Export** \u2014 \"export the report as JSON\"\n- **Add more workflows** \u2014 \"add security to the results\"\n\n---\n\nTo specify a path upfront, try:\n```\n/workflow-orchestration docs\n```\nthen provide your target path when prompted.", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 0.9411764705882353, "supported_claims": 16, "unsupported_claims": 1, "total_claims": 17, "generate_ms": 6744.485000032, "judge_ms": 9949.9955419451 }, { "id": "gq-014", "query": "look for dangerous eval calls", "difficulty": "hard", "variant": "strict", "answer": "Based on the context, I can walk you through scanning for dangerous eval calls using the bug-predict skill.\n\n## Running the Scan\n\nYou can trigger it directly:\n\n```\n/bug-predict src/\n```\n\nOr with natural language like your request \u2014 the skill will interpret \"look for dangerous eval calls\" and scope the scan accordingly.\n\n---\n\n## What to Expect: `dangerous_eval` Findings\n\nThe scanner flags a pattern called **`dangerous_eval`**, which covers `eval()` and `exec()` usage. Results appear in the **HIGH severity** section of the report. For example:\n\n```\nHIGH (2 findings)\n src/hooks/executor.py:89 dangerous_eval eval() on user input\n src/plugins/loader.py:142 dangerous_eval exec() in plugin loader\n```\n\n---\n\n## \u26a0\ufe0f Important Warning Before You Act\n\nThe context includes a **verified warning** about this exact pattern:\n\n> **Bug-predict `dangerous_eval` flags `subprocess_exec` as a false positive.**\n> The scanner's regex matches `create_subprocess_exec` as containing `exec`, which produces a false positive for `dangerous_eval` in `hooks/executor.py`.\n\n**Mitigation:** Always verify HIGH severity `dangerous_eval` findings against the actual source code before treating them as real vulnerabilities.\n\n---\n\n## Next Steps After Reviewing Results\n\n- Ask **\"fix the dangerous_eval in [file]\"** for a guided fix on confirmed findings\n- Ask **\"write tests for the flagged files\"** to prevent regressions\n- Say **\"scan for security issues\"** to run a full security audit instead", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 0.9375, "supported_claims": 15, "unsupported_claims": 1, "total_claims": 16, "generate_ms": 8117.180667002685, "judge_ms": 9343.59341696836 }, { "id": "gq-015", "query": "sniff out hard-to-catch bugs", "difficulty": "hard", "variant": "strict", "answer": "Based on the provided context, here's how to sniff out hard-to-catch bugs:\n\n## Use `/bug-predict` to Find Likely Bug Locations\n\n```\n/bug-predict \n```\n\nThis scanner looks for patterns that **historically cause production incidents** \u2014 before they surface as real failures.\n\n---\n\n### What It Catches\n\n| Pattern | Severity | Examples |\n|---|---|---|\n| **dangerous_eval** | HIGH | `eval()`, `exec()`, `compile()` on input \u2014 code injection risks |\n| **broad_exception** | MEDIUM | Bare `except:` or unlogged `except Exception:` \u2014 silently swallowed errors |\n| **incomplete_code** | LOW | TODO, FIXME, HACK, XXX comments \u2014 unfinished paths that break on edge cases |\n\n### Beyond Pattern Matching\n\nIt also weighs contextual signals that make bugs *more likely*:\n\n- **Cyclomatic complexity** \u2014 deeply nested conditionals and long functions\n- **Change frequency** \u2014 frequently modified (\"hot\") files have higher regression risk\n- **Code smells** \u2014 functions over 50 lines, too many methods per class, duplicated logic\n\n### Smart Filtering\n\nIt automatically suppresses known false positives (e.g., `eval()` inside test fixture strings, JavaScript `regex.exec()` calls) so results stay focused on real risks.\n\n---\n\n### When to Run It\n\n- Before merging a large PR\n- During code review, to focus attention on real risks\n- After picking up unfamiliar code\n- Before a release, to verify no new high-severity patterns crept in", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 18, "unsupported_claims": 0, "total_claims": 18, "generate_ms": 6517.347833025269, "judge_ms": 7253.820999991149 } ] }, { "variant": "citation", "total_queries": 15, "precision_at_1": 0.7333333333333333, "recall_at_k": 0.8666666666666667, "mean_faithfulness": 0.996078431372549, "refusal_rate": 0.0, "hallucination_rate": 0.06666666666666667, "mean_generate_ms": 7303.399327723309, "mean_judge_ms": 8083.806930536714, "runs": [ { "id": "gq-001", "query": "how do I run a security audit", "difficulty": "easy", "variant": "citation", "answer": "## How to Run a Security Audit\n\nThere are a few ways to run a security audit:\n\n### Option 1: Natural language\nJust say what you want to scan:\n```\nscan src/ for security issues\n```\n[P2]\n\n### Option 2: Use the skill directly\n```\n/security-audit \n```\n[P2, P3]\n\n### Option 3: Use the workflow command\n```\nattune workflow run security-audit --path \"src/\"\n```\n[P1]\n\n---\n\n### Choosing what to scan\n\nYou can scan a single file, a directory, or your entire project [P2]:\n\n| Command | What it scans |\n|---------|---------------|\n| `/security-audit src/auth.py` | One file |\n| `/security-audit src/` | A directory tree |\n| `/security-audit .` | The whole project |\n\n### Reading the results\n\nResults come back as a table sorted by severity, and each finding includes the file, line number, description, and CWE identifier [P2]. File links are clickable, allowing you to jump directly to the issue [P2].\n\n### What to do next\n\nAfter reviewing results, you can [P2]:\n- **Fix critical issues** \u2014 ask \"fix the critical findings\" to generate patches\n- **Generate security tests** \u2014 ask \"write tests for the flagged files\"\n- **Go deeper** \u2014 ask \"deep scan src/auth/\" for a thorough review of a specific area", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 12, "unsupported_claims": 0, "total_claims": 12, "generate_ms": 5021.266040974297, "judge_ms": 6681.066874996759 }, { "id": "gq-002", "query": "generate tests for my code", "difficulty": "easy", "variant": "citation", "answer": "Here's how to generate tests for your code:\n\n## Getting Started\n\nYou can kick things off with a simple natural language request or a direct command [P3]:\n\n```\ngenerate tests for src/auth/\n```\nor\n```\n/smart-test src/auth/\n```\n\n## What Happens Next\n\nIf you don't specify a target and approach upfront, you'll be guided through two quick questions [P3]:\n\n| Step | What you're asked |\n|------|-------------------|\n| 1. Target | \"Which file or module needs tests?\" |\n| 2. Approach | \"Gap analysis, generate tests, or both?\" |\n\nIf you provide both details upfront (e.g. `\"generate tests for src/auth/\"`), the questions are skipped and it runs immediately [P1, P3].\n\n## Choosing Your Target\n\n| Command | What it covers |\n|---------|---------------|\n| `/smart-test src/auth/login.py` | One file |\n| `/smart-test src/auth/` | A directory tree |\n| `\"what needs testing?\"` | Entire project gaps |\n| `\"test the top 10 gaps\"` | Worst offenders |\n\n[P3]\n\n## What You'll Get\n\nThe generated tests output will look like this [P3]:\n\n```\nGenerated Tests\nFiles: 3 | Tests: 18 | Edge Cases: 12\n\n tests/test_session.py 6 tests (2 edge cases, 1 error path)\n tests/test_login.py 7 tests (3 parametrized, 2 error paths)\n tests/test_webhook.py 5 tests (2 edge cases, 1 error path)\n```\n\nSpecifically, test generation produces working pytest files with edge cases and takes approximately **1\u20132 minutes per module** [P3]. All generated tests use pytest conventions with descriptive names and docstrings [P3].\n\nThe output includes edge cases (boundary values, empty inputs, None handling), error path tests, and parametrized tests using `@pytest.mark.parametrize` [P1].\n\n## After the Results\n\nYou'll be offered follow-up options such as [P3]:\n- `\"run the tests to verify\"` \u2014 to run the generated tests\n- `\"now check src/api/\"` \u2014 to check a different module\n- `\"show me coverage now\"` \u2014 to see updated coverage", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 23, "unsupported_claims": 0, "total_claims": 23, "generate_ms": 9360.346458037384, "judge_ms": 9017.428208957426 }, { "id": "gq-003", "query": "fix failing tests", "difficulty": "easy", "variant": "citation", "answer": "## Fixing Your Failing Tests\n\nTo get started, the fastest way is to name the test inline \u2014 this skips any setup questions and jumps straight to diagnosis [P3]:\n\n```\nfix tests/unit/test_auth.py\n```\n\nOr use the skill directly with a specific test:\n\n```\n/fix-test tests/unit/test_auth.py::test_login_expired\n```\n\n---\n\n### What Happens Next\n\nThe fix-test skill follows this flow [P1]:\n\n1. **Diagnosis** \u2014 the failing test is run and the traceback is classified into a root cause category\n2. **Repair + retry** \u2014 a fix is applied and the test is re-run; if it still fails, a new diagnosis is made and another fix is attempted\n3. **Report** \u2014 you see which tests were fixed, which still fail, how many attempts were used, and what to try next\n\nIt retries **up to 3 attempts total**, with each attempt diagnosing fresh from the latest error [P3].\n\n---\n\n### Root Causes It Can Handle\n\n| Root Cause | Auto-Fixable? |\n|---|---|\n| Import error | Yes |\n| Mock mismatch | Yes |\n| Assertion drift | Yes |\n| Type error | Yes |\n| Fixture missing | Usually |\n| Environment issue | Sometimes |\n\n[P1]\n\n---\n\n### If It Can't Fix Automatically\n\nIf all 3 attempts are exhausted, you'll get a clear summary of what was tried and what still needs **manual attention**, along with a suggestion for how to proceed [P3].\n\n---\n\n**If you haven't specified which test is failing**, the skill will ask you two questions: which test is failing, and whether it broke recently [P3]. Providing the test name upfront skips those questions [P1].", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 21, "unsupported_claims": 0, "total_claims": 21, "generate_ms": 8649.368916987441, "judge_ms": 10115.05424999632 }, { "id": "gq-004", "query": "review code quality", "difficulty": "easy", "variant": "citation", "answer": "I'll help you run a code quality review. Here's what you need to know to get started:\n\n## Running the Review\n\nYou can kick it off in plain language or with a direct command [P2]:\n\n```\nreview src/auth/\n```\nor\n```\n/code-quality \n```\n\n## What You'll Be Asked\n\nBefore the review runs, you'll be guided through two quick questions [P2]:\n\n| Step | Question | Why |\n|------|----------|-----|\n| 1. **Scope** | Which files or directory to review? | Keeps results focused |\n| 2. **Depth** | Quick, standard, or deep? | Matches thoroughness to your needs |\n\nIf you provide both upfront (e.g. *\"deep review of src/auth/\"*), the questions are skipped [P2].\n\n## Choose Your Depth\n\n| Depth | What Runs | Time | Best For |\n|-------|-----------|------|----------|\n| **Quick** | Style and formatting only | Seconds | Pre-commit, fast feedback |\n| **Standard** | + Logic errors, likely bugs | ~1 min | Pull requests (default) |\n| **Deep** | + Security, architecture, test gaps | ~3 min | Pre-release, critical modules |\n\n[P2, P3]\n\n## What It Catches\n\n| Category | Examples | Auto-fixable? |\n|----------|---------|---------------|\n| **Style** | Unused imports, line length | Often yes |\n| **Correctness** | Unreachable code, wrong return types | Sometimes |\n| **Likely Bugs** | Broad exceptions, race conditions | No |\n| **Structural** | High coupling, circular deps | No |\n\n[P3]\n\n---\n\n**To proceed \u2014 which files or directory would you like me to review, and how deep should I go?**", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 25, "unsupported_claims": 0, "total_claims": 25, "generate_ms": 8224.635959020816, "judge_ms": 8946.45299995318 }, { "id": "gq-005", "query": "deep review my PR", "difficulty": "easy", "variant": "citation", "answer": "Here's how to run a deep review on your PR:\n\n## Running a Deep Review\n\nYou can trigger a deep review using natural language \u2014 simply saying **\"deep review of src/auth/\"** (or whatever your PR's target directory is) will skip any guided questions and run immediately [P2].\n\n### What \"Deep\" depth includes\n\nA deep review runs style checks, logic errors, likely bugs, **plus security, architecture, and test gaps** [P2]. It takes approximately **~3 minutes** and is best suited for pre-release or critical modules [P2].\n\n### How to run it\n\n**Natural language:**\n```\ndeep review of src//\n```\n[P2]\n\n**CLI:**\n```\nattune workflow run code-review --path \"src/\"\n```\n[P3]\n\n**Tool directly:**\n```\ndeep_review(path=\"...\")\n```\n[P1]\n\n### What you'll get back\n\nResults come back as a scored report with a **health score** and issues grouped by category: Style, Correctness, Likely Bugs, and Structural \u2014 each with file, line number, and a clickable link to jump to the issue [P2].\n\n### After the review\n\nOnce results are in, you can:\n- Fix auto-fixable issues by saying **\"fix the quality issues\"**\n- Run a security audit by saying **\"scan for vulnerabilities\"**\n- Generate tests by saying **\"write tests for the risky files\"** [P2]", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 14, "unsupported_claims": 0, "total_claims": 14, "generate_ms": 7145.529499975964, "judge_ms": 6711.133209057152 }, { "id": "gq-006", "query": "predict bugs before they happen", "difficulty": "medium", "variant": "citation", "answer": "## Bug Prediction: Catch Problems Before They Hit Production\n\nBug prediction scans your codebase for patterns that historically cause production incidents, analyzing code structure, complexity, and known anti-patterns to identify where failures are most likely to happen \u2014 before they surface [P1].\n\n---\n\n### How to Run It\n\nThe fastest way is to just describe what you want:\n\n```\npredict bugs in src/\n```\n\nOr use the skill directly:\n\n```\n/bug-predict src/\n```\n\nYou'll get a risk report grouped by severity with clickable file links and risk scores [P3].\n\n---\n\n### What It Detects\n\nThe scanner looks for three categories of risk [P1]:\n\n| Pattern | Severity | What It Catches |\n|---|---|---|\n| **dangerous_eval** | HIGH | `eval()`, `exec()`, `compile()` on any input \u2014 code injection vectors |\n| **broad_exception** | MEDIUM | Bare `except:`, unlogged `except Exception:` \u2014 silently swallowed errors |\n| **incomplete_code** | LOW | TODO, FIXME, HACK, XXX comments \u2014 unfinished code paths that break under edge cases |\n\nBeyond patterns, the scanner also weighs **cyclomatic complexity**, **change frequency** (frequently modified \"hot\" files), and **code smells** like functions over 50 lines or duplicated logic [P1].\n\n---\n\n### Smart False-Positive Filtering\n\nNot every match is a real bug. The scanner automatically suppresses known-safe patterns, including `eval()` inside test fixture strings, JavaScript `regex.exec()` calls, and broad exceptions marked with `# INTENTIONAL:` comments or `# noqa: BLE001` [P1].\n\n---\n\n### Scoping Your Scan\n\n| Command | What It Scans |\n|---|---|\n| `/bug-predict src/auth.py` | One file |\n| `/bug-predict src/` | A directory tree |\n| `/bug-predict .` | The whole project |\n\n[P3]\n\n---\n\n### What to Do with Results\n\n- **Fix HIGH findings first** \u2014 ask for a guided fix on specific files [P3]\n- **Generate tests** \u2014 ask to write tests for flagged files to prevent regressions [P3]\n- **Track trends** \u2014 run the scan weekly to see if risk scores are improving or drifting [P3]\n\n---\n\n### When to Use It\n\n- Before merging a large PR\n- During code review to focus human attention on real risks\n- After onboarding unfamiliar code to map risk hotspots\n- Before a release to verify no new high-severity patterns crept in\n\n[P1]", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 30, "unsupported_claims": 0, "total_claims": 30, "generate_ms": 8192.683833069168, "judge_ms": 11774.321541073732 }, { "id": "gq-007", "query": "refactor my code", "difficulty": "medium", "variant": "citation", "answer": "## Getting Started with Refactoring Your Code\n\nTo refactor your code effectively, here's what you need to know:\n\n---\n\n### Run a Refactor Plan\n\nThe quickest way to start is with the refactor-plan skill [P2]:\n\n```\n/refactor-plan \n```\n\nThis will give you structured results directly in your Claude Code conversation [P2].\n\n---\n\n### What Gets Analyzed\n\nA refactor plan scans for the following categories of issues [P1]:\n\n| Category | What it finds |\n|----------|---------------|\n| **Code smells** | Long methods, god classes, feature envy, data clumps |\n| **Duplication** | Copy-pasted blocks, near-duplicates, DRY violations |\n| **Complexity** | High cyclomatic complexity, deep nesting, long chains |\n| **Coupling** | Circular imports, tight dependencies, shotgun surgery |\n| **Naming** | Abbreviations, generic names, inconsistent conventions |\n| **Dead code** | Unreachable branches, unused params, vestigial modules |\n\n---\n\n### How Issues Are Prioritized\n\nIssues are ranked by four factors [P1]:\n\n- **Severity** \u2014 how much the issue hurts readability, testability, or safety\n- **Effort** \u2014 lines of code affected, number of files touched\n- **Impact** \u2014 how much better the code gets after the fix\n- **Risk** \u2014 chance of introducing regressions\n\nItems that are high-severity, low-effort, and high-impact float to the top, while risky changes get flagged so you can plan extra testing [P1].\n\n---\n\n### Why a Plan Matters\n\nRefactoring without a plan leads to yak-shaving \u2014 you start fixing one thing and end up touching 20 files. The roadmap tells you which changes deliver the most improvement per hour invested [P1].\n\n---\n\n### If Your Refactor Involves Large-Scale Changes\n\nIf your refactoring is part of a larger migration (e.g., upgrading Python or a framework), consider an **incremental strategy** \u2014 migrating one module at a time so you can ship, test, and roll back each piece independently [P3]. Starting without sufficient test coverage or trying to migrate everything at once are among the most common causes of failure [P3].\n\n---\n\nWould you like to go deeper? You can ask for **step-by-step refactor instructions** or a **code quality scan** to identify where to focus first [P1].", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 19, "unsupported_claims": 0, "total_claims": 19, "generate_ms": 8194.106832961552, "judge_ms": 7923.337750020437 }, { "id": "gq-008", "query": "prepare a release", "difficulty": "medium", "variant": "citation", "answer": "# Preparing a Release\n\nHere's how to run release prep for your project.\n\n## Quickest Way to Start\n\nJust say what you want in plain language:\n\n```\nis this project ready to release?\n```\n\nOr invoke the skill directly:\n\n```\n/release-prep check\n```\n\nThis kicks off the full preflight checklist and you'll get a go/no-go assessment in about two minutes [P3].\n\n## What Gets Checked\n\nRelease prep runs a preflight checklist across five areas [P1]:\n\n| Check Area | What It Verifies | Blocking? |\n|------------|-----------------|-----------|\n| **Health** | Tests pass, lint clean, coverage above threshold | Yes if tests fail |\n| **Security** | No new CVEs, no eval/exec, secrets scan clean | Yes if critical findings |\n| **Changelog** | Entry exists for this version, date is current | Yes if missing |\n| **Dependencies** | Pinned versions, no known vulnerabilities | Yes if vulnerable dep |\n| **Version** | Semver bump matches changes, pyproject.toml updated, dist builds cleanly | Yes if version mismatch |\n\n## If You Know Your Target Version\n\n```\n/release-prep 5.6.0\n```\n\nThe skill will ask two questions before running: **what version** you're releasing, and **what stage** (full release prep, prep check only, changelog only, or security only) [P3].\n\n## Reading the Result\n\nYou'll get a **GO** or **NO-GO** verdict [P1]:\n\n- **GO** \u2014 every check passed or has only advisory warnings. Safe to tag, build, and publish [P1].\n- **NO-GO** \u2014 one or more blocking issues found. The report lists exactly what to fix before retrying [P1].\n\nAny blockers are listed separately so you know exactly what to fix [P3].\n\n## After Reviewing the Report\n\n- Say **\"fix the blockers\"** \u2014 the skill will attempt to resolve each issue [P3]\n- Say **\"re-run the check\"** \u2014 runs the assessment again after you've made fixes [P3]\n- Say **\"tag and publish\"** \u2014 if the verdict is GO, proceeds to tag and upload [P3]", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 26, "unsupported_claims": 0, "total_claims": 26, "generate_ms": 8456.197374965996, "judge_ms": 8396.849583019502 }, { "id": "gq-009", "query": "audit documentation for staleness", "difficulty": "medium", "variant": "citation", "answer": "## Documentation Audit for Staleness\n\nTo audit existing documentation for staleness, broken links, and drift from source code, you can use the `doc_audit` tool [P1].\n\n### Tool Details\n\nThe tool is called `doc_audit()` and accepts the following parameter [P1]:\n\n| Parameter | Type | Description | Default |\n|-----------|------|-------------|---------|\n| `path` | string | Project root path | `.` |\n\n### Usage\n\n```\ndoc_audit()\n```\n\nSimply invoke the tool with an optional `path` argument pointing to your project root. If no path is provided, it defaults to the current directory (`.`) [P1].\n\n### What It Audits\n\nThe `doc_audit` tool specifically targets [P1]:\n- **Staleness** \u2014 documentation that is out of date\n- **Broken links** \u2014 references that no longer resolve\n- **Drift from source code** \u2014 documentation that no longer reflects the actual codebase\n\n### Related Workflows\n\nOnce documentation issues are identified, you may also want to consider running related audits [P1]:\n- **Security Audit** \u2014 scans for vulnerabilities in the codebase\n- **Bug Predict** \u2014 analyzes code patterns to predict failures\n- **Code Review** \u2014 provides comprehensive code quality review", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 13, "unsupported_claims": 0, "total_claims": 13, "generate_ms": 5733.133125002496, "judge_ms": 6249.022000003606 }, { "id": "gq-010", "query": "plan a new feature", "difficulty": "hard", "variant": "citation", "answer": "## Planning a New Feature\n\nYou can use the `/planning` skill to plan a new feature [P1].\n\n### Quick Start\n\nRun the following command in your Claude Code conversation:\n\n```\n/planning \n```\n\n[P1]\n\n**Result:** You'll get structured results directly in your Claude Code conversation [P1].\n\n### Learn More\n\nFor the full reference, run: `attune help-docs ref-skill-planning` [P1]", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 5, "unsupported_claims": 0, "total_claims": 5, "generate_ms": 2685.582916950807, "judge_ms": 4437.9647499881685 }, { "id": "gq-011", "query": "vulnerability scan", "difficulty": "hard", "variant": "citation", "answer": "## Running a Vulnerability Scan\n\nYou can scan your codebase for security vulnerabilities using either of these approaches:\n\n### Option 1: CLI Workflow\n\n```\nattune workflow run security-audit --path \"src/\"\n```\n\nThis returns severity-grouped findings with CWE identifiers. [P2]\n\n### Option 2: Claude Code Skill\n\n```\n/security-audit \n```\n\nThis delivers structured results directly in your Claude Code conversation. [P3]\n\n---\n\n### What It Detects\n\nThe scan looks for: [P1]\n\n| Category | Examples |\n|----------|----------|\n| **Code injection** | `eval()`, `exec()`, `compile()` on untrusted input |\n| **Path traversal** | File operations without path validation |\n| **Hardcoded secrets** | API keys, tokens, passwords in source |\n| **SQL/command injection** | String concatenation in queries or shell commands |\n| **SSRF** | HTTP requests to user-controlled URLs |\n| **Weak cryptography** | MD5/SHA1 for security, hardcoded IVs |\n\n### How Deep to Go\n\n| Depth | Time | What You Get |\n|-------|------|-------------|\n| **Quick** | ~30s | Surface scan \u2014 eval/exec, obvious secrets |\n| **Standard** | ~2 min | Full pattern matching with severity ratings |\n| **Deep** | ~5 min | Multi-pass review with OWASP mapping and fix suggestions |\n\n[P1]", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 13, "unsupported_claims": 0, "total_claims": 13, "generate_ms": 5598.55166694615, "judge_ms": 7010.914999991655 }, { "id": "gq-012", "query": "write documentation for my module", "difficulty": "hard", "variant": "citation", "answer": "I can help you generate documentation for your module. Here's how the process works:\n\n## Getting Started\n\nUse the `/doc-gen` command to kick things off:\n\n```\n/doc-gen \n```\n\nYou can also use natural language, for example:\n\n```\ndocument src/models/\nadd docstrings to auth.py\ngenerate a README for the cli module\n```\n[P3]\n\n## What Happens Next\n\nBefore generating anything, the skill asks you two questions [P1]:\n\n1. **What to document** \u2014 a single file, a module, or a directory tree\n2. **What format** \u2014 docstrings, README, API reference, or a full pipeline that audits gaps and generates docs for all of them\n\n## What It Can Generate\n\n| Doc type | What you get |\n|----------|-------------|\n| **Docstrings** | Google-style per function/class/method |\n| **README** | Feature list, usage examples, install instructions |\n| **API reference** | Full signatures with types and descriptions |\n| **Module overview** | Architecture summary, dependency map |\n| **Full pipeline** | Audit gaps, generate docs, review coverage |\n\n[P3]\n\n## How It Reads Your Code\n\nDoc-gen reads your actual source code to produce accurate documentation, including function signatures, type hints, class hierarchies, and module structure [P1]. Specifically, it extracts:\n\n- Parameter names, types, and defaults from function signatures\n- Return types and generic parameters from type hints\n- Inheritance chains and abstract methods from class hierarchies\n- Public API surface from module `__all__`\n- Which exceptions a function raises from exception handlers [P3]\n\n## After Generating\n\nOnce the docs are produced, you can [P3]:\n\n- **Apply them** \u2014 \"apply these to the files\"\n- **Audit further** \u2014 \"audit the whole project for gaps\"\n- **Generate a README** \u2014 \"generate a README for this module\"\n- **Export as markdown** \u2014 \"export the docs as a markdown file\"\n\n---\n\n**What would you like to document \u2014 a single file, a module, or a full directory? And what format do you need?** [P1]", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 24, "unsupported_claims": 0, "total_claims": 24, "generate_ms": 10268.66370800417, "judge_ms": 8980.851958040148 }, { "id": "gq-013", "query": "orchestrate documentation workflow", "difficulty": "hard", "variant": "citation", "answer": "## Running a Documentation Workflow with Workflow Orchestration\n\nYou can orchestrate documentation workflows using the `/workflow-orchestration` command or by describing your goal in plain English.\n\n### Quick Start\n\nRun it directly:\n\n```\n/workflow-orchestration docs\n```\n\nOr describe what you need:\n\n```\nrun documentation workflow on \n```\n\n[P1, P2]\n\n### Available Documentation Workflows\n\nThe orchestrator includes two documentation-focused workflows [P3]:\n\n| Workflow | What it does | Typical time |\n|----------|-------------|-------------|\n| **Doc Audit** | Documentation freshness and gap analysis | ~1 min |\n| **Doc Generation** | Generates docs from source code | ~2 min |\n\n[P3]\n\n### Choosing a Path\n\nIf you don't specify a path up front, the skill will ask:\n\n> \"Which path or files should I analyze?\"\n\n[P2]\n\n### What You'll Get\n\nYou'll receive a combined report with findings grouped by severity, including an overall score and a summary broken down by workflow. Each finding shows which workflow flagged it. [P2]\n\n### What to Do Next\n\nAfter reviewing the results you can:\n\n- **Fix critical issues** \u2014 \"fix the critical findings\"\n- **Go deeper** \u2014 \"deep review ``\"\n- **Export** \u2014 \"export the report as JSON\"\n\n[P2]\n\n### When Documentation Workflows Pair Well With Others\n\nA useful combination for documentation is a **doc audit + code review**, for example during **new contributor onboarding** to understand a module's state. [P3]", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 0.9411764705882353, "supported_claims": 16, "unsupported_claims": 1, "total_claims": 17, "generate_ms": 7639.184833038598, "judge_ms": 8572.536624968052 }, { "id": "gq-014", "query": "look for dangerous eval calls", "difficulty": "hard", "variant": "citation", "answer": "## Scanning for Dangerous `eval` Calls\n\nI can run bug prediction to look for dangerous eval patterns. Here's what you need to know:\n\n### How to Run the Scan\n\nYou can trigger the scan with a simple command [P1]:\n\n```\npredict bugs in src/\n```\n\nOr directly:\n\n```\n/bug-predict src/\n```\n\n### What to Expect\n\nThe scanner detects a `dangerous_eval` pattern, which flags things like `eval()` on user input and `exec()` in plugin loaders [P1]. Results are ranked by severity, and HIGH findings show up first [P1].\n\n### \u26a0\ufe0f Important False Positive Warning\n\nBefore acting on any `dangerous_eval` HIGH findings, be aware of a **known false positive**: the scanner's regex matches `create_subprocess_exec` as containing `exec`, which produces a false positive for `dangerous_eval` [P2]. This has been verified by a prior incident [P2].\n\n**Mitigation:** Always verify HIGH severity `dangerous_eval` findings against the actual source code before treating them as real vulnerabilities [P2].\n\n### What to Do After the Scan\n\nOnce you have results, you can [P1]:\n- Ask for a **guided fix** on confirmed findings (e.g., *\"fix the dangerous_eval in executor.py\"*)\n- **Generate tests** for flagged files to prevent regressions\n- Run a more focused scan on a specific subdirectory\n\nWould you like me to run the scan now, and on which path?", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 1.0, "supported_claims": 14, "unsupported_claims": 0, "total_claims": 14, "generate_ms": 7508.14879196696, "judge_ms": 7236.098667024635 }, { "id": "gq-015", "query": "sniff out hard-to-catch bugs", "difficulty": "hard", "variant": "citation", "answer": "## Sniffing Out Hard-to-Catch Bugs\n\nThe best tool for this is **`/bug-predict`**, which scans your codebase for patterns that historically cause production incidents \u2014 before they surface as real failures [P2].\n\n---\n\n### What It Catches\n\nThe scanner targets three categories of risky patterns [P2]:\n\n| Pattern | Severity | What It Means |\n|---|---|---|\n| `eval()`, `exec()`, `compile()` on input | HIGH | Code injection vectors |\n| Bare `except:` / unlogged exception catches | MEDIUM | Errors silently swallowed |\n| TODO, FIXME, HACK, XXX comments | LOW | Unfinished code paths that break on edge cases |\n\nBeyond patterns, it also weighs [P2]:\n- **Cyclomatic complexity** \u2014 deeply nested conditionals and long functions correlate with higher defect rates\n- **Change frequency** \u2014 frequently modified (\"hot\") files are more likely to contain regressions\n- **Code smells** \u2014 functions over 50 lines, classes with too many methods, duplicated logic\n\n---\n\n### Smart False-Positive Filtering\n\nIt won't cry wolf on known-safe patterns. For example, it automatically suppresses [P2]:\n- `eval()` inside test fixture strings\n- JavaScript `regex.exec()` calls (not Python's `exec()`)\n- Broad exceptions marked with `# INTENTIONAL:` or `# noqa: BLE001`\n\n---\n\n### How to Run It\n\n```\n/bug-predict \n```\n[P3]\n\n---\n\n### When to Use It\n\n- Before merging a large PR\n- During code review to focus attention on real risks\n- After inheriting unfamiliar code\n- Before a release to verify no new high-severity patterns crept in [P2]", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 24, "unsupported_claims": 0, "total_claims": 24, "generate_ms": 6873.589957947843, "judge_ms": 9204.070540959947 } ] }, { "variant": "anti_prior", "total_queries": 15, "precision_at_1": 0.7333333333333333, "recall_at_k": 0.8666666666666667, "mean_faithfulness": 0.9504761904761906, "refusal_rate": 0.0, "hallucination_rate": 0.3333333333333333, "mean_generate_ms": 6667.941299942322, "mean_judge_ms": 8246.679044608027, "runs": [ { "id": "gq-001", "query": "how do I run a security audit", "difficulty": "easy", "variant": "anti_prior", "answer": "## Running a Security Audit\n\nYou have a few ways to do this:\n\n### Option 1: Natural language\n```\nscan src/ for security issues\n```\n\n### Option 2: Direct skill command\n```\n/security-audit src/\n```\n\n### Option 3: CLI workflow\n```\nattune workflow run security-audit --path \"src/\"\n```\n\n---\n\n### What you can scan\n\n| Command | What it scans |\n|---------|---------------|\n| `/security-audit src/auth.py` | One file |\n| `/security-audit src/` | A directory tree |\n| `/security-audit .` | The whole project |\n\n---\n\n### What you get back\n\nResults are grouped by severity with file links, line numbers, and CWE identifiers. For example:\n\n```\nCritical\n src/hooks/executor.py:89 \u2014 eval() on user input (CWE-95)\n\nHigh\n src/config.py:203 \u2014 Path not validated (CWE-22)\n```\n\n---\n\n### After the audit\n\n- **Fix issues** \u2014 ask *\"fix the critical findings\"*\n- **Generate tests** \u2014 ask *\"write tests for the flagged files\"*\n- **Go deeper** \u2014 ask *\"deep scan src/auth/\"*\n- **Full reference** \u2014 run `attune help-docs ref-skill-security-audit` or say *\"tell me more\"*", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 17, "unsupported_claims": 0, "total_claims": 17, "generate_ms": 5222.434625029564, "judge_ms": 5733.310166979209 }, { "id": "gq-002", "query": "generate tests for my code", "difficulty": "easy", "variant": "anti_prior", "answer": "I can help you generate tests! Based on the smart-test skill, here's how to get started:\n\n## Quickest Way\n\nSince you haven't specified a target yet, I have two quick questions:\n\n1. **Which file or module needs tests?**\n - A single file (e.g. `src/auth/login.py`)\n - A directory (e.g. `src/auth/`)\n - Or say \"what needs testing?\" to find the weakest spots across the whole project\n\n2. **What approach do you want?**\n\n| Approach | What you get | Time |\n|----------|-------------|------|\n| **Gap analysis** | List of untested functions ranked by risk | Seconds |\n| **Generate tests** | Working pytest files with edge cases | ~1-2 min per module |\n| **Both** | Gaps first, then tests for the top gaps | ~2-3 min |\n\n---\n\n## Skip the Questions\n\nIf you give me both details upfront, it runs immediately \u2014 no back and forth:\n\n```\n/smart-test src/auth/\n```\nor just say:\n```\ngenerate tests for src/auth/\n```\n\n---\n\n## What You'll Get\n\nThe generated tests will include:\n- Working **pytest functions** with assertions\n- **Edge cases** (empty inputs, None, boundary values)\n- **Error path tests** for exceptions\n- **Parametrized tests** for input combinations\n\n**What's the path to the code you want tested?**", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 19, "unsupported_claims": 0, "total_claims": 19, "generate_ms": 6527.968791080639, "judge_ms": 8422.845457913354 }, { "id": "gq-003", "query": "fix failing tests", "difficulty": "easy", "variant": "anti_prior", "answer": "# Fix Failing Tests\n\nI can help you fix failing tests! To get started, I need a couple of details:\n\n1. **Which test is failing?** \u2014 Provide a file, a specific test name, or should I find failures automatically?\n2. **When did it break?** \u2014 Did this start failing after a recent change, or has it been broken for a while?\n\n---\n\n## Skip the questions\n\nIf you already know the target, just tell me directly and I'll jump straight to diagnosis:\n\n```\nfix tests/unit/test_auth.py\n```\n\nOr with a specific test:\n\n```\n/fix-test tests/unit/test_auth.py::test_login_expired\n```\n\n---\n\n## What happens next\n\nOnce you specify a target, here's the flow:\n\n1. **Diagnosis** \u2014 The failing test is run and the traceback is classified into a root cause category\n2. **Repair + retry** \u2014 A fix is applied and the test is re-run (up to **3 attempts**)\n3. **Report** \u2014 You'll see which tests were fixed, which still fail, how many attempts were used, and what to try next\n\n---\n\n**Which test should I fix?**", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 10, "unsupported_claims": 0, "total_claims": 10, "generate_ms": 4793.558083008975, "judge_ms": 8324.980583041906 }, { "id": "gq-004", "query": "review code quality", "difficulty": "easy", "variant": "anti_prior", "answer": "I'll help you run a code quality review! Before starting, I have two quick questions to scope the work:\n\n---\n\n**1. Which files or directory should I review?**\n\nFor example:\n- A single file: `src/config.py`\n- A directory: `src/auth/`\n- The whole project: `.`\n\n---\n\n**2. How deep should the review go?**\n\n| Depth | What it covers | Time |\n|-------|---------------|------|\n| **Quick** | Style and formatting only | Seconds |\n| **Standard** | + Logic errors, likely bugs | ~1 min |\n| **Deep** | + Security, architecture, test gaps | ~3 min |\n\n*(Default is **Standard** if you're not sure.)*\n\n---\n\nIf you'd like to skip the questions, just tell me both upfront \u2014 for example: **\"deep review of src/auth/\"** \u2014 and it will run immediately.", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 12, "unsupported_claims": 0, "total_claims": 12, "generate_ms": 4900.445292005315, "judge_ms": 6923.051167046651 }, { "id": "gq-005", "query": "deep review my PR", "difficulty": "easy", "variant": "anti_prior", "answer": "## Deep Review Your PR\n\nTo run a deep review, you need to specify the **path** to the files you want reviewed. Here are your options:\n\n### Option 1: Use the tool directly\n```\ndeep_review(path=\"src/\")\n```\n\n### Option 2: Use natural language\n```\ndeep review of src/\n```\n\n### Option 3: Use the skill\n```\n/code-quality src/\n```\n\n---\n\n### What \"Deep\" includes\n\n| Depth | What runs | Time |\n|-------|-----------|------|\n| **Deep** | Style + Logic errors + **Security, architecture, test gaps** | ~3 min |\n\nThis is the most thorough option \u2014 best for pre-release or critical modules like a PR review.\n\n---\n\n### What I need from you\n\n**Which files or directory should I review?**\n\nFor a PR, you'd typically point it at the directory or files you've changed, for example:\n\n- `src/auth/` \u2014 a specific module\n- `src/` \u2014 the full source tree\n- `src/api/handler.py` \u2014 a single file\n\nOnce you give me the path, the deep review will run immediately without any further questions.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.9166666666666666, "supported_claims": 11, "unsupported_claims": 1, "total_claims": 12, "generate_ms": 6396.138458978385, "judge_ms": 8354.05012499541 }, { "id": "gq-006", "query": "predict bugs before they happen", "difficulty": "medium", "variant": "anti_prior", "answer": "# Bug Prediction\n\nYou can predict bugs before they happen using the `/bug-predict` skill. Here's how to get started:\n\n## Quick Start\n\n```\n/bug-predict src/\n```\n\nOr just use natural language:\n\n```\npredict bugs in src/\n```\n\n---\n\n## What It Scans For\n\nThe scanner looks for three categories of risk:\n\n| Pattern | Severity | What It Catches |\n|---------|----------|-----------------|\n| **dangerous_eval** | \ud83d\udd34 HIGH | `eval()`, `exec()`, `compile()` on user input \u2014 code injection risks |\n| **broad_exception** | \ud83d\udfe1 MEDIUM | Bare `except:` or unlogged exceptions \u2014 silently swallowed errors |\n| **incomplete_code** | \ud83d\udd35 LOW | TODO, FIXME, HACK, XXX comments \u2014 unfinished code paths |\n\nIt also weighs **contextual risk factors** like:\n- Cyclomatic complexity (deeply nested logic)\n- Change frequency (frequently modified \"hot\" files)\n- Code smells (long functions, too many methods, duplicated logic)\n\n---\n\n## Choose Your Scope\n\n| Command | What It Scans |\n|---------|---------------|\n| `/bug-predict src/auth.py` | One file |\n| `/bug-predict src/` | A directory tree |\n| `/bug-predict .` | The whole project |\n\n---\n\n## Example Output\n\n```\nBug Prediction Report\nRisk Score: 73/100 | Files: 34 | Findings: 8\n\nHIGH (2 findings)\n src/hooks/executor.py:89 dangerous_eval eval() on user input\n src/plugins/loader.py:142 dangerous_eval exec() in plugin loader\n\nMEDIUM (3 findings)\n src/api/webhook.py:67 broad_exception bare except: masks errors\n ...\n```\n\n---\n\n## Next Steps After Scanning\n\n- **Fix HIGHs first** \u2014 ask *\"fix the dangerous_eval in executor.py\"*\n- **Generate tests** \u2014 ask *\"write tests for the flagged files\"*\n- **Go deeper** \u2014 ask *\"predict bugs in src/auth/\"* for a focused scan\n- **Track trends** \u2014 run weekly to monitor whether risk scores improve\n\n---\n\n**Want the full reference?** Say **\"tell me more\"** for complete pattern, scoring, and configuration details.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 26, "unsupported_claims": 0, "total_claims": 26, "generate_ms": 9971.99112502858, "judge_ms": 10180.708750034682 }, { "id": "gq-007", "query": "refactor my code", "difficulty": "medium", "variant": "anti_prior", "answer": "## Refactoring Your Code\n\nTo run a refactor analysis, use the `/refactor-plan` command:\n\n```\n/refactor-plan \n```\n\nThis will scan your code and build a **prioritized roadmap** of structural issues to fix.\n\n---\n\n### What it analyzes\n\n| Category | What it finds |\n|----------|---------------|\n| **Code smells** | Long methods, god classes, feature envy, data clumps |\n| **Duplication** | Copy-pasted blocks, near-duplicates, DRY violations |\n| **Complexity** | High cyclomatic complexity, deep nesting, long chains |\n| **Coupling** | Circular imports, tight dependencies, shotgun surgery |\n| **Naming** | Abbreviations, generic names, inconsistent conventions |\n| **Dead code** | Unreachable branches, unused params, vestigial modules |\n\n### How issues get prioritized\n\nItems are ranked by **severity**, **effort**, **impact**, and **risk** \u2014 so high-severity, low-effort, high-impact fixes float to the top. Risky changes get flagged so you can plan extra testing.\n\n---\n\n### Good times to run this\n\n- A module feels hard to change or test\n- You're about to add features to a tangled area\n- A code review flagged complexity hotspots\n- You need data to justify refactoring time to stakeholders\n\n---\n\n### Want more detail?\n\n- **\"tell me more\"** \u2014 step-by-step instructions\n- **\"what is code quality?\"** \u2014 broader code review options\n- **\"tell me about security audit\"** \u2014 vulnerability scanning instead", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 18, "unsupported_claims": 0, "total_claims": 18, "generate_ms": 8787.27854101453, "judge_ms": 7657.44287497364 }, { "id": "gq-008", "query": "prepare a release", "difficulty": "medium", "variant": "anti_prior", "answer": "# Preparing a Release\n\nHere's how to kick off release prep:\n\n## Fastest way\n\nJust say what you want in natural language:\n\n```\nis this project ready to release?\n```\n\nOr invoke the skill directly:\n\n```\n/release-prep check\n```\n\n## If you know your target version\n\n```\n/release-prep 5.6.0\n```\n\n---\n\n## What happens next\n\nRelease prep will ask you two quick questions before running:\n\n1. **What version?** \u2014 Your target version, or it can check the current version and suggest the next bump\n2. **What stage?** \u2014 Full release prep, or a specific check area\n\nIf you say *\"just check everything\"*, it runs the full assessment.\n\n---\n\n## What gets checked\n\n| Check Area | What It Verifies | Blocks Release? |\n|---|---|---|\n| **Health** | Tests pass, lint clean, coverage above threshold | Yes, if tests fail |\n| **Security** | No new CVEs, no eval/exec, secrets scan clean | Yes, if critical findings |\n| **Changelog** | Entry exists for this version, date is current | Yes, if missing |\n| **Dependencies** | Pinned versions, no vulnerabilities | Yes, if vulnerable dep found |\n| **Version** | Semver bump matches changes, dist builds cleanly | Yes, if version mismatch |\n\n---\n\n## Reading your result\n\nYou'll get a **GO** or **NO-GO** verdict with a structured report. If there are blockers, they'll be listed explicitly so you know exactly what to fix. After fixing, just say **\"re-run the check\"** to try again.\n\nOnce you have a **GO**, say **\"tag and publish\"** to proceed.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 20, "unsupported_claims": 0, "total_claims": 20, "generate_ms": 7764.089542091824, "judge_ms": 8718.880250002258 }, { "id": "gq-009", "query": "audit documentation for staleness", "difficulty": "medium", "variant": "anti_prior", "answer": "## Auditing Documentation for Staleness\n\nBased on the context, attune has a built-in tool for exactly this purpose.\n\n### The `doc_audit` Tool\n\nThe **Doc Audit** tool audits existing documentation for:\n\n- **Staleness**\n- **Broken links**\n- **Drift from source code**\n\nIt belongs to the **workflow** group of tools.\n\n### Usage\n\n```\ndoc_audit()\n```\n\n### Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `path` | string | `.` | Project root path |\n\nBy default it runs against the current directory (`.`), but you can point it at a specific project root path.\n\n---\n\n### What the Context Doesn't Cover\n\nThe context provides the tool's signature and purpose, but does **not** detail:\n\n- What the output report looks like\n- How \"drift from source code\" is detected\n- Whether there are depth levels (like the security audit has quick/standard/deep)\n- A CLI equivalent command (like `attune workflow run ...`)\n\nIf you need those details, you may want to consult the source file referenced in the docs: `src/attune/mcp/tool_schemas.py`.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.7333333333333333, "supported_claims": 11, "unsupported_claims": 4, "total_claims": 15, "generate_ms": 6009.103666990995, "judge_ms": 8941.657459014095 }, { "id": "gq-010", "query": "plan a new feature", "difficulty": "hard", "variant": "anti_prior", "answer": "## Planning a New Feature with Attune\n\nBased on the context, you can use the `/planning` skill for high-level development planning, including feature planning.\n\n### Quick Usage\n\n```\n/planning \n```\n\nFor a new feature specifically, you'd run something like:\n\n```\n/planning feature\n```\n\n**Result:** You'll get structured results directly in your Claude Code conversation.\n\n---\n\n### Want More Details?\n\nThe quickstart points to a full reference doc for deeper guidance:\n\n```\nattune help-docs ref-skill-planning\n```\n\nThat will give you the complete reference for the `/planning` skill.\n\n---\n\n> **Note:** The context provided covers the basics of the `/planning` command but doesn't detail the full step-by-step output or all available options. For the most complete information, the `ref-skill-planning` help doc (referenced above) would be the authoritative source.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 0.875, "supported_claims": 7, "unsupported_claims": 1, "total_claims": 8, "generate_ms": 5220.724207931198, "judge_ms": 7273.964083986357 }, { "id": "gq-011", "query": "vulnerability scan", "difficulty": "hard", "variant": "anti_prior", "answer": "## Running a Vulnerability Scan with Attune\n\nAttune has a **security audit** feature that scans your codebase for vulnerabilities. Here are two ways to run it:\n\n---\n\n### Option 1: CLI Workflow\n\n```\nattune workflow run security-audit --path \"src/\"\n```\n\n**Result:** Severity-grouped findings with CWE identifiers.\n\n---\n\n### Option 2: Claude Code Skill\n\n```\n/security-audit \n```\n\n**Result:** Structured results directly in your Claude Code conversation.\n\n---\n\n## What It Detects\n\n| Category | Examples |\n|----------|---------|\n| **Code injection** | `eval()`, `exec()`, `compile()` on untrusted input |\n| **Path traversal** | File operations without path validation |\n| **Hardcoded secrets** | API keys, tokens, passwords in source |\n| **SQL/command injection** | String concatenation in queries or shell commands |\n| **SSRF** | HTTP requests to user-controlled URLs |\n| **Weak cryptography** | MD5/SHA1 for security, hardcoded IVs |\n\n## Scan Depth Options\n\n| Depth | Time | What You Get |\n|-------|------|-------------|\n| **Quick** | ~30s | Surface scan \u2014 eval/exec, obvious secrets |\n| **Standard** | ~2 min | Full pattern matching with severity ratings |\n| **Deep** | ~5 min | Multi-pass review with OWASP mapping and fix suggestions |\n\n---\n\n**After scanning:** Fix critical issues, then consider running `attune workflow run test-gen`.", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 15, "unsupported_claims": 0, "total_claims": 15, "generate_ms": 5427.771290997043, "judge_ms": 6243.702959036455 }, { "id": "gq-012", "query": "write documentation for my module", "difficulty": "hard", "variant": "anti_prior", "answer": "# Generating Documentation for Your Module\n\nThe **doc-gen skill** can write documentation for your module directly from your source code. Here's how to get started:\n\n## Invoke the Skill\n\n```\n/doc-gen \n```\n\nOr use natural language, for example:\n```\ndocument src/models/\nadd docstrings to auth.py\ngenerate a README for the cli module\n```\n\n---\n\n## What Happens Next\n\nBefore generating anything, doc-gen will ask you **two scoping questions**:\n\n1. **What to document** \u2014 a single file, a module, or a directory tree\n2. **What format** \u2014 choose from:\n\n| Doc Type | What You Get |\n|----------|-------------|\n| **Docstrings** | Google-style per function/class/method |\n| **README** | Feature list, usage examples, install instructions |\n| **API reference** | Full signatures with types and descriptions |\n| **Module overview** | Architecture summary, dependency map |\n| **Full pipeline** | Audit gaps and generate all of the above |\n\n---\n\n## What It Reads From Your Code\n\nDoc-gen analyzes your actual source to produce accurate docs:\n\n- Function signatures, parameter names, types, and defaults\n- Type hints and return types\n- Class hierarchies and abstract methods\n- `__all__` exports (your public API surface)\n- Existing docstrings (preserves them, fills gaps)\n- Decorators and exception handlers\n\n---\n\n## After It Generates\n\nOnce you see the results, you can say things like:\n\n- **\"apply these to the files\"** \u2014 write the docstrings to disk\n- **\"audit the whole project for gaps\"** \u2014 check what's still missing\n- **\"generate a README for this module\"** \u2014 switch to a different doc type\n- **\"export the docs as a markdown file\"** \u2014 save the output\n\n---\n\nWhat kind of documentation are you looking to generate \u2014 docstrings, a README, an API reference, or a full audit?", "retrieval_top1_match": true, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 23, "unsupported_claims": 0, "total_claims": 23, "generate_ms": 7154.069374897517, "judge_ms": 7388.438749941997 }, { "id": "gq-013", "query": "orchestrate documentation workflow", "difficulty": "hard", "variant": "anti_prior", "answer": "## Running a Documentation Workflow\n\nBased on the context, there are two documentation-related workflows available:\n\n| Workflow | What it does | Typical time |\n|----------|-------------|-------------|\n| **Doc Audit** | Documentation freshness and gap analysis | ~1 min |\n| **Doc Generation** | Generates docs from source code | ~2 min |\n\n---\n\n### To run it, you can use either approach:\n\n**Plain English:**\n```\nrun doc audit and doc generation on \n```\n\n**Explicit command:**\n```\n/workflow-orchestration docs\n```\n\n---\n\n### What to expect\n\nYou'll receive a **combined report** with findings grouped by severity, showing an overall score and a summary per workflow. For example, a doc-focused run would surface:\n\n- Documentation gaps or stale docs (Doc Audit)\n- Generated documentation from your source code (Doc Generation)\n\n---\n\n### What you can do after\n\n- **Fix gaps** \u2014 \"fix the critical doc findings\"\n- **Go deeper** \u2014 \"deep review src/your-module/\"\n- **Add more workflows** \u2014 \"add code-review to the results\"\n- **Export** \u2014 \"export the report as JSON\"\n\n---\n\n**Need more detail?** Say `attune help-docs ref-skill-workflow-orchestration` for the full reference, or ask me which path or files you'd like to analyze.", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 0.875, "supported_claims": 14, "unsupported_claims": 2, "total_claims": 16, "generate_ms": 6171.3583749951795, "judge_ms": 9702.38783408422 }, { "id": "gq-014", "query": "look for dangerous eval calls", "difficulty": "hard", "variant": "anti_prior", "answer": "## Scanning for Dangerous eval Calls\n\nBased on the context, here's how to run that scan and what to expect:\n\n### Run the scan\n\n```\n/bug-predict src/\n```\n\nOr naturally:\n```\nfind risky code with eval usage\n```\n\n---\n\n### What you'll get\n\nThe scanner uses a pattern called **`dangerous_eval`** that flags `eval()` and `exec()` usage. These show up as **HIGH severity** findings. Example output:\n\n```\nHIGH (2 findings)\n src/hooks/executor.py:89 dangerous_eval eval() on user input\n src/plugins/loader.py:142 dangerous_eval exec() in plugin loader\n```\n\n---\n\n### \u26a0\ufe0f Important Warning\n\nThere is a **verified false positive** you should know about:\n\n> The scanner's regex matches `create_subprocess_exec` as containing `exec`, which produces a **false positive** for `dangerous_eval` in `hooks/executor.py`.\n\n**Mitigation:** Always verify HIGH severity `dangerous_eval` findings against the actual source code before treating them as real vulnerabilities. Don't assume every flag is a genuine `eval()`/`exec()` risk.\n\n---\n\n### Next steps after reviewing findings\n\n- Ask **\"fix the dangerous_eval in [file]\"** for a guided fix\n- Ask **\"write tests for the flagged files\"** to prevent regressions\n- Run **`security_audit(path=\"src/\")`** for a broader security audit beyond just eval patterns", "retrieval_top1_match": false, "retrieval_topk_match": false, "faithfulness_score": 0.8571428571428571, "supported_claims": 12, "unsupported_claims": 2, "total_claims": 14, "generate_ms": 7370.608750032261, "judge_ms": 10785.81487503834 }, { "id": "gq-015", "query": "sniff out hard-to-catch bugs", "difficulty": "hard", "variant": "anti_prior", "answer": "## Sniffing Out Hard-to-Catch Bugs\n\nThe **`/bug-predict`** skill is built exactly for this. It scans your codebase for patterns that historically cause production incidents \u2014 before they surface as real failures.\n\n### How to run it\n\n```\n/bug-predict \n```\n\n---\n\n### What it catches\n\nThese are the patterns it specifically hunts for:\n\n| Pattern | Severity | What it means |\n|---|---|---|\n| `eval()`, `exec()`, `compile()` on input | **HIGH** | Code injection vectors |\n| Bare `except:` or unlogged `except Exception:` | **MEDIUM** | Errors silently swallowed \u2014 bugs that vanish without a trace |\n| TODO, FIXME, HACK, XXX comments | **LOW** | Unfinished code paths that break under edge cases |\n\n---\n\n### Beyond pattern matching\n\nIt also weighs signals that *increase* the likelihood a file will have bugs:\n\n- **Cyclomatic complexity** \u2014 deeply nested conditionals and long functions\n- **Change frequency** \u2014 \"hot\" files modified often are regression-prone\n- **Code smells** \u2014 functions over 50 lines, too many methods per class, duplicated logic\n\n---\n\n### Smart noise reduction\n\nIt filters out known false positives automatically, such as:\n- `eval()` inside test fixture strings\n- JavaScript `regex.exec()` calls (not Python's `exec()`)\n- Broad exceptions intentionally marked with `# INTENTIONAL:` or `# noqa: BLE001`\n\n---\n\n### Best times to run it\n\n- Before merging a large PR\n- After picking up unfamiliar code\n- Before a release, to verify no new HIGH-severity patterns crept in\n- As a periodic health check on high-churn modules", "retrieval_top1_match": false, "retrieval_topk_match": true, "faithfulness_score": 1.0, "supported_claims": 15, "unsupported_claims": 0, "total_claims": 15, "generate_ms": 8301.579375052825, "judge_ms": 9048.950333031826 } ] } ]