AI Eval Harness
The AI eval harness measures how well the 15 AI analyzers find what they should and stay quiet when they should. It replays recorded LLM responses for a small labelled corpus of merge requests, scores the findings against the labels and fails the build when precision, recall or F1 drop below a checked-in baseline.
It lives in the plugins/integration-tests/ai-eval module (commonTest, so it runs on JVM and native) as part of jvmTest and
macosArm64Test/linuxX64Test. The default mode is offline and deterministic: no provider credentials, no
network. Live mode is JVM-only in practice: native test tasks blank KARANDASH_EVAL_LIVE.
Running
./gradlew :plugins:integration-tests:ai-eval:jvmTest
Each run writes two reports:
plugins/integration-tests/ai-eval/build/reports/ai-eval/report.json: precision, recall, F1 and TP/FP/FN per analyzer and overall, plus every miss (false positive or false negative) with its case and analyzer.plugins/integration-tests/ai-eval/build/reports/ai-eval/summary.txt: the same numbers as a short table.
The gate test fails if any analyzer, or the overall score, is below the baseline in
plugins/integration-tests/ai-eval/src/commonTest/resources/eval/baseline.json. The failure message lists each
regressed metric next to the summary. A second test checks coverage: every analyzer must be scored and have at
least one positive label, and the corpus must contain at least one known-clean MR.
Layout
All data sits under plugins/integration-tests/ai-eval/src/commonTest/resources/eval/:
| Path | Content |
|---|---|
corpus/*.json |
One labelled MR per file (EvalCase) |
recorded/*.json |
Recorded LLM response per analyzer for the case with the same file name |
baseline.json |
The measured scores the gate compares against |
A corpus case holds the MR (title, description, labels, commit messages, files with unified diff lines), an
optional design doc (passed as extra instructions), optional prior findings (fed to fix-suggestions as
accumulated findings), the analyzers to run and the expected findings. A trimmed example:
{
"id": "01-example",
"title": "Add user lookup by email",
"files": [
{ "path": "src/main/kotlin/UserRepository.kt", "diff": ["@@ -10,3 +10,8 @@", "+fun findByEmail(email: String) ="] }
],
"analyzers": ["code-quality", "risk-scoring"],
"expected": [
{ "analyzer": "code-quality", "file": "src/main/kotlin/UserRepository.kt", "startLine": 12, "category": "security" },
{ "analyzer": "risk-scoring" }
]
}
An analyzer listed in analyzers with no expected entry for it must produce no findings: any finding is a
false positive. The clean cases rely on this.
Matching
Each finding matches at most one label, and each label at most one finding. A finding matches a label of the same analyzer when:
- File: if the label names a file, the finding carries a
FileRefto the same path (leading./and/ignored). If both sides have lines, the ranges must overlap within a 3-line window. A finding without a line matches on the file alone. - Category: if the label names a category, some finding metadata value equals it ignoring case, either whole
or as one item of a comma- or semicolon-separated list. This covers
category,dimension, suggested labels, risk level and so on.
A label with neither file nor category matches any finding of its analyzer. A further finding that would only match an already-matched label counts as a duplicate, not as a false positive.
Full-pipeline replay
EvalReplayTest also replays every case through the real validate pipeline (ValidationPipeline): config
loading and AI rule synthesis, the AI rule with prompt assembly, redaction and the interceptor chain, the AI
pipeline extension filters, markdown formatting and platform output. Only the LLM (a cassette interceptor) and the
platform API are fake. Each case runs in a fresh temp directory, so no cache leaks between runs.
Per case it captures the exact prompts after redaction, the findings left after filtering, the rendered summary,
the inline notes and stage counters. Golden snapshots sit in replay/<id>.md. The test also checks that two runs
give byte-identical prompts, summary and notes. The run log goes to build/reports/ai-eval/replay/<id>.log.
Two optional corpus keys feed the replay:
cassette: analyzer ID to the raw response text. It wins overrecorded/<id>.json, for instance for an analyzer that only the full pipeline calls.minedFrom: where a mined case came from (group/project!123). Informational only.
The baseline's pipeline section holds per-case notes, kept and dropped findings, silent drops and estimated
tokens (prompt characters / 4). The gate fails if notes grow more than 10%, estimated tokens more than 15%, or any
silent drop appears. A silent drop is an analyzer that sent no prompt, produced no finding and logged no skip
reason, or a cassette miss.
Updating the baseline
After an intentional change in analyzer behaviour, prompts or recordings, rewrite the baseline and the replay goldens from the current run, review the diff and commit it with the change:
./gradlew :plugins:integration-tests:ai-eval:jvmTest -Dkarandash.updateBaseline=true -Dkarandash.updateGolden=true
KARANDASH_EVAL_UPDATE_BASELINE=1 still works for the baseline.
Live mode and re-recording
Live mode sends the corpus to a real provider instead of replaying it. It never runs by default, and the gate is not enforced in live mode: it writes the report so you can compare live quality with the baseline.
| Variable | Effect |
|---|---|
KARANDASH_EVAL_LIVE=1 |
Call a real provider |
KARANDASH_EVAL_RECORD=1 |
With live mode, overwrite recorded/*.json with the live responses |
KARANDASH_EVAL_PROVIDER |
anthropic (default) or openai |
KARANDASH_EVAL_MODEL |
Model to use; defaults to the provider's default model |
KARANDASH_EVAL_BASE_URL |
Optional provider endpoint override |
ANTHROPIC_API_KEY / OPENAI_API_KEY |
Provider credentials |
KARANDASH_EVAL_LIVE=1 KARANDASH_EVAL_RECORD=1 ANTHROPIC_API_KEY=... \
./gradlew :plugins:integration-tests:ai-eval:jvmTest
KARANDASH_EVAL_UPDATE_BASELINE=1 ./gradlew :plugins:integration-tests:ai-eval:jvmTest
The second command re-scores the new recordings offline and writes the baseline. Review both diffs before
committing. The task is never up to date while KARANDASH_EVAL_LIVE is set, so Gradle does not reuse a cached
result.
!!! note The recordings checked in today are hand-authored seeds with deliberate imperfections, so the gate has something to measure. Re-record them against a real provider to get a baseline that reflects actual model quality.
Adding a case
- Add
corpus/NN-name.jsonwith the MR, the analyzers to run and the expected findings. - Record responses with live mode, or write
recorded/NN-name.jsonby hand. Itsresponsesobject maps each analyzer ID to the raw response, either a JSON object or a string. - Run the gate, check
summary.txtand update the baseline if the change is intended.