# judgekit — a small LLM-as-Judge harness Companion artifact for the CISOSE 2026 tutorial **Building and Verifying AI Systems with Agentic AI** (Chia-Kai Chang). It exists to make three claims checkable rather than asserted: | Claim | How this repo makes it checkable | |---|---| | **runnable** | one command, no network, no API key, no third-party packages | | **versioned prompts** | a rubric is a file with a version and a content hash; change it without bumping the version and the harness *refuses to run* | | **cross-model reproducibility** | every run writes a record pinning rubric hash, model ids, parameters and every raw reply | The headline output is **not** a mean score. It is the list of items the judges disagree on — with each other and with your human labels. Agreement is a summary; disagreement is a work queue. --- ## 60-second start (offline) ```bash python judge.py run \ --rubric rubrics/dialogue_quality.v3.toml \ --set goldensets/dialogue.jsonl \ --models mock:strict,mock:lenient ``` No key, no install, nothing to sign up for. Python 3.11+ (for `tomllib`; on 3.9/3.10 run `pip install tomli`). Then swap in real models: ```bash export OPENAI_API_KEY=... # and/or ANTHROPIC_API_KEY pip install openai anthropic python judge.py run \ --rubric rubrics/dialogue_quality.v3.toml \ --set goldensets/dialogue.jsonl \ --models openai:gpt-4o-mini,anthropic:claude-haiku-4-5 ``` --- ## The four commands ```bash python judge.py check rubrics/dialogue_quality.v3.toml [--show-prompt] python judge.py pin rubrics/dialogue_quality.v3.toml python judge.py run --rubric R --set S --models M1,M2 [--limit N] [--out DIR] python judge.py report runs/.json ``` `check` validates the rubric and reports its pin status. `--show-prompt` prints the system prompt the harness will actually send — read it before you trust a score. --- ## What is in the box ``` judge.py CLI judgekit/rubric.py rubric loading, validation, version pinning judgekit/providers.py mock / openai / anthropic adapters, one return shape judgekit/runner.py golden set x judges -> run record judgekit/report.py agreement statistics + the disagreement queue rubrics/ two real rubrics (redacted) + the pin lock file goldensets/ a ten-item synthetic set + a starter template example-run.json one complete run record, so you can see the shape before running runs/ your own run records land here ``` Open `example-run.json` before you run anything. It is what "reproducible" means in practice: the rubric id and content hash, the model ids and parameters, and every raw reply — enough for someone else to tell whether your numbers came from what you say they came from. ### Two rubrics, one golden set, on purpose `rubrics/dialogue_quality.v3.toml` and `rubrics/dialogue_participation.v1.toml` judge **the same transcripts** to **different standards**, and they are both correct. The strict one asks "is this good dialogue?". The lenient one asks "is this a real person engaging honestly?". Both shipped in production, because the strict rubric had been used as the pass mark and returned a 0–38% pass rate — which contradicted what students had been promised. The fix was not to loosen the strict rubric; it was to add a second question and keep the first intact as a higher tier. Run both over `goldensets/dialogue.jsonl` and compare. **A judge encodes a policy, not a truth** — that is the single most useful thing in this repository. Note the `negative_instructions` block in the lenient rubric. Every line in it was added because the judge had been observed failing an item for that exact reason. ### The version guard ``` $ python judge.py run --rubric rubrics/dialogue_quality.v3.toml ... REFUSING TO RUN -- rubric drift rubric 'dialogue_quality' version 'v3' is pinned to 75003a6086bd but this file hashes to 29a083291bf3. ``` Editing a judge prompt without bumping its version silently invalidates every score you have ever recorded under that name. This is the one error the harness will not let you make. `--allow-drift` exists; do not use it. ### The mock judges are honest about what they are `mock:strict` and `mock:lenient` are **not models**. They are a transparent heuristic over turn length, filler ratio, causal markers and repetition, with a downward or upward bias. They exist so the harness runs when the conference wifi does not. They are still instructive. Run the offline demo and look at item `dlg-05` — a fluent, well-formed pasted essay that never answers the tutor. Both heuristic judges score it highly; the human label is near zero. **That is length-and-fluency bias, reproduced offline, with no model involved** — which tells you the bias is a property of shallow evaluation, not of any particular vendor. --- ## Statistics Dependency-free and deliberately plain: - **exact agreement** — how often two raters gave the identical value - **mean |d|** — average distance on the ordinal scale - **qwk** — quadratic-weighted kappa; chance-corrected, penalises far misses more than near ones - **kappa** — Cohen's kappa, for boolean verdict rubrics Two judges that rank items the same way but at different levels will show **low exact agreement and high kappa**. That is systematic bias, and it is fixable by rewriting the rubric. Low kappa is noise, and it is not. If a golden set item carries a `human` field, `human` joins the comparison as another rater automatically. Calibrating a judge against a person is the only way to know whether it measures what you think it measures. --- ## Writing your own 1. Copy `goldensets/_starter.jsonl`. Five items is enough on day one: three representative, two adversarial. Add one item every time production surprises you. 2. Copy a rubric, change the `id`, set `version = "v1"`, run `judge.py pin`. 3. Run with two models. Read the disagreement queue **before** touching the prompt. 4. When you edit the rubric, bump the version. Always. The set matters more than the prompt. Models change; a good evaluation set outlives all of them. --- ## Data ethics No real student data ships here. The transcripts in `goldensets/dialogue.jsonl` were written to reproduce failure shapes seen in production, not copied from it. That is also the rule worth taking home: **an eval set you cannot publish is an eval set your collaborators cannot check.** If yours contains real user data, build a synthetic mirror of it for exactly this reason. --- ## Licence and citation MIT — see `LICENSE`. Use it, change it, ship it. If it is useful in published work, cite the tutorial: > C.-K. Chang, *Building and Verifying AI Systems with Agentic AI*, tutorial at > IEEE CISOSE 2026, Fukuoka, Japan.