judgekit — a small LLM-as-Judge harness

Companion artifact for the CISOSE 2026 tutorial Building and Verifying AI Systems with Agentic AI (Chia-Kai Chang).

It exists to make three claims checkable rather than asserted:

The headline output is not a mean score. It is the list of items the judges disagree on — with each other and with your human labels. Agreement is a summary; disagreement is a work queue.

60-second start (offline)

python judge.py run \
    --rubric rubrics/dialogue_quality.v3.toml \
    --set    goldensets/dialogue.jsonl \
    --models mock:strict,mock:lenient

No key, no install, nothing to sign up for. Python 3.11+ (for tomllib; on 3.9/3.10 run pip install tomli).

Then swap in real models:

export OPENAI_API_KEY=...          # and/or ANTHROPIC_API_KEY
pip install openai anthropic

python judge.py run \
    --rubric rubrics/dialogue_quality.v3.toml \
    --set    goldensets/dialogue.jsonl \
    --models openai:gpt-4o-mini,anthropic:claude-haiku-4-5

The four commands

python judge.py check  rubrics/dialogue_quality.v3.toml [--show-prompt]
python judge.py pin    rubrics/dialogue_quality.v3.toml
python judge.py run    --rubric R --set S --models M1,M2 [--limit N] [--out DIR]
python judge.py report runs/<record>.json

check validates the rubric and reports its pin status. --show-prompt prints the system prompt the harness will actually send — read it before you trust a score.

What is in the box

judge.py                  CLI
judgekit/rubric.py        rubric loading, validation, version pinning
judgekit/providers.py     mock / openai / anthropic adapters, one return shape
judgekit/runner.py        golden set x judges -> run record
judgekit/report.py        agreement statistics + the disagreement queue
rubrics/                  two real rubrics (redacted) + the pin lock file
goldensets/               a ten-item synthetic set + a starter template
example-run.json          one complete run record, so you can see the shape before running
runs/                     your own run records land here

Open example-run.json before you run anything. It is what "reproducible" means in practice: the rubric id and content hash, the model ids and parameters, and every raw reply — enough for someone else to tell whether your numbers came from what you say they came from.

Two rubrics, one golden set, on purpose

rubrics/dialogue_quality.v3.toml and rubrics/dialogue_participation.v1.toml judge the same transcripts to different standards, and they are both correct.

The strict one asks "is this good dialogue?". The lenient one asks "is this a real person engaging honestly?". Both shipped in production, because the strict rubric had been used as the pass mark and returned a 0–38% pass rate — which contradicted what students had been promised. The fix was not to loosen the strict rubric; it was to add a second question and keep the first intact as a higher tier.

Run both over goldensets/dialogue.jsonl and compare. A judge encodes a policy, not a truth — that is the single most useful thing in this repository.

Note the negative_instructions block in the lenient rubric. Every line in it was added because the judge had been observed failing an item for that exact reason.

The version guard

$ python judge.py run --rubric rubrics/dialogue_quality.v3.toml ...
REFUSING TO RUN -- rubric drift

rubric 'dialogue_quality' version 'v3' is pinned to 75003a6086bd
but this file hashes to 29a083291bf3.

Editing a judge prompt without bumping its version silently invalidates every score you have ever recorded under that name. This is the one error the harness will not let you make. --allow-drift exists; do not use it.

The mock judges are honest about what they are

mock:strict and mock:lenient are not models. They are a transparent heuristic over turn length, filler ratio, causal markers and repetition, with a downward or upward bias. They exist so the harness runs when the conference wifi does not.

They are still instructive. Run the offline demo and look at item dlg-05 — a fluent, well-formed pasted essay that never answers the tutor. Both heuristic judges score it highly; the human label is near zero. That is length-and-fluency bias, reproduced offline, with no model involved — which tells you the bias is a property of shallow evaluation, not of any particular vendor.

Statistics

Dependency-free and deliberately plain:

• exact agreement — how often two raters gave the identical value
• mean |d| — average distance on the ordinal scale
• qwk — quadratic-weighted kappa; chance-corrected, penalises far misses more than near ones
• kappa — Cohen's kappa, for boolean verdict rubrics

Two judges that rank items the same way but at different levels will show low exact agreement and high kappa. That is systematic bias, and it is fixable by rewriting the rubric. Low kappa is noise, and it is not.

If a golden set item carries a human field, human joins the comparison as another rater automatically. Calibrating a judge against a person is the only way to know whether it measures what you think it measures.

Writing your own

1. Copy goldensets/_starter.jsonl. Five items is enough on day one: three representative, two adversarial. Add one item every time production surprises you.
2. Copy a rubric, change the id, set version = "v1", run judge.py pin.
3. Run with two models. Read the disagreement queue before touching the prompt.
4. When you edit the rubric, bump the version. Always.

The set matters more than the prompt. Models change; a good evaluation set outlives all of them.

Data ethics

No real student data ships here. The transcripts in goldensets/dialogue.jsonl were written to reproduce failure shapes seen in production, not copied from it. That is also the rule worth taking home: an eval set you cannot publish is an eval set your collaborators cannot check. If yours contains real user data, build a synthetic mirror of it for exactly this reason.

Licence and citation

MIT — see LICENSE. Use it, change it, ship it. If it is useful in published work, cite the tutorial:

C.-K. Chang, Building and Verifying AI Systems with Agentic AI, tutorial at IEEE CISOSE 2026, Fukuoka, Japan.