# Tutorial materials

**Building and Verifying AI Systems with Agentic AI**
Tutorial at IEEE CISOSE 2026, Fukuoka, Japan — Chia-Kai Chang, National Central University.

Everything promised on stage, in one place. Nothing here needs an account, a key, or a
signup.

| # | What | File |
|---|---|---|
| 01 | **Runnable LLM-as-Judge harness** — two real rubrics, a synthetic golden set, runs offline | `judgekit/` · `judgekit.zip` |
| 02 | **Six-layer checklist** — the self-assessment sheet, printable | `six-layer-checklist.md` |
| 03 | **Spec template** — the five parts, the twelve sections, and a blank form | `spec-template.md` |
| 04 | **A real spec, redacted** — an actual production spec, with the redactions marked | `real-spec-redacted.md` |
| 05 | **Governance template** — consent and data handling, each clause paired with its test | `governance-template.md` |

Slides: [uedu.tw/cisose-2026-tutorial/slides/](https://uedu.tw/cisose-2026-tutorial/slides/)

---

## Start here, in 60 seconds

```bash
cd judgekit
python judge.py run \
    --rubric rubrics/dialogue_quality.v3.toml \
    --set    goldensets/dialogue.jsonl \
    --models mock:strict,mock:lenient
```

Python 3.11+, no network, no API key, no third-party packages. Then read the
disagreement queue rather than the mean score.

To see the version guard refuse to run — the part worth thirty seconds — edit one word in
the rubric file and run the same command again.

---

## What these are, and what they are not

They are one team's working artifacts, cleaned up enough to hand over. They are not a
framework, and there is nothing to adopt wholesale. The checklist is a map of blind spots,
the templates are prompts for questions you should answer yourself, and the harness is
small on purpose so that reading all of it is realistic.

Three things in here are worth more than the rest:

1. **The golden set outlives the prompt.** Models change; a good evaluation set does not.
   Five items on day one, plus one every time production surprises you.
2. **A judge encodes a policy, not a truth.** The two shipped rubrics score the same
   transcripts to different standards, and both are correct. Ask what standard yours
   encodes, and whether that is the one you promised.
3. **Find three clauses an agent could not have inferred.** If your spec has none, it is a
   prompt, and what it produces cannot be reviewed because there is nothing to review it
   against.

---

## Data ethics

No real student data ships in any of these files. The transcripts in the golden set were
written to reproduce failure shapes observed in production, not copied from it. The
redacted spec and governance template mark what was removed and why.

That is also the rule worth taking home: **an evaluation set you cannot publish is an
evaluation set your collaborators cannot check.** If yours contains real user data, build
a synthetic mirror of it for exactly this reason.

---

## Licence

Harness: MIT (`judgekit/LICENSE`). Documents: reuse freely, attribution welcome; adapt to
your own institution's requirements rather than adopting as-is.

If useful in published work:

> C.-K. Chang, *Building and Verifying AI Systems with Agentic AI*, tutorial at
> IEEE CISOSE 2026, Fukuoka, Japan.

Questions: ckchang@uedu.tw