← Tutorial page  ·  All materials  ·  Markdown source

Tutorial materials

Building and Verifying AI Systems with Agentic AI Tutorial at IEEE CISOSE 2026, Fukuoka, Japan — Chia-Kai Chang, National Central University.

Everything promised on stage, in one place. Nothing here needs an account, a key, or a signup.

# What File
01 Runnable LLM-as-Judge harness — two real rubrics, a synthetic golden set, runs offline judgekit/ · judgekit.zip
02 Six-layer checklist — the self-assessment sheet, printable six-layer-checklist.md
03 Spec template — the five parts, the twelve sections, and a blank form spec-template.md
04 A real spec, redacted — an actual production spec, with the redactions marked real-spec-redacted.md
05 Governance template — consent and data handling, each clause paired with its test governance-template.md

Slides: uedu.tw/cisose-2026-tutorial/slides/

Start here, in 60 seconds

cd judgekit
python judge.py run \
    --rubric rubrics/dialogue_quality.v3.toml \
    --set    goldensets/dialogue.jsonl \
    --models mock:strict,mock:lenient

Python 3.11+, no network, no API key, no third-party packages. Then read the disagreement queue rather than the mean score.

To see the version guard refuse to run — the part worth thirty seconds — edit one word in the rubric file and run the same command again.

What these are, and what they are not

They are one team's working artifacts, cleaned up enough to hand over. They are not a framework, and there is nothing to adopt wholesale. The checklist is a map of blind spots, the templates are prompts for questions you should answer yourself, and the harness is small on purpose so that reading all of it is realistic.

Three things in here are worth more than the rest:

  1. The golden set outlives the prompt. Models change; a good evaluation set does not. Five items on day one, plus one every time production surprises you.
  2. A judge encodes a policy, not a truth. The two shipped rubrics score the same transcripts to different standards, and both are correct. Ask what standard yours encodes, and whether that is the one you promised.
  3. Find three clauses an agent could not have inferred. If your spec has none, it is a prompt, and what it produces cannot be reviewed because there is nothing to review it against.

Data ethics

No real student data ships in any of these files. The transcripts in the golden set were written to reproduce failure shapes observed in production, not copied from it. The redacted spec and governance template mark what was removed and why.

That is also the rule worth taking home: an evaluation set you cannot publish is an evaluation set your collaborators cannot check. If yours contains real user data, build a synthetic mirror of it for exactly this reason.

Licence

Harness: MIT (judgekit/LICENSE). Documents: reuse freely, attribution welcome; adapt to your own institution's requirements rather than adopting as-is.

If useful in published work:

C.-K. Chang, Building and Verifying AI Systems with Agentic AI, tutorial at IEEE CISOSE 2026, Fukuoka, Japan.

Questions: [email protected]

CISOSE 2026 tutorial materials · Chia-Kai Chang, National Central University · uedu.tw