# A real spec, redacted Companion handout for the CISOSE 2026 tutorial **Building and Verifying AI Systems with Agentic AI** — Chia-Kai Chang, National Central University. This is the actual specification for a production feature, not one written for a talk. It was drafted on 2026-07-06 and the first phase was implemented on 2026-07-07 — one afternoon of writing, roughly the same again of building. The feature is a **blind model-comparison arena**: a student asks one question, two anonymous models answer side by side, the student votes, and only then are the two models revealed. It exists to move a student from *consumer* of one AI to *judge* of several. **Read it for the shape, not the subject.** The point is which clauses had to be written down because no agent could have inferred them. --- ## What was removed, and why Redaction is part of the method, so it is marked rather than hidden. | Removed | Reason | |---|---| | Full `CREATE TABLE` statements | Internal schema detail; the fields that matter to the teaching points are summarised in §7. | | Internal hostnames and GPU inventory | Operational detail, no teaching value, mild disclosure risk. | | Source file layout and blueprint names | Specific to one codebase. | | Two clauses in §3 and §5 — a roster exclusion policy, and an enumerated contested factual claim | Both are specific to one jurisdiction's politics. **The mechanism is kept in full; only the enumerated instance is removed.** Marked `[REDACTED]` in place. | Nothing was added, softened, or rewritten to look better after the fact. Where the spec was vague, it is still vague. --- ## §1 Goal and teaching claim Core claim: a single AI reads as an oracle. Several AIs — from different origins, with different value commitments, disagreeing with each other — force a student from "accept the answer" to "judge which answer is better, and say why". **That shift is itself AI literacy.** Four-philosophy roster, one model each: | Philosophy | Label | Origin type | |---|---|---| | `commercial` | commercial frontier | large US vendor | | `community` | community capability | national university lab | | `sovereign` | national sovereignty | state-funded programme | | `open_science` | open science | European public research consortium | Research output: votes plus free-text reflection become AI-literacy signals; preferences on value-laden prompts become governance data. In-platform and authenticated, so it falls under the existing umbrella ethics approval rather than needing its own. **Stated red line, verbatim from the spec:** the rating is *student preference*, not an objective capability benchmark, and it must never be described as equivalent to one. --- ## §2 Flow ``` ① entry (four philosophies explained + first-time research consent) → choose mode: free (own prompt) | curated (prompt bank) ② prompt ③ POST /battle/start server: draw 2 from the active roster → randomise slot (A/B) AND screen side → create battle(status=streaming) → return battle id + two ANONYMOUS slots ④ SSE /battle//stream call both models in parallel, multiplex tokens tagged by slot → both columns stream at once. Self-disclosure suppressed + filtered (see §4) ⑤ vote (opens only after both columns finish) verdict: left | right | tie | BOTH BAD, plus optional "why did you pick it" ⑥ reveal + rating update → front-end flips the cards: "you chose the community model; the other was commercial" ⑦ reflect / play again ``` --- ## §3 Roster and availability — **the arena deliberately does not fall back** This is the clause most worth reading. > **與一般對話不同：一般對話允許 fallback 回其他供應商；Arena 絕不 fallback——fallback 會污染「這顆模型實際表現」的比較。** > > *Unlike ordinary chat, which is allowed to fall back to another provider, the arena > never falls back. A fallback would contaminate the comparison — you would be rating a > model that did not answer.* Instead of falling back, the spec requires a **health gate**: probe each model before the draw and exclude unhealthy ones from that round. If a locally hosted model is offline the roster simply shrinks; the arena keeps working. If a stream fails mid-answer, that response is marked `error` and the student may vote "both bad" or discard the round without it counting. `[REDACTED — one clause stating a roster exclusion policy and its reasoning.]` **Why this is the example:** the same engineering instinct — "always degrade gracefully, always have a fallback" — is correct everywhere else in this system and produces a measurement error here. No coding agent infers that from the goal. It comes from knowing what the numbers will be used for. --- ## §4 Blind-test integrity Observed in testing: some models state their own name and developer unprompted. That alone destroys the blind comparison. Two layers, because an instruction is not enforcement: 1. **System-prompt suppression (primary).** Every arena model is told: *you are an AI assistant in an anonymous comparison; do not reveal your name, developer, model family or country of origin; answer the question directly.* 2. **Stream redaction (secondary).** A configurable blocklist of model and vendor names is masked in the token stream as `[model name hidden]`. 3. **Position randomisation.** Slot and screen side are randomised every round, and model identity is held server-side only — never sent to the client before the vote. Layer 1 is a request to a system that is free to ignore it. Layer 2 is the enforcement. Writing both down is the difference between a policy and a hope. --- ## §5 Sensitive-topic policy Stance: present plural perspectives, and **do not reproduce specific distortions**. The clause that matters architecturally: > The policy is written at the **platform** layer. It is **not** delegated to any single > model's built-in alignment. `[REDACTED — the spec enumerates one specific contested factual claim that must not be presented as fact. The enumerated instance is jurisdiction-specific; the requirement that such instances be enumerated *in the spec*, rather than left to a vendor's alignment, is the transferable part.]` Prompts touching values or politics are tagged, and those rounds are retained as governance data. Framing to students is "how do AI systems with different value commitments answer this", never "which political position is correct". --- ## §6 The maths, written out - Start 1500, K = 32 (K may be lowered as a model accumulates games). - Expected score `E_a = 1 / (1 + 10^((R_b − R_a) / 400))`. - Actual `S_a`: left win = 1, right win = 0, tie = 0.5. - **"Both bad" does not update the rating — it is recorded only.** It is not a relative-preference signal; it is a failure signal. - Update `R_a' = R_a + K·(S_a − E_a)`, symmetric for `R_b`. Every change is appended to a rating-history table. - The maths lives in a pure module so that the scheduled job, the admin console and the live path cannot drift apart. That fourth bullet is the third un-inferrable clause. Fold "both bad" into the score and the ranking quietly stops meaning what its label says. --- ## §7 Data model (summarised) Five tables: the roster with current rating, one row per battle, one row per response, one vote per battle, and an append-only rating history. Conventions from the codebase standard: UTC timestamps, binary UUIDs, soft delete. Fields worth noting because they exist for measurement rather than for the feature: | Field | Why it is there | |---|---| | `ttft_ms`, `latency_ms` | Does answer speed influence preference? Unanswerable later if not captured now. | | `decision_ms` | Time from "both ready" to vote. | | `reflection_text` | The free-text reason. Treated as sensitive; de-identified on export. | | `verdict = both_bad` | Stored distinctly so it can be excluded from the rating and counted on its own. | --- ## §8–§9 API and front-end Endpoints for start, stream, vote, leaderboard, prompt bank, history, and two admin routes. Response envelope and rate limiting follow existing codebase conventions. The front-end section exists for one reason: **so the agent reuses the design language instead of inventing a new one.** It names the brand colours, the sanitiser configuration required for rendering model-generated markdown, the loading indicator, and the responsive behaviour of the two columns. Cheap to write, and it removes an entire category of review comment. --- ## §10 Measurement mapping Each captured signal is mapped, in the spec, to the research construct it serves and the exact field it comes from. Doing this **before** implementation is what prevents the familiar situation where the analysis needs a field nobody stored. --- ## §11 Ethics and approval - In-platform and authenticated → covered by the existing umbrella approval; no separate amendment needed. - First entry shows a research-data explanation and consent, and **keeps a "play without taking part in the research" option**. - Free-text reflections are treated as sensitive; research export is de-identified and goes through the single approved export route. - External wording: the rating is preference, not capability. That some models are weaker is deliberate teaching material, not a defect. --- ## §12 Phasing - **Phase 1 (shipped):** both modes, two models, vote, reveal, rating, leaderboard, self-identification suppression, position randomisation. - **Phase 2:** two further models, reflection text, per-category voting, rating history charts, value-laden prompt bank. - **Phase 3:** classroom mode, research export, preference data used for fine-tuning. Writing down what deliberately does **not** ship is what stops phase 1 from quietly growing into phase 3. --- ## The three clauses again If you take one thing from this document, take the exercise: after writing your spec, find three clauses an agent could not have inferred. 1. §3 — never fall back, *because* a fallback is a measurement error here even though it is good practice everywhere else. 2. §4 — suppress self-identification, *because* it was observed happening, and enforce it in the stream rather than trusting the instruction. 3. §6 — "both bad" does not move the rating, *because* it is a different kind of signal. Three judgements about what the thing is for. That is the part that did not get cheaper. --- *Released with the tutorial materials. Reuse freely; attribution welcome.*