# The six-layer checklist

Companion handout for the CISOSE 2026 tutorial
**Building and Verifying AI Systems with Agentic AI** — Chia-Kai Chang, National Central University.

Print this. Take it to a system you own. Mark each row honestly. **Your biggest ✗ is
Monday morning's first task.**

---

## Score your own system

| # | Layer | Ask of your own system | ✓ △ ✗ |
|---|---|---|---|
| 1 | **Code** | Does it run, typecheck, and pass unit tests? | ☐ |
| 2 | **Pipeline** | Does it flow end to end with no silent loss? | ☐ |
| 3 | **Behavior** | Is it right on representative *and* adversarial input? | ☐ |
| 4 | **Guardrail** | Does it refuse unsafe input and resist prompt injection? | ☐ |
| 5 | **Governance** | Is the data consented, redacted, and auditable? | ☐ |
| 6 | **Drift** | Has it been re-checked across model versions? | ☐ |

`✓` solid · `△` partial · `✗` gap

---

## It is coverage, not a score

Six ✓ is not a grade and five ✓ is not 83%. The layers are not comparable in weight and
they are not substitutes for each other. The checklist answers one question only:

> **Which kinds of failure am I currently unable to detect?**

Read it as a map of blind spots. A system with excellent unit tests and nothing at layers
3–6 is not "one sixth done"; it is well covered against one failure mode and blind to
five.

---

## What each layer catches, and the method

| Layer | The failure it catches | Method |
|---|---|---|
| **Code** | It does not do what the code says. | Types, unit tests, review of intent rather than of lines. |
| **Pipeline** | It does what the code says, and the data never arrives. | End-to-end tests, ingestion counts, reconciliation between stages. |
| **Behavior** | Everything runs, and the answer is wrong. | LLM-as-Judge against a versioned rubric and a golden set. |
| **Guardrail** | The answer is right for a request that should have been refused. | Adversarial probes as a standing test, not a one-off audit. |
| **Governance** | It is correct, and you were not permitted to do it. | Testable clauses — see the governance template. |
| **Drift** | It was all true last month. | Re-run the golden set on every model or dependency change; monitor cost and volume. |

---

## The three questions the layers came from

Not from a framework. From three questions people actually ask:

- A colleague: **"does it work?"** → layers 1 and 2
- A manager: **"can we ship it?"** → layers 3 and 4
- An ethics committee: **"should we?"** → layers 5 and 6

Layered testing is an old idea and there is published work on it — the ML test-score
literature is the closest neighbour. This particular cut is not derived from it. It comes
from incidents, which is why the layer boundaries fall where they do rather than where a
taxonomy would put them.

---

## Why single-lens testing fails

Three real incidents, three different layers:

| Incident | Which layer would have caught it |
|---|---|
| A background job re-ran the same work over and over; the bill, not the tests, raised the alarm. | **Drift** — nothing was broken, the cost changed. |
| A rule written for whole messages met a system that delivered them in fragments. | **Pipeline** — each component behaved correctly on its own. |
| A judge scored honestly and consistently against a standard that was wrong for its purpose. | **Behavior** — the score was stable, valid, and measuring the wrong thing. |

**No single kind of testing catches more than one of these.** Unit tests were green in all
three. That is the argument for coverage over depth in any one place.

---

## The rows nobody fills in honestly

- **Layer 2** is marked ✓ far more often than it is true. "It works in production" is not
  the same as "no silent loss": silent loss is silent. Ask instead whether you count what
  goes in and what comes out, and compare them.
- **Layer 6** is the one most people have never done at all. If your model provider
  updated a version and you did not re-run anything, this row is ✗, not △.
- **Layer 5** is often marked ✓ on the strength of a document. The test is whether any
  clause in that document would fail a check if it stopped being true.

---

## After you mark it

1. Pick the single largest ✗.
2. Write the smallest check that would have caught one real past incident at that layer.
   Not a framework. One check.
3. Put it where it runs without you — CI, a scheduled job, an alert.
4. Re-mark the sheet in a month.

The value is not in the marks. It is in discovering that a row you assumed was ✓ is
actually △, and that you learned this from a sheet of paper rather than from an incident.

---

*Released with the tutorial materials. Reuse freely; attribution welcome.*