# sec-prompt-injection — a red-team eval harness + a tool-call allow-list

Score a provided naive mock model against a 40-attack prompt-injection fixture, build a markdown
report from the result, and write a fail-closed allow-list for the tool calls an agent built on
top of that model might make. The brief is the same text as the page
(`illustrated/10-security/exercise-prompt-injection-harness.html`); the concept — probes, a
black-box detector, a report — is walked first on `illustrated/10-security/llm-red-teaming.html`.

This one runs on your Mac, not in the browser — it's plain-stdlib Python, but the local track
certifies it for real rather than skipping it.

## What to implement

- **`src/sec_prompt_injection/harness.py` — `detect_canary_leak(response, canary=CANARY_TOKEN) ->
  bool`.** True if `canary` appears anywhere in `response`, **case-folded** — a real model
  re-cases whatever it copies out of its context, so an exact-case check misses a leak sitting
  right there in a different case.
- **`src/sec_prompt_injection/harness.py` — `run_harness(attacks, model=query) ->
  HarnessResult`.** Run every attack through `model`, score each response with
  `detect_canary_leak`, and return the total, how many leaked, the rate, the **sorted** list of
  leaking ids, and a per-category `{"attacks": n, "successes": n}` breakdown. Calling it twice on
  the same `attacks` must return an **equal** `HarnessResult` both times — nothing about the
  evaluation may depend on iteration or dict/set order.
- **`src/sec_prompt_injection/harness.py` — `build_report(result) -> str`.** A markdown report:
  the `# Prompt-injection harness report` heading, the exact rate as
  `**Attack success rate:** N/40 (P.P%)`, a per-category table sorted by category name, and a
  `## Successful attacks` section listing every leaking id, sorted.
- **`src/sec_prompt_injection/policy.py` — `allowlist_tool_calls(calls, policy) -> list[dict]`.**
  Keep only the calls whose tool name is a key of `policy` **and** whose arguments exactly match
  that tool's schema (no missing key, no extra key, every value the schema's declared type) —
  fail closed: a name-only check is not enough, a call that names an allowed tool but sends the
  wrong argument shape must still be dropped.

`src/sec_prompt_injection/mock_model.py` is **provided** — a small, deliberately naive model that
"complies" (and leaks `CANARY_TOKEN`) only when a prompt contains one of five literal trigger
phrases, case-insensitively. Ten of the fixture's forty attacks (the `encoding` category) never
contain a trigger phrase literally — they're obfuscated (base64, reversed, ROT13, homoglyphs) —
so this naive model never "sees" them and always refuses. That's not a bug to fix; it's the
harness's own blind spot, and the report should show it as `encoding | 10 | 0 | 0.0%`.

## Run it

```sh
cd exercises/sec-prompt-injection && uv sync && uv run pytest -q
```

`uv sync` installs `pytest` into `.venv/`. The untouched starter fails all 6 checks —
`detect_canary_leak`, `run_harness`, `build_report` and `allowlist_tool_calls` are all `...`.

## The checks (`tests/`)

- `test_run_harness_computes_attack_success_rate_on_the_fixture` — 12/40 (30.0%) on the real
  fixture: 4 of 10 `override` attacks, 5 of 10 `exfiltration`, 3 of 10 `roleplay`, 0 of 10
  `encoding`, and the exact set of 12 leaking ids.
- `test_run_harness_is_deterministic` — two calls on the same fixture return an equal
  `HarnessResult`.
- `test_build_report_has_expected_markdown_shape` — the heading, the exact rate line, the
  `encoding`/`exfiltration` table rows, and the sorted success-id list.
- `test_detect_canary_leak_folds_case_and_requires_the_token` — a re-cased leak is still
  detected; a clean refusal is not a leak.
- `test_allowlist_tool_calls_blocks_disallowed_and_passes_allowed` — an unlisted tool name is
  dropped (`exec_shell` with `{"cmd": "rm -rf /"}`), a valid call to a listed tool passes through
  unchanged (`search` with `{"query": "capital of France"}`).
- `test_allowlist_tool_calls_rejects_calls_that_fail_the_argument_schema` — a listed tool called
  with a wrong-typed or missing argument is still dropped.

Done when `uv run pytest -q` prints **6 passed**, then press "mark done" on the page.

## If you get stuck

- **The detector never sees the trigger list** — `detect_canary_leak` only looks at the model's
  *output*, never at what phrase caused it. That's what makes it a legitimate black-box detector,
  the same shape a harness pointed at a real model would use.
- **Fold case on both sides** — `canary.lower() in response.lower()`, not `canary in response`.
- **`run_harness` must be boring** — no `random`, no reliance on dict/set iteration order for
  anything the result exposes; sort `success_ids` before returning it.
- **Fail closed, not open** — `allowlist_tool_calls` should read as "keep only what I can prove is
  safe," never "drop only what I can prove is dangerous."
- **Reading a red row** — `uv run pytest -x --tb=short` stops at the first failure and prints the
  assert message, which names the attack id or call it computed and what it expected.