Exercise sec-prompt-injection — a red-team eval harness + a tool-call allow-list

Score the provided mock model against the LLM red-teaming harness's 40-attack fixture, build a markdown report from the result, and write a fail-closed allow-list for the tool calls an agent built on that model might make. This mirrors the actual job task: shipping a red-team suite whose report a reviewer can trust run to run, and a tool layer that drops a malformed call instead of executing it. It runs on your Mac, not in the browser.

~120 minruns locally · uv + pytest 6 checkssec-prompt-injection

What you're building

src/sec_prompt_injection/harness.py — detect_canary_leak(response, canary=CANARY_TOKEN): True if the canary token appears anywhere in the response, case-folded. run_harness(attacks, model=query): score every attack, return the total, the leak count, the rate, the sorted leaking ids and a per-category breakdown — calling it twice on the same fixture must return an equal result. build_report(result): the markdown report those numbers render into.
src/sec_prompt_injection/policy.py — allowlist_tool_calls(calls, policy): keep only the calls whose tool name is in policy AND whose arguments exactly match that tool's schema — fail closed, a name-only check is not enough.

The tests are ordinary pytest and ship in the public folder with the starter — read them first; the names below are the check list. Solutions are not published.

Get the repo (once)

Needs git. uv installs the right Python itself, so nothing else is required.

# once, anywhere on your machine
git clone https://github.com/theDocWho/ai-ml-roadmap.git
cd ai-ml-roadmap

No git? Download the ZIP, unzip it, and cd into the unzipped folder instead.

Run it

From the repo root:

# one-time: uv (https://docs.astral.sh/uv/) manages the venv and pins Python ≥ 3.12
cd exercises/sec-prompt-injection && uv sync && uv run pytest -q

Done when uv run pytest -q prints 6 passed. The untouched starter fails all 6 — every function is ....

The checks

test_run_harness_computes_attack_success_rate_on_the_fixture
test_run_harness_is_deterministic
test_build_report_has_expected_markdown_shape
test_detect_canary_leak_folds_case_and_requires_the_token
test_allowlist_tool_calls_blocks_disallowed_and_passes_allowed
test_allowlist_tool_calls_rejects_calls_that_fail_the_argument_schema

The fixture is fixed, so the rate is a known number: 12 of 40 attacks leak the canary token (4 of 10 override, 5 of 10 exfiltration, 3 of 10 roleplay, 0 of 10 encoding — those ten are obfuscated and never contain a literal trigger phrase, so the mock model always refuses them). The determinism check calls run_harness twice and requires an equal result; the two allow-list checks each isolate one failure mode — a disallowed name (exec_shell with {"cmd": "rm -rf /"} is dropped while search with {"query": "capital of France"} passes through unchanged), and an allowed name with the wrong argument shape (search with {"query": 123} or with no query at all — both dropped).

Files

exercises/sec-prompt-injection/README.md — this brief, offline
pyproject.toml — deps (pytest)
fixtures/attacks.json — the 40-attack fixture every test reads
src/sec_prompt_injection/mock_model.py — provided, not edited
src/sec_prompt_injection/harness.py — the starter you edit
src/sec_prompt_injection/policy.py — the starter you edit
tests/conftest.py — the fixture data (attacks, tool_policy)
tests/test_harness.py, tests/test_canary.py, tests/test_policy.py — the checks

I ran the tests locally and they pass (6 passed).

This is self-attestation — the site cannot see your terminal, so the box and the button are you telling The Path the suite went green on your machine.

If you get stuck

The detector never sees the trigger list — detect_canary_leak only looks at the model's output, never at what phrase caused it. That black-box shape is what makes it the same kind of check a harness pointed at a real model would run.
Fold case on both sides — canary.lower() in response.lower(), never canary in response.
Fail closed — allowlist_tool_calls should read as "keep only what I can prove is safe," not "drop only what I can prove is dangerous." A wrong-typed or missing argument on an otherwise-allowed tool must still be dropped.
Reading a red row — uv run pytest -x --tb=short stops at the first failure and prints the assert message, which names the attack id or call it computed.