Project — Flagship 4: a LoRA fine-tune with a before/after eval harness

Every fine-tuning tutorial shows the training loss going down and stops there — which proves the optimizer works, not that the model got better at the task. This project's whole point is the number on either side of the training run: an eval harness that runs against the frozen base model before a single LoRA weight changes, the same harness run again after, and a table with both numbers next to each other. If you can't show that table, you haven't proven the fine-tune did anything — you've only proven the loss curve went down, which a bug can also do.

~8 hruns in a notebook · RTX 3060 / Colab 6 milestonesf4-lora

What you're building

A decontaminated dataset — a task dataset split into train and eval, checked with ex-minhash's find_contaminated so the eval set you report numbers on shares no near-duplicate with anything the model trains on.
An eval harness that runs before training exists — the same harness scores the base model first. Skipping this and only running the harness once, after training, is the single most common way this project's number stops meaning anything: there is no "before" to compare "after" to.
A LoRA run with its hyperparameters logged — HF PEFT, rank r, scaling alpha and the exact target_modules list written down next to the run, not left as whatever the library's default happened to be.
The same eval harness run again, after — same held-out set, same metric, same code path as the "before" run; only the weights underneath changed.
A before/after table — the two numbers from the same harness, side by side, in the adapter's model card and README.
A published adapter — LoRA weights only (not the merged model) in safetensors format, with a model card naming the base model, the dataset, the hyperparameters and the before/after table.

The milestones

Dataset + MinHash decontamination — the train/eval split exists, and ex-minhash's find_contaminated run over train against eval returns nothing above threshold. Log how many candidate eval examples it rejected, if any.
Eval harness before training — the harness runs end-to-end against the unmodified base model and produces the "before" number, committed before the LoRA run starts.
PEFT LoRA run, hyperparameters logged — r, alpha and target_modules (typically the attention projections, q_proj/ v_proj at minimum) recorded in the training script or a run config, not just in your head.
Eval harness after training — the identical harness, identical eval set, run again against the fine-tuned adapter, producing the "after" number.
Before/after table — both numbers, same metric, in one table. A fine-tune that made the number worse is still a valid milestone here — it means the harness is real, not cherry-picked to only ever show improvement.
Adapter published — the LoRA adapter's weights (safetensors, not a full merged checkpoint) pushed to a public repo with a model card: base model, dataset summary, r/alpha/target_modules, and the before/after table from milestone 5.

Suggested build order

Assemble the task dataset and split it first, then run ex-minhash's decontamination check before writing a single line of training code — an eval set discovered to be contaminated after training is a wasted run.
Write the eval harness next and point it at the base model. This is deliberately before LoRA exists at all: if the harness cannot produce a number from the untouched base model, it will not produce a trustworthy one from the fine-tuned model either.
The LoRA run, with r/alpha/target_modules written into the script before the first launch, not reconstructed afterward from memory.
Re-run the exact same harness against the adapter. Same code path as step 2 — a rewritten or "improved" harness for the after-run breaks the comparison.
The before/after table and the model card come last, once both numbers exist.

All 6 milestones above are done: dataset decontaminated, before-eval committed, LoRA run's hyperparameters logged, after-eval committed, before/after table written, adapter published with a model card.

This is self-attestation — the site cannot see your notebook or your Hub repo, so the box and the button are you telling The Path the adapter, the model card and both eval numbers are real.

If you get stuck

MinHash threshold — start at the 0.8 ex-minhash default and lower it if your dataset is short-document (tweets, single sentences), where even unrelated pairs can share more shingles by chance than a paragraph-length corpus would. Log whatever threshold you settle on next to the count it rejected.
Which eval metric — pick whatever fits the task (exact-match, F1, a small rubric-scored set) but use the same one before and after; switching metrics between the two runs is equivalent to not having a before number at all.
target_modules — if you are not sure which projections your base model exposes, print model.named_modules() once and read the attention block's actual names before guessing; a mistyped module name silently attaches to nothing and PEFT will not error, it will just train zero adapter weights on that layer.
safetensors, not a merged checkpoint — publish the adapter's own small weight file (typically tens of MB) via model.save_pretrained(...) on the PEFT model, not merge_and_unload()'s multi-GB full model — the model card should let someone else load the same base model and apply your adapter on top of it.