Training a 14B to patch authorization bugs

Security is largely absent from the loop that trains coding models. Training rewards output that works, so a model learns to ship a feature even when the feature leaves an exploit behind. VibeSec exists to put that signal back: every task is a working app with a proven vulnerability, and a patch only counts when a real exploit stops firing and the app's own tests still pass.

The obvious question is whether training on that signal actually moves a model. It does, and by more than we expected.

The result

We fine-tuned Qwen3-14B-Instruct on 800 VibeSec V1 tasks and evaluated on the 200 it never saw.

+72 percentage points, on tasks held out of training.

Secure-patch rate · 200 held-out tasks

Every model is scored on the same 200 tasks. The exploit must stop firing and the app’s spec test must still pass, graded by execution in a sandbox with no LLM judge.

Qwen3-14B (SFT on VibeSec)

78.0%

Claude Opus 4.8

57.5%

Claude Sonnet 4.6

33.5%

Kimi K2.7 Code

32.0%

GLM 5.2

30.0%

Nemotron 3 Ultra

26.0%

GPT-OSS 120B

9.0%

Mistral Medium 3.5

8.0%

Qwen3-14B (base)

6.0%

How a patch is graded

Nothing here is scored by a model. Each task carries four executable gates, and a task only enters the dataset when all four hold:

1. The generated app boots and passes its own spec test.
2. A real exploit runs against it and prints PWNED.
3. A candidate patch makes that exact exploit fail.
4. The patched app still passes the original spec test.

A model scores only on gates 3 and 4 together. That pairing is the whole design. Closing a hole is easy if you are allowed to break the feature — delete the endpoint and no exploit fires. Requiring the spec to keep passing is what makes the reward mean fixed rather than removed.

Grading runs in a Modal sandbox against the real process, so the reward is a program executing, not an opinion about code.

Why the comparison is on 200 tasks, not 1,000

The fine-tuned model can never be scored on the full 1,000-task set, because 800 of those tasks are its training data. Testing on them would be testing on train.

That constraint decides the methodology. The held-out split is the only ground where the fine-tune and an untrained model can be compared at all, so every model in the chart above is scored on those same 200 tasks — not on the public 1,000.

This matters more than it sounds. The held-out split is harder than the full set for every model we tested, by 2 to 7 points:

The split was built group-aware — grouping by seed prompt and unioning near-duplicates — which strips the repeated, easier variants and leaves a harder remainder. Quoting a fine-tune's held-out score next to a frontier model's full-set score would quietly credit the frontier model for 800 tasks the fine-tune never saw. It is the kind of comparison that looks generous and is not.

Paired outcomes, not two percentages

Because every model saw identical tasks, the outcomes are paired, and the informative quantity is the disagreements: how often does the fine-tune fix a task the other model leaves exploitable, and how often the reverse.

Against Claude Opus 4.8, on the same 200 tasks:

Those 18 are worth stating plainly. This is not domination. There is a set of tasks a frontier model handles and a specialised 14B does not, and any honest read of this result has to include it.

What the split correction cost

An earlier run of this experiment reported 86.0%. It was measured on a split where 60 of the 200 test tasks shared a seed prompt with training data — a 30% leak.

The base model moved one point, which is just a different 200 tasks — it was never trained, so leakage cannot flatter it. The SFT arm dropped eight. That eight is the size of the inflation, and it is why 86.0% is not the number we publish.

The gain is on security, not formatting

The reasonable objection to any result like this is that the model simply learned the output format, and the apparent security gain is really a parsing gain. The failure-mode breakdown rules that out:

spec_broken is exactly identical, 10 to 10, while exploit_still_works collapses from 178 to 34. The base model could already emit valid, spec-passing patches; it just failed to secure them. Of the 139 tasks that flipped to passing, every one moved out of exploit_still_works. The entire gain sits on the security gate.

Limitations

• Same distribution. Every test app comes from the same generator: two files, roughly 3 KB, FastAPI, in-memory dicts. The similarity analysis rules out near-neighbour lookup; it does not rule out distribution shift. Cross-distribution evaluation on CVE-Bench is the open question, and until it is run, "learned security" is a claim about this distribution.
• Specialist against generalists. The fine-tune was trained on this exact task format and output format. The frontier models are zero-shot. The comparison shows what training on this data buys, not that a 14B is better at security than Opus.
• The format/security decomposition is inferred, not measured. The control arm that trains on the vulnerable app with correct formatting has not been run. The flat spec_broken is strong evidence, not a measurement.
• Single seed, single run. No variance estimate. Splits for seeds 43 and 44 are built and unused.
• General capability is unmeasured. Final training loss was 0.0079, which is very low. Whether the model now over-applies 403s to benign code is untested.
• The seven vulnerability classes behave like one. V1 is 68.8% broken object-level authorization, and the per-class gains move together. Treat this as a depth probe of one class rather than a survey of seven.

Reproducing it

The dataset, all 1,000 tasks and every model outcome are public. The comparison above is generated from a script rather than typed in, so it cannot drift from the eval:

python3 scripts/build_sft_comparison.py \
  --research ../muence-research \
  --factory  ../muence-factory

Browse the tasks, exploits and reference patches in the data browser, or take the raw files from the dataset page.