Prompt Lab

The assumption under test. That the instability and bias in LLM résumé screening are mostly a prompt problem, fixable with better prompt engineering. The sharpest version, argued widely, is that the naive prompt asks for the score first and writes everything after it to justify a number already chosen. Make the model reason first and decide the score last, and the results should turn markedly more stable, coherent and fair.

Six prompt strategies, all emitting the identical output schema. Only the technique changes, and every strategy is scored on the same résumés, jobs and models.

1. Take the naive production prompt, then write five variations of it. Same output fields every time, only the technique changes. Score last moves the number to the end so the model writes its reasoning before committing. Few-shot prepends two worked examples. Competency rubric demands a piece of evidence per requirement before scoring.
2. Run all six prompts over the same résumés, jobs and models, five times each. Every prompt sees an identical workload, so the only difference between two runs of one cell is the technique and ordinary sampling noise.
3. Compute the same four measures for every prompt. Stability is how much the score moves across repeat runs of the same input. Coherence is whether the score matches the model's own stated reasons. Bias is how much the score moves on a demographic swap. Decision flips is how often the yes or no answer is not unanimous.
4. Compare each prompt against the naive baseline, one measure at a time. Few-shot lands a pooled score stdev of 0.52 against the baseline's 0.53. Score last pushes run to run decision flips from 0.33 up to 0.54.
5. A prompt only counts as a fix if it beats the baseline on one measure without damaging another. Few-shot improves stability and both flip rates at once. Score last leaves the score wobble unchanged and makes the yes or no call less repeatable, which is a trade, not a fix.

A note on the stability numbers. 33 of the 80 baseline cells score an identical number on every repeat run. Their stdev is exactly zero, so no alternative prompt can improve on them, only tie or lose. That makes a pooled comparison flatter the baseline. Wherever stability is judged below, the figure quoted is the paired difference over the 47 cells that had room to move.

Aggregated over 10 models. Lower is better for stability, bias and flip rates. Higher is better for coherence. Use the comparator below to inspect any pair head-to-head and see how each model reacts.

What the results say about the assumption. It is largely dismissed. Prompt wording moves the numbers only a little, and the most-hyped fix does not deliver. Reordering so the model decides the score last (the Score last strategy) leaves repeat-run score stability untouched. Pooled stdev reads 0.64 against the baseline's 0.53, but 33 of the 80 baseline cells already sit at exactly zero stdev and can only move upwards, and on the 47 cells with room to move the paired difference is −0.003 points (t = −0.06), a dead wash. What it does change is the decision. Run to run recommendation flips rose from 0.33 to 0.54, with 25 cells going from unanimous to split against only 8 the other way (McNemar χ² = 7.76, p ≈ 0.005), a comparison based on discordant pairs, so no floor effect explains it away. The one technique that helps across the board is Few-shot examples, and even it only trims score stdev to 0.52 and decision flips to 0.25, a small nudge. What would have supported the assumption, a strategy that sharply cut both the score wobble and the flips at once, never appeared. The wobble these models show is coming from the models themselves, not from how the question was worded. Rewording it does not help.