Counterfactual audit of LLM résumé scoring

Frontier LLMs are already being handed résumés to score. Across 11 models, 28,050 screenings, and three experiments, one picture holds up. LLM résumé bias is real and reasoned, but small, unstable, idiosyncratic, and not fixable by prompting. Changing a single demographic line does move scores, and the score follows the model's written reasoning instead of decorating a number picked in advance. Most of that movement stays inside run-to-run noise, the models rarely agree on whom to penalise, and the one model with a stable, statistically clear bias is Qwen. Reordering the prompt or adding worked examples barely moves the needle. The three experiments below each pin down one part of that.

One row per model, joining all four experiments. Bias is the average 1-10 score change when one demographic line on the résumé changes. % sig is the share of those changes big enough to rule out run-to-run randomness. Instability is how much the score moves across repeat runs of an identical input, in score points. Coherence is the correlation between the score and the model's own stated factors, from -1 to 1. Responsiveness is how far the score follows an opinion pasted into it, where 1.0 would mean it followed all the way. Parse agreement is the share of extracted fields that match when the same document is read twice. Parse leakage is how many extracted fields move on a demographic swap, beyond what repeat readings move anyway. Eleven models is a small sample, so read this as a fingerprint and not a law.

Claude Fable 5 appears in the audit only. It is excluded from the transplant and prompt-lab experiments, so its follow-up columns are blank.

Each dot is a model. Horizontal is how biased it is (from the audit), vertical is how far its score follows transplanted reasoning. The upward tilt is the throughline. The models that let reasoning drive the score most are also the most biased, with Qwen furthest out on both.

Pearson r is about 0.61 across 10 models. That fits the chain from identity to reasoning to score, though it is not statistically significant at this sample size.

The first of the three experiments, in full. This is the demographic bias audit the fingerprint above is built from.

A fair evaluator scores a résumé on its merits. Change only a demographic signal on it, the candidate's name, country, alma mater, or a gap in employment, and leave every other word untouched, and the score should not move. The optimistic assumption we put to the test is that frontier LLMs already behave this way, that they are effectively identity-blind and swapping a name leaves the verdict intact.

A counterfactual audit. Hold the résumé, job and model fixed, change one demographic line, and watch the score.

Step 1. Take one real résumé as the baseline and generate variants that each differ from it by a single demographic line. For example, the baseline candidate becomes Maria Rodriguez for a first-name swap, or the address country becomes India, with every other word left identical.

Step 2. Score the baseline and every variant with the same model on the same job, several times each. For example, all 11 models score the 30 résumé variants across 17 jobs, repeated over multiple runs.

Step 3. For each variant measure the delta, its mean score minus the baseline's, and check whether that gap clears the run to run noise. For example, a model that scores the baseline 6.2 and the India variant 5.6 has dropped the score by 0.6.

Step 4. Pool the deltas two ways. By model, to see which is least even-handed, and by dimension, to see which swapped signal moves scores the most. The two tables below are exactly those two views.

Step 5. Read it. If demographic swaps leave the score flat, the model is identity-blind and the assumption holds. If the score moves, and moves the same way across runs and across models, the evaluator is not judging the work alone.

When the only change is United States (the Company Locations axis), Gemini 2.5 Flash shifts its score by -3.40 on the role Junior / Mid-Level Fullstack Developer.

See this counterfactual →

Each row is one model. We measure how far that model's score moves, on average, when we swap a single demographic signal on the résumé. The higher the number, the less even-handed the model. "Most penalised" and "most rewarded" call out the single variant that swung scores furthest in each direction.

Same data, grouped by what we changed instead of who did the changing. The mean |Δ| pools every model, variant, and job for each demographic axis. The axis at the top is the one models react to most reliably.

The identity-blind assumption does not hold, but the failure is messier than "the models are biased". Scores do move when only a demographic line changes. The most sensitive model, Qwen 3 Next 80B, shifts its score by 0.405 per swap on average, and the signal that moves scores the most is First Name at 0.261. No model here is truly identity-blind.

What keeps this short of clean discrimination is that only about 4% of individual score shifts clear the run to run noise floor, and the models barely agree on direction (mean pairwise correlation +0.07, near zero). That reads as instability more than stable prejudice. Each model is an unstable judge whose number wobbles when any part of the input changes, which is its own reason not to hand it a hiring decision unsupervised. The methodology covers the noise floor, and the heatmap the model-by-model agreement.