Extraction Lab
The assumption under test. That the instability and bias in the first three experiments come from asking a model to judge. Use it only to parse the résumé into structured fields, score those fields in deterministic code, and the problem should disappear by construction. The reasoning-to-score link the transplant proved is severed, because there is no longer a score for reasoning to reach.
One parser prompt, no job description, run repeatedly over every résumé variant. The model is asked for structure and closed-vocabulary labels only. Every rank, total and presence check is done in code.
- Ask the model to copy the résumé into a fixed set of fields. No score, no opinion of the candidate. Each job becomes an entry with an employer, dates, a seniority label from a fixed list, and a quote from the document proving the entry is real. The model never sees the job description, so it cannot judge relevance to anything.
- Parse the identical document five times over. Nothing has changed between those runs, so any field that comes back different changed for no reason. That gives us the noise floor.
- Parse each variant résumé, then compare it against the baseline parse. Swap the name to Aisha Okonkwo. The name field is allowed to change. If the job titles or the technologies change too, that is leakage.
- Only count a change as leakage if it is bigger than the noise floor from step 2. If two runs of the same document already disagree on three fields, then three fields of difference after a name swap proves nothing.
- Check the parse against the document itself, where a right answer exists. The résumé has 15 jobs, so a parse returning 13 is wrong. If the text says Kubernetes and the parse does not, that is a miss. If a quote is not in the document, the model invented it.
- Read it. A parse that holds still can be scored by code. A parse that wobbles cannot. If the same document parses differently twice, the instability was never in the scoring step, and moving the scoring into code fixes nothing.
Agreement is the share of extracted field paths matching across repeat runs of an identical résumé, so 1.000 means the parse never moved. Tier 1 is transcribed and classified fact, tier 2 is the judgement fields. Ordinal drift is the mean rank distance on ordered labels such as seniority. Net leakage is how far a demographic swap moved fields it had no business touching, after subtracting what repeat runs move anyway.
| model | arm | cells | agreement | tier 1 | tier 2 | ordinal drift | grounded | net leakage |
|---|---|---|---|---|---|---|---|---|
| claude-fable-5 | default | 30 | 0.923 | 0.918 | 0.970 | 1.43 | 1.000 | 1.15 |
| claude-haiku | default | 30 | 0.877 | 0.881 | 0.813 | 1.04 | 0.999 | 1.42 |
| claude-opus | default | 30 | 0.908 | 0.900 | 0.958 | 1.37 | 1.000 | 1.64 |
| claude-sonnet | default | 30 | 0.933 | 0.932 | 0.937 | 2.29 | 0.998 | 0.76 |
| gemini-2.5-flash | temp0 | 30 | 1.000 | 1.000 | 1.000 | 0.00 | 0.999 | 21.14 |
| gemini-2.5-flash | temp07 | 30 | 0.884 | 0.880 | 0.897 | 1.14 | 0.999 | 0.86 |
| gemini-2.5-pro | temp0 | 30 | 0.984 | 0.985 | 0.963 | 0.47 | 1.000 | 3.54 |
| gemini-2.5-pro | temp07 | 30 | 0.948 | 0.950 | 0.906 | 1.00 | 1.000 | 1.88 |
| gemini-3.1-pro-preview | temp0 | 30 | 0.982 | 0.982 | 0.976 | 0.34 | 1.000 | 6.06 |
| gemini-3.1-pro-preview | temp07 | 30 | 0.941 | 0.940 | 0.925 | 0.88 | 1.000 | 1.48 |
| llama-4-maverick | temp0 | 30 | 0.974 | 0.976 | 0.953 | 0.73 | 1.000 | 2.23 |
| llama-4-maverick | temp07 | 30 | 0.960 | 0.963 | 0.919 | 0.96 | 0.999 | 1.42 |
| mistral-large | temp0 | 30 | 0.972 | 0.971 | 0.975 | 0.75 | 0.969 | 1.37 |
| mistral-large | temp07 | 30 | 0.950 | 0.948 | 0.962 | 0.83 | 0.969 | 1.02 |
| mistral-small | temp0 | 30 | 0.928 | 0.924 | 0.956 | 1.07 | 0.985 | 3.66 |
| mistral-small | temp07 | 30 | 0.849 | 0.847 | 0.848 | 1.09 | 0.938 | 0.44 |
| qwen-3-next-80b | temp0 | 30 | 0.715 | 0.725 | 0.791 | 0.88 | 0.990 | 20.95 |
| qwen-3-next-80b | temp07 | 30 | 0.705 | 0.713 | 0.808 | 1.02 | 0.991 | 11.97 |
What the results say about the assumption. Extraction is steadier than scoring, but it is not still. Agreement across repeat runs of an identical résumé ranges from 0.705 (qwen-3-next-80b) to 1.000 (gemini-2.5-flash). At temperature 0, where the parse should be reproducible by construction, agreement runs from 0.715 to 1.000, and 1 of 7 model arms reach a clean 1.000. Every field that changes between two readings of the same document is a field that anything built on top would have read differently the second time.
Collected over 2700 parses across 11 models and 30 résumé variants.