CV fairness benchmark

How the test works

We took 60 CVs from the generator behind our CV matching benchmark, 15 for each job: 9 true fits, who meet every must-have, and 6 near misses, who fall short on one. Each CV was written 15 times:

A name copy changes only the lines that carry the name: the name, the email address and the website. The two age copies differ from each other only in the date of birth line. All 900 went in as real applications to a separate test form, in a shuffled order, were read by Formzed like any CV, and were matched against each job's own requirements, 3,600 CV and job pairs in all.

Every copy of a CV carries the same experience, so a fair matcher gives every copy the same fit. The exact twins measure the noise, and every gap between names is judged against it.

Twelve names, one CV

For every CV we took the fit gap between each pair of its name copies, and between the CV and its exact twin. Two names were 0.36 points apart on average (interval 0.22 to 0.55); the same CV twice, 0.25 (0.14 to 0.43). Compared CV by CV, two names sat +0.11 points further apart than the twins, an interval of −0.01 to +0.26 that includes zero. 13% of the CVs got exactly the same fit under all twelve names.

The quick rating agrees: two names were 1.00 apart out of 10 on average, the same CV twice 1.07.

Origin and gender

Each copy's score is compared with the average of its own CV's name copies, so differences between CVs drop out and only the name is left. An origin whose names did better would sit right of zero. The largest average effect was 0.08 points on the fit and 0.13 on the quick rating, and the intervals of 6 of 6 origins include zero on the fit, 6 of 6 on the quick rating.

Women's copies scored −0.02 points against men's copies of the same CVs (interval −0.10 to +0.06), and −0.01 on the quick rating. In every job the interval includes zero.

Who makes the short list

The four-fifths rule, used in hiring audits, compares how often each group is picked: a group picked at under 0.80 of the best group's rate is a warning sign. At the cut that matters most, as many places as a job has true-fit copies, every qualified copy made it: 216 of 216 women's copies, 216 of 216 men's, at least 72 of 72 for every origin, and 36 of 36 for each date of birth. Every ratio was 1.00.

Short lists are a harder test. We counted who reached each job's top 10, 25, 50 and 100 places, all name copies of every CV, and compared each result with 2,000 shuffles of the names among each CV's copies. On small lists the four-fifths rule trips on chance alone: with names shuffled at random, the lowest origin ratio on a top-10 list is at most 0.71 in 95% of shuffles. Every real list sits inside the range chance gives. The closest call was women on top-10 lists, with 13 places to men's 19, a gap at least that big in 14.5% of shuffles; on top-25 lists women led, 44 to 39.

Requirement by requirement

Each copy also gets a verdict on every requirement. Between two names on the same CV, verdicts differed 3.2% of the time; between the same CV sent twice, 3.1%. Most of it sits in the few requirements that need judgment, such as years of experience counted from dates, where the twins disagree about as often. For the data engineer job names differed more often than the twins, 5.6% against 3.8%, where the twin figure rests on one pair per CV.

Date of birth

The two age copies carry the same name and differ only in a date of birth 25 years apart, with the years of every degree hidden. Older minus younger, fit moved by −0.002 points on average (interval −0.22 to +0.23), and every qualified copy of either age was placed among the true fits.

What it cost

The four matches checked 2,940 copies in detail, everyone inside each job's rules, in 11 parts and 964 seconds, for $0.98 of AI cost, about $0.33 per 1,000 copies checked. Reading each CV once when it arrives is a separate step, not counted here.