CV matching benchmark

Five runs, four fixes

The first run cut every requirement to its first 60 characters before the AI checked CVs against it, so long either-or lists lost options such as PHP, C#, Storybook and Snowflake. The first fix keeps each requirement in the job's own words, up to 160 characters, and ranks a candidate who only partly meets a must-have below those who meet them all.

The second fix stops Formzed's proof check turning down correct answers when the AI quotes only the matching items of a skills list (“Storybook, Playwright, Jest” out of a longer list), checks old “missing” must-haves once more, and tells the AI plainly that a technology the requirement names always counts.

The third fix aims at no qualified candidate missed: everyone inside the job's rules gets the detailed check, where before only the best rated did; any requirement judged missing where the candidate's own text names it gets a second, careful look on its own; and a certificate, course or degree naming a skill shows it.

The fourth fix counts a technology named in a candidate's skills list (“Python, Django, PostgreSQL”) as meeting a requirement that only names technologies, such as “Python or Scala”, whatever a verdict said.

What each match cost

Run 4 was the first to record what each match used, and the first to check everyone inside the job's rules in detail. The four jobs checked 2,865 CVs in 8 parts and 702 seconds, for 944 credits, or $0.75 of AI cost to Formzed at $0.0007925 a credit. Answers saved by earlier runs were reused where the requirement and CV were unchanged, so a first match on CVs never matched before can cost a little more.

Asked again in the latest run, after the fourth fix, the same four jobs cost 141 credits ($0.11) in 245 seconds: nearly every answer was saved by the run before, so asking about the same job again costs a fraction of the first match.

Written tokens, the AI's verdicts, cost five times as much as the CV text it reads, so they are about half the bill although there are far fewer of them. The runs before the third fix didn't record what they used.

R-precision by job

R-precision: of a job's true fits, the share Formzed placed within as many places as there are true fits. A perfect ranking scores 100%.

Recall@k: how many true fits are in the top k places

Score vs. true fit

True fit: the answer key's share of the job's requirements a CV meets (a must-have counts three times as much as a nice-to-have). Formzed gives every candidate inside the rules a quick rating, then checks them in detail for a fit score.

Precision and recall down the ranking

Calibration: can a score be read as a chance of fitting?

A quick rating is read as a chance out of ten of being a true fit, a detailed fit as a chance out of a hundred. Formzed's scores rank candidates and don't promise probabilities; this shows how far a score can be taken literally. Circles grow with the number of candidates in the band; whiskers are 95% intervals.

No true fit lost

Every true fit in the four jobs is placed within as many places as there are true fits. The causes that took true fits before are gone: none was kept from the detailed check by a low quick rating, and none was lost to a certificate, a turned-down quote, a cut requirement or a misread skills list.

How this was measured