Recruiter time to a shortlist: Formzed vs OpenCATS
A search is only as good as the reading it saves. On the same 400 messy CVs as our messy CV benchmark, we counted how many CVs a recruiter opens, going down each tool's list, to reach 10 qualified people for each of four jobs. With Formzed it was 41. With OpenCATS's keyword search it was about 182, and reading the pile with no tool about 490.
Worked out from the messy CV benchmark's own results and its answer key, frozen before either tool saw a CV. No CV was read again and no AI was called. Every CV and person is made up.
CVs opened for 10 qualified, four jobs
OpenCATS: about 182. No tool: about 490
CVs opened to see every qualified person
Formzed. OpenCATS: about 1,524, nearly the whole pile for every job
Warehouse job, 10 qualified
Formzed. OpenCATS: about 140, because its list held only 5 qualified people
How it was counted
The recruiter starts at the top of a tool's list and opens CVs in order until they have 10 qualified people, then carries on until they have seen every qualified person. Whether someone is qualified comes from the answer key, not from either tool.
Formzed
Its ranking of every CV for the job, best fit first, as the match returned it. Its top 10 was 39 of 40 qualified across the four jobs, so 10 qualified people usually take 10 CVs.
OpenCATS
What its resume search returned with the recruiter's terms, after the job's start-date rule, in OpenCATS's own order: last name A to Z, as it doesn't rank. If its list runs out first, the recruiter reads the rest of the pile in no particular order.
No tool
All 400 CVs read in no particular order: the baseline a search has to beat.
Reading the pile in no order can't be counted exactly, so those numbers (marked ≈) are averages: among N unread CVs holding K qualified people, the m-th qualified person turns up on average at CV m × (N + 1) ÷ (K + 1).
CVs opened to reach 10 qualified people
Formzed needed 10 CVs for three jobs and 11 for the warehouse job, where one wrong pick sat in its top 10. OpenCATS's lists got there in 12 to 15 CVs for the three office jobs. On the warehouse job its list ran out with 5 qualified people, and the rest had to come from the pile.
CVs opened to reach 10 qualified people, by job
Qualified people found as CVs are opened
The first 200 CVs of each job. The orange dot is where OpenCATS's list ends; after it, and for no tool, the line is the average count.
The warehouse job, worked through
Most of OpenCATS's total comes from this one job, so here is the arithmetic. Its search returned 9 CVs after the start-date rule, and 5 of them were qualified. The other 5 have to come from the 391 CVs it didn't return, which hold the remaining 14 qualified people. Read in no order, the 5th of 14 turns up on average at CV 5 × 392 ÷ 15 ≈ 131, so the total is 9 + 131 ≈ 140. It's an average: a lucky recruiter finds them sooner, an unlucky one later.
A recruiter whose search comes back short might search again with more words instead. The recruiter-terms search had already added FLT, reach truck, counterbalance, MHE and JDA, and still found 5: the rest show their licence or system in pictures of badges, on a scanned page or misspelt, which keyword search can't read.
CVs opened to see every qualified person
To be sure no qualified person was missed, the recruiter keeps going. Formzed ranks the people it missed low but still ranks them, so all of them turn up within 214 CVs across the four jobs. OpenCATS's search left out about half of the qualified people, and finding them means reading nearly the whole pile: about 1,524 of 1,600, barely fewer than with no tool at all.
CVs opened to see every qualified person, by job
What that is in hours
Reading time wasn't measured. The counts above are the result; turning them into hours takes a guess at how long one CV takes to read, so set your own.
| To reach | Formzed | OpenCATS | No tool |
|---|---|---|---|
| 10 qualified people for each of the four jobs | 1.4 h | ≈6.1 h | ≈16 h |
| Every qualified person for each of the four jobs | 7.1 h | ≈51 h | ≈52 h |
CVs counted × your minutes per CV. An example, not a measurement.
Every number
| Job | Tool | Its list | CVs to 10 qualified | CVs to all qualified |
|---|---|---|---|---|
| Backend Engineer | Formzed | ranks all 400 | 10 | 46 |
| Backend Engineer | OpenCATS, recruiter terms | 26 listed, 22 qualified | 12 | ≈381 |
| Backend Engineer | OpenCATS, strict terms | 23 listed, 19 qualified | 12 | ≈384 |
| Backend Engineer | No tool | — | ≈98 | ≈391 |
| Data Engineer | Formzed | ranks all 400 | 10 | 44 |
| Data Engineer | OpenCATS, recruiter terms | 33 listed, 23 qualified | 15 | ≈384 |
| Data Engineer | OpenCATS, strict terms | 33 listed, 23 qualified | 15 | ≈384 |
| Data Engineer | No tool | — | ≈91 | ≈392 |
| HR Generalist | Formzed | ranks all 400 | 10 | 78 |
| HR Generalist | OpenCATS, recruiter terms | 27 listed, 17 qualified | 15 | ≈385 |
| HR Generalist | OpenCATS, strict terms | 22 listed, 13 qualified | 16 | ≈387 |
| HR Generalist | No tool | — | ≈100 | ≈391 |
| Warehouse Supervisor | Formzed | ranks all 400 | 11 | 46 |
| Warehouse Supervisor | OpenCATS, recruiter terms | 9 listed, 5 qualified | ≈140 | ≈375 |
| Warehouse Supervisor | OpenCATS, strict terms | 9 listed, 5 qualified | ≈140 | ≈375 |
| Warehouse Supervisor | No tool | — | ≈201 | ≈381 |
≈ marks averages over reading the rest of the pile in no order. OpenCATS's lists are after the job's start-date rule.
The CVs, the four job descriptions, the answer key and both tools' results are in the messy CV benchmark's dataset, free under CC BY 4.0, so every number here can be checked.