Messy CV benchmark: Formzed vs OpenCATS

The CVs, the jobs and the answer key

We generated 400 full-length CVs in one ordinary layout: a summary, education, three or four jobs with dates and bullet points, projects, a technical skills block, certificates, hobbies and a referee. 300 run to two pages. 68 were written near each of four jobs, and 128 belong to people in other work (nurses, teachers, accountants, sales), so the pile looks like a real inbox.

Each skill a person really has is written one way, chosen at random: in the job ad's own words, under another name, misspelt, only inside a sentence, as a picture of skill badges (64 CVs) or on a scanned page with no text in the PDF at all (36 CVs). 62 CVs carry a trap: a word that looks like a skill the person doesn't have.

The four jobs have full job descriptions, word for word below, and each has three must-haves: a minimum number of years, worked out from the CV's own dates, and two skills or kinds of experience. Two of them also need the person to start within two months, which the application form asks. Who is qualified came from the generator's own record of each person, not from reading the CVs, and the key was frozen and hashed (bc13a99f…) before either tool saw a CV.

Backend Engineer at Lumora (fictional company)

About the role
Lumora runs a booking platform used by 2,000 clinics. You'll join a team of six engineers who build and run the APIs and services behind it. The role is fully remote.

What you'll do
- Design, build and run the services behind our public and internal APIs.
- Own your services in production: monitoring, on-call (one week in six) and fixing what breaks.
- Review code and design docs with the rest of the team.
- Work with product managers to break large features into small releases.

Must have
- 3+ years of professional software engineering experience
- PostgreSQL or MySQL
- Experience with one of: Go, Java, Python, Node.js or C#

Nice to have
- Docker
- Cloud experience (AWS or GCP)

Start date: you can start within 2 months.
Location: remote.

Data Engineer at Fieldline (fictional company)

About the role
Fieldline sells farm equipment across Europe. Our data team of four builds the pipelines that feed finance, sales and service reporting. The role is remote, with any start date.

What you'll do
- Build and maintain batch and streaming pipelines from our ERP, CRM and telematics systems.
- Model data in the warehouse so analysts can trust the numbers.
- Add tests and alerts so broken data is caught before it reaches a dashboard.
- Work with analysts on what each new source should look like.

Must have
- 2+ years of professional data engineering experience
- Python or Scala
- One of: Spark, Airflow, dbt, Kafka or Snowflake

Nice to have
- SQL
- Data quality checks

Start date: any.
Location: remote.

HR Generalist at Brightwater Housing (fictional organisation)

About the role
Brightwater is a housing association with 450 staff across 12 sites. You'll join a people team of five and look after the day-to-day people processes for two regions. Any start date.

What you'll do
- Advise line managers on policy, absence and employee relations.
- Run hiring and new-starter processes for your regions.
- Keep employee records and our HR system accurate.
- Support projects such as policy reviews and engagement surveys.

Must have
- 2+ years of experience working in HR
- Recruitment or onboarding experience
- Experience with an HRIS: BambooHR, Workday, SAP SuccessFactors or HiBob

Nice to have
- CIPD qualification
- Employee relations casework

Start date: any.

Warehouse Operations Supervisor at Kestrel Logistics (fictional company)

About the role
Kestrel runs a 220,000 sq ft distribution centre for grocery and household brands. You'll supervise one of three shifts, with around 40 operatives.

What you'll do
- Plan and run your shift: labour, picking, loading and dispatch to cut-off times.
- Keep the site safe: briefings, checks, near-miss reporting.
- Coach team leaders and operatives and handle day-to-day people issues.
- Report shift performance and fix stock and system errors with the WMS team.

Must have
- 2+ years supervising warehouse or logistics staff
- A forklift / MHE licence
- Experience with a WMS: SAP EWM, Manhattan, Oracle WMS or Blue Yonder

Nice to have
- IOSH or similar health and safety training
- Peak planning

Start date: you can start within 2 months.

What OpenCATS is

OpenCATS is a free, open-source, web-based applicant tracking system written in PHP on a MySQL database. It began as CATS, built by Cognizo Technologies around 2005 to 2007; that project later split into the open-source OpenCATS and a separate commercial product. Its public code repository dates from 2009, which makes it one of the longest-running open-source applicant tracking systems. It is still maintained: version 0.12.0 came out on 22 October 2026, and it's the version we ran.

It does what recruiters expect of an applicant tracking system: job orders with a pipeline of candidates for each, candidate records with their resumes and contact history, companies and their contacts, career portal pages, lists, a calendar, email and reports. To find people by their resumes it pulls the text out of each uploaded file, stores it, and returns every resume that contains the words searched for. Results come back in a fixed order (by last name), not ranked by how well someone fits. Its documentation says resume search has been unreliable in some versions and is being restored, so we didn't rely on the search page: we called the search code in the current release directly.

That makes it a fair stand-in for the keyword filter inside many applicant tracking systems: exact words in, a list out. We installed it locally and used its own code to pull the text out of each PDF and to build each resume-search query, run on its own database. Its combined AND/OR queries didn't work, so we ran one search per keyword and combined the lists in code, exactly as the query meant.

Sources: the OpenCATS repository and its releases, the OpenCATS documentation, and the CATS 0.8.0 quick guide (© 2005–2007 Cognizo Technologies). OpenCATS code is under the Mozilla Public License 2.0.

How each tool searched

What each found

Across the four jobs Formzed found 127 of 141 qualified people (90%). OpenCATS found 67 with recruiter terms and 60 with strict terms. The gap is widest where keywords can't see: on the warehouse job, where most qualified people had a licence or system in badges, under another name or misspelt, OpenCATS found 5 of 19.

The top 10

A recruiter mostly reads the top of the list. Formzed put a qualified person in 39 of the 40 top-10 places. OpenCATS doesn't rank, so its first ten are simply the first ten names alphabetically among its matches: 28 of 40 were qualified with recruiter terms and 26 with strict terms.

Where keywords go blind

Split by how the skill was written, the two tools are level when it's spelt out plainly or sits inside a sentence. Keyword search can't see skills in pictures or on scanned pages, and it misses misspellings. For the 72 qualified people whose skills were on a scanned page, in badges or misspelt, Formzed found 64 and OpenCATS 7.

Wrong picks

A keyword can't tell a skill from a word that only looks like one. OpenCATS with recruiter terms picked 16 people for a trap word on their CV and 12 with too few years, which it has no way to read from dates. Formzed picked 2 wrong people, both for a trap word.

What Formzed missed

Formzed missed 14 qualified people. Most were judgement calls on wording a recruiter would accept: duties described in other words, earlier jobs whose duties counted toward the years, and a hosted edition of a database. Formzed's matching has been changed since to catch each of these, but the changes haven't been measured on a new set of CVs yet, so the numbers here are the test as it ran.

What it cost

Formzed's AI cost for the whole test was $1.30: $0.27 to read all 400 CVs once, scans and badges included, and $1.03 to match them to the four jobs. That's under a cent per CV for all four jobs together. A CV is read once, so matching the same pile to another job costs only the matching.

Every number

The dataset

Everything is free to download and re-run against any CV screening tool: the 400 CVs as PDFs, the four job descriptions, the frozen answer key with how each skill was written and every trap, the OpenCATS search terms and both tools' scored results. It's under CC BY 4.0.