Keyword stuffing benchmark: Formzed vs OpenCATS

The CVs, the cheats and the answer key

We generated 400 full-length CVs: a summary, education, three or four jobs with dates and bullet points, projects, a skills block, certificates, hobbies and a referee. 85 were written near each of four jobs, and 60 belong to people in other work, so the pile looks like a real inbox. 54 show skills as a picture of badges and 31 are scanned pages with no text in the PDF.

For each job, 32 people cheat: four for each of eight tricks. Of all 128, 57 work in the job's own field but lack a must-have, and 71 come from other work entirely. A trick never makes anyone qualified in the answer key: only what the person really did counts, so no cheater is qualified.

There are honest controls too, so a tool can't win by rejecting anything unusual. 32 honest people have long genuine skills lists of 35 to 60 tools, and 20 qualified people also put their own real skills in white text, which is harmless padding rather than a lie. The four jobs, word for word below, each have three must-haves: a minimum number of years, worked out from the CV's own dates, and two skills or kinds of experience. The key was frozen and hashed (5e69af64…) before either tool saw a CV. Formzed's matching was developed on separate CV sets made the same way; these 400 CVs, with their own trap lines and misspellings, were generated afterwards, and each tool ran on them once.

Forward Deployed Engineer at Corvane (fictional company)

About the role
Corvane builds a data platform that hospitals and insurers use to plan their staffing. Forward deployed engineers work inside our customers' teams to get the platform live and keep it useful. Expect around 40% travel.

What you'll do
- Work on site and on calls with customer engineers to connect their systems to our platform.
- Write the code each deployment needs: data connectors, transformations and small internal tools.
- Turn what you learn from customers into clear requests for our product team.
- Own each deployment from kickoff until the customer uses it every week.

Must have
- 3+ years of professional software engineering experience
- Customer-facing deployment work: implementing or integrating software together with customers
- Python or TypeScript

Nice to have
- SQL
- Experience with healthcare or insurance data

Start date: you can start within 2 months.
Location: remote, with travel.

UI/UX Designer at Wrenfield (fictional company)

About the role
Wrenfield makes budgeting software used by 30,000 small charities. You'll join a design team of four and work with two product squads.

What you'll do
- Design new features from the first sketches to files that developers can build from.
- Plan and run research with charity staff to understand how they really work.
- Test prototypes with users and turn what you learn into clear changes.
- Keep our screens consistent, simple and accessible to everyone.

Must have
- 2+ years of professional UI/UX or product design experience
- Figma
- User research or usability testing

Nice to have
- Accessibility (WCAG)
- Experience designing for non-profits

Start date: any.
Location: remote.

Head of Design at Ostrava Health (fictional company)

About the role
Ostrava Health builds patient booking and messaging apps used by 300 clinics. You'll lead a design team of seven and report to the Chief Product Officer.

What you'll do
- Hire, coach and grow a team of product designers and a user researcher.
- Own the design system and make sure every product team builds with it.
- Set the standard for quality, accessibility and research across the whole product.
- Work with product and engineering leaders on the roadmap and team plans.

Must have
- 7+ years of professional product or UX design experience
- Experience leading a team of designers
- Built or owned a design system

Nice to have
- Healthcare product experience
- Figma

Start date: any.
Location: remote.

Senior Product Manager at Tallyroll (fictional company)

About the role
Tallyroll sells payroll and expenses software to companies with 50 to 2,000 employees. You'll own the expenses product and work with eight engineers and two designers.

What you'll do
- Own the roadmap for the expenses product and decide what the team builds next.
- Talk to customers every week and turn what you learn into clear problems to solve.
- Measure how features are used and whether they move the goals we set.
- Work with sales, support and marketing on every launch from start to finish.

Must have
- 4+ years of professional product management experience
- Product analytics: Amplitude, Mixpanel, Heap or Pendo
- Led a B2B SaaS product end to end

Nice to have
- SQL
- Payroll or fintech experience

Start date: you can start within 2 months.
Location: remote.

How each tool searched

Which tricks got through

Keyword search reads text, and most tricks are text. OpenCATS put 47 of the 80 hidden-text cheaters on its shortlists, because white, tiny and covered text is still text to it; only text past the page's edge never reached either tool. Keyword walls and skills lists with no work behind them fooled it just as easily. Formzed put none of the 128 cheaters on a shortlist.

What Formzed did with the cheats

Formzed checks what is visible on each page as it reads a PDF. Text in white, at a tiny size, under a box or past the page's edge is set aside, and the matching works only on what a person would see. The hidden note to the AI was set aside the same way, so it was never read as an instruction. The 20 honest people who padded their CVs with their own skills in white text had that text set aside too, and were judged on their visible CV.

Pasted job ads and keyword walls are visible text, so they need a different check. Formzed notices lines copied from the job description and lines that are piles of the job's words, leaves those lines out, and tells its AI the CV looks stuffed. All 16 pasted job ads and all 16 walls were caught. The check also flagged 1 honest CV of 400: a frontend developer whose skills list says front-end, back-end and end-to-end, flagged for the product manager job because "end" came up three times; nothing on the CV was left out, and they weren't a candidate for that job.

A CV listing the job's must-haves with no work behind them is the hardest trick, because a skills list is normally good evidence. Formzed counts a duty such as customer deployments or leading a team only when one of the person's jobs shows that kind of work, so an accountant who lists "Customer deployments" doesn't meet it. None of the 16 got through.

What each found

Across the four jobs Formzed found 134 of 163 qualified people (82%) with 1 wrong pick. OpenCATS found 97 with recruiter terms, but 97 of its 194 picks were wrong: one in two. With strict terms it found 77.

The top 10

A recruiter mostly reads the top of the list. Formzed put a qualified person in 40 of the 40 top-10 places and no cheater in any. OpenCATS doesn't rank, so its first ten are the first ten names alphabetically among its matches: with recruiter terms 21 were qualified and 15 were cheaters.

Formzed's wrong pick

Formzed picked one person who isn't qualified. A product manager, picked for the UI/UX Designer job: their years as a product manager were counted as product design experience, and "Figma (viewer)" in their tools list counted as Figma. Telling a product manager from a product designer by their job history is a known weak spot.

What Formzed missed

Formzed missed 29 qualified people, about one in six. Most were judgement calls on leadership and ownership: whether a product manager's CV shows they led a B2B SaaS product end to end, whether a designer built or owned a design system or led other designers. Formzed marked them only partly shown and passed over them. A recruiter reading the same CVs might decide either way, but the answer key counts them as qualified, so they count as misses here. Being strict is why so few wrong people got through; this is what it costs.

Every number