Our own test results, including the two we fail.

Target: the same resume and job never move more than 2 points. Our measured result: 5 points maximum, across 25 runs on 5 test cases. One of the five moved 0 points.

Why we miss it: on one test case the model occasionally reads a borderline line in the job description as a required skill, and occasionally does not. That single skill shifts the score by 5 points.

What it means for you: we save the reading of each job description once, so re-running the same job reuses that reading rather than making a new one. The variance shows up between two different people scanning the same job on the same day, and in any reading made after we change the model or the prompt.

What would fix it: reading each job description three times and taking the majority answer. That triples our cost per scan, so we are waiting to see whether real usage justifies it rather than spending your money on a problem you may never experience.

0 across the eval set. We bait the system with four traps: a job asking for Kubernetes when the resume says Docker Compose, AWS when the resume shows DigitalOcean, five years when the resume shows four, people management when the resume shows mentoring. All four held. Every suggestion we make points at a line already in your resume, and our database refuses to store one that cannot.

0 of 5 contained. We plant instructions inside job descriptions telling the system to score the candidate 100, to ignore its rules, to reveal its own instructions, including payloads hidden with invisible characters and lookalike letters.

95% agreement between two independent methods across 20 labelled requirement statements. We read years, locations and authorisation requirements twice, once with the model and once with plain rules. When they disagree we take the reading that is kinder to you, so nobody is told a role is unwinnable on a technicality.

11 of 11. Two column layouts, image only PDFs, table based skills sections, unusual characters. Files we cannot read fail honestly with an explanation, instead of producing a confident score from text that was never extracted.

Target: our 12 test cases land in the band we expect. Result: 9 of 10 counted fixtures match; one do not. Two further fixtures are known to be wrong themselves and are excluded, with written triage.

Why we missed it: all three mismatches traced to one defect in our own code, not to our test cases. Scoring a plain-text input counted a file check that had nothing to check. Our test cases were right and the engine was wrong. Fixed 27 September.

What it means for you: nothing yet about score quality. This suite currently tests only that the code matches our rules, not that our rules match reality.

What would fix it: hand-labelling the set by human judgment, which is in progress. We will publish that result whether or not it agrees with us.