Self-evaluation

How well does the verifier actually work? Measured on 24 test questions. Full table and method in eval/RESULTS.md.

Generated 2026-10-21 16:48 UTC · runs completed 24/24

Verifier catch rate 94.5% of 55 planted false claims rejected

Wrongful rejections 0.0% of 60 true claims re-checked

Independent groundedness 94.0% 100 report claims, judged by google/gemini-2.5-flash-lite

Agreement with hand labels 86.0% κ = 0.70 · 50 labels

Catch rate by kind of planted error

entity 76.9% n=13

negation 100.0% n=19

number 100.0% n=12

year 100.0% n=11

number: a figure changed · year: a date shifted · negation: direction flipped (rose↔fell, is↔is not) · entity: a different organisation or country named.