/proof
How accuracy is measured
Method, in full. Results, under NDA.
01
A held-out question set
Accuracy is measured continuously against questions the system has not been tuned against, drawn from the institution's real work.
02
Graded on three axes
Groundedness, citation correctness, and correct refusal. An unsupported claim and a wrong citation are both failures, and so is answering what should have been refused.
03
Component ablation
Components are switched off and the set re-measured, so each one's contribution is demonstrated rather than claimed.
04
No change on assertion
Nothing ships on the belief that it improves accuracy. If the measurement does not move, the change does not go out.
Scores go in the NDA pack. Nothing on this page is a number we chose to flatter ourselves with.
Request the pack