Generated 2026-08-17 from our offline judge, which inspects real research output produced by this build.
An earlier draft of this card graded each category on a different metric. That was wrong, and we would rather say so than quietly change it: if one surface is scored on task success and another on groundedness, the two letters are not comparable, and whoever chooses the metric can choose a flattering one. We now follow the design of Stanford’s HELM — apply one shared rubric to every category and report a cell wherever it applies. 30 of 42 cells are graded on this build; the rest read NA with the reason stated. NA is a published result here, not a gap we hide.
We report the observed result as a letter, and how much weight it can bear as a separate certainty rating, following GRADE — the evidence-grading standard used by Cochrane, the WHO and NICE. Certainty starts at High and is downgraded for:
A cell can therefore read A / Very Low: an excellent result you should not yet lean on. That is more useful, and more honest, than folding the uncertainty into the letter and publishing a quietly deflated grade with no explanation of what deflated it.
| # | Dimension | Question asked of every category | Modelled on |
|---|---|---|---|
| 1 | Productive work | Did it actually produce substantive output, rather than terminating empty? | Modelled on agentic task-success rate (tau-bench, Yao et al. 2024; WebArena, Zhou et al. 2023) — the unit is work that achieved its goal, not work that merely finished. We compute a success rate over our own runs; we do not run those benchmarks. |
| 2 | Substantiation | Can every claim be traced to evidence the system actually holds? | Modelled on AIS (Rashkin et al., TACL 2023) and RAGAS faithfulness (Es et al., 2023). NOTE: AIS is a human-evaluation protocol and RAGAS scores claim-level entailment; we do neither. We check structural traceability — whether a claim has evidence attached at all. |
| 3 | Reference integrity | Do the references resolve, and point at the right record? | Modelled on ALCE citation precision (Gao et al., EMNLP 2023). NOTE: ALCE judges whether a citation SUPPORTS its statement, via entailment. We do not do that. We check that a reference is well-formed and specific — a weaker property, and we would rather name the gap than imply the stronger one. |
| 4 | Specificity | Is the output actionable and particular, rather than boilerplate? | Modelled on RAGAS answer relevance (Es et al., 2023). We use pattern-based detectors for boilerplate rather than RAGAS’s model-scored relevance. |
| 5 | Instruction adherence | Does it honour the machine-checkable output contract it was given? | IFEval-style strict accuracy (Zhou et al., 2023) — verifiable instructions scored pass/fail with no partial credit. This is the closest genuine match in the table: our contracts are machine-checkable, so the method really applies. |
| 6 | Calibration | Does stated confidence match the evidence actually behind the claim? | Modelled on the QUESTION HELM’s calibration metric asks — does stated confidence match reality. We do NOT compute expected calibration error: our outputs are discrete states and grades, not probabilities, so there is no calibration curve to score. |
| 7 | Non-redundancy | Is the same thing emitted more than once? | Modelled on the MQM error typology (themqm.org), which scores redundancy by user impact. We apply the idea, not MQM’s weighted scoring formula. |
The rubric is modelled on published evaluation work; it does not reimplement it. That distinction matters in a document arguing for honest measurement, so here it is explicitly.
Borrowed as structure, used faithfully: HELM’s one-suite-across-every-scenario design; GRADE’s separation of estimate from certainty (three of its five downgrade domains; the sample-size thresholds are ours); IFEval’s strict pass/fail scoring of machine-checkable instructions, which is the closest genuine match here.
Computed exactly: the Wilson score interval behind every certainty rating.
Named but deliberately NOT implemented — each row states the weaker thing we actually do:
If a number here matters to you, the definition behind it is the one in this document — not the one in the cited paper.
| Category | Productive work | Substantiation | Reference integrity | Specificity | Instruction adherence | Calibration | Non-redundancy | Overall |
|---|---|---|---|---|---|---|---|---|
| Research runs | A | A | NA | NA | A | NA | NA | A |
| Research work log | A | A | A | A | B | A | A | B |
| Sources & citations | A | A | A | A | B | NA | A | B |
| Person identification | A | A | A | A | A | B | A | B |
| Insights | A | A | A | A | A | A | A | A |
| Historical places | NA | NA | NA | NA | NA | NA | NA | NA |
Letters are the observed rate: A ≥ 95% · B ≥ 85% · C ≥ 70% · D ≥ 50% · F < 50%. Overall is the worst cell in the row, not an average — a system that substantiates nothing is not redeemed by tidy formatting. Per-cell certainty follows.
Records from this build: 5 (capture holds 49).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 3/3 = 100.0% | A | Low | serious imprecision: only 3 records in scope |
| Substantiation | 3/3 = 100.0% | A | Low | serious imprecision: only 3 records in scope |
| Reference integrity | — | NA | — | A run is not itself an authored claim; its references are graded on the sources and insights it wrote. |
| Specificity | — | NA | — | Not applicable to a run record. |
| Instruction adherence | 3/3 = 100.0% | A | Low | serious imprecision: only 3 records in scope |
| Calibration | — | NA | — | No records in scope for this dimension on this build. |
| Non-redundancy | — | NA | — | Run ids are deterministic and rows are reused by design, so a duplicate run cannot occur. |
Records from this build: 57 (capture holds 270).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Substantiation | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Reference integrity | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Specificity | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Instruction adherence | 18/19 = 94.7% | B | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Calibration | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Non-redundancy | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
Records from this build: 20 (capture holds 78).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 20/20 = 100.0% | A | Moderate | imprecision: 20 records in scope, short of the 30 a tight interval needs |
| Substantiation | 20/20 = 100.0% | A | Low | imprecision: 20 records in scope, short of the 30 a tight interval needs; indirectness: the presence of a source URL is checked, not whether the record at it supports the claim |
| Reference integrity | 20/20 = 100.0% | A | Low | imprecision: 20 records in scope, short of the 30 a tight interval needs; indirectness: link syntax and specificity are checked, but the URLs are never fetched, so reachability is unverified |
| Specificity | 20/20 = 100.0% | A | Moderate | imprecision: 20 records in scope, short of the 30 a tight interval needs |
| Instruction adherence | 18/20 = 90.0% | B | Moderate | imprecision: 20 records in scope, short of the 30 a tight interval needs |
| Calibration | — | NA | — | A source’s only confidence signal is provisional, which asserts that its URL was found in the research corpus. The capture cannot see corpus membership, so the claim cannot be checked against evidence. |
| Non-redundancy | 20/20 = 100.0% | A | Moderate | imprecision: 20 records in scope, short of the 30 a tight interval needs |
Records from this build: 19 (capture holds 68).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Substantiation | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Reference integrity | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Specificity | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Instruction adherence | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Calibration | 17/19 = 89.5% | B | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
| Non-redundancy | 19/19 = 100.0% | A | Moderate | imprecision: 19 records in scope, short of the 30 a tight interval needs |
Records from this build: 7 (capture holds 25).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 7/7 = 100.0% | A | Low | serious imprecision: only 7 records in scope |
| Substantiation | 7/7 = 100.0% | A | Low | serious imprecision: only 7 records in scope |
| Reference integrity | 7/7 = 100.0% | A | Low | serious imprecision: only 7 records in scope |
| Specificity | 7/7 = 100.0% | A | Low | serious imprecision: only 7 records in scope |
| Instruction adherence | 7/7 = 100.0% | A | Low | serious imprecision: only 7 records in scope |
| Calibration | 7/7 = 100.0% | A | Low | serious imprecision: only 7 records in scope |
| Non-redundancy | 7/7 = 100.0% | A | Low | serious imprecision: only 7 records in scope |
All dimensions NA. Our own measurement is defective, and we would rather say so than publish grades from it. The capture reads 93 records; the database held 20,355 for this account at capture time, the newest written minutes earlier. It joins each place to its originating event and silently drops those whose event our test reset had already deleted, so it has re-reported the same frozen hundred records from February across nineteen builds. Tracked as GGENIE-1136.
Records from this build: 93 (capture holds 93).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 93/93 = 100.0% | NA | Low | risk of bias: a known defect in our own capture |
| Substantiation | 93/93 = 100.0% | NA | Low | risk of bias: a known defect in our own capture |
| Reference integrity | 93/93 = 100.0% | NA | Very Low | risk of bias: a known defect in our own capture; indirectness: URL presence is checked; the URLs are never fetched |
| Specificity | 93/93 = 100.0% | NA | Low | risk of bias: a known defect in our own capture |
| Instruction adherence | 93/93 = 100.0% | NA | Low | risk of bias: a known defect in our own capture |
| Calibration | 65/65 = 100.0% | NA | Low | risk of bias: a known defect in our own capture |
| Non-redundancy | 89/93 = 95.7% | NA | Low | risk of bias: a known defect in our own capture |