| Build judged | 2026101016 — a build number, not a date |
| That build deployed | 2026-10-10 16:07 UTC |
| Data captured from the app | 2026-10-10 |
| This card generated | 2026-10-10 |
| Deployed now | this is the build currently deployed |
Produced by our offline judge, which inspects real research output this build wrote.
⚠️ Which grades actually describe THIS build. Each capture records how many of its rows postdate the deploy; the counts in brackets are those post-deploy rows.
- Mixed: Research runs (7), Research work log (104), Sources & citations (20), Person identification (32), Jev screen decisions (98), Insights (5), Historical places (34), AI QA (4). A flat rate over these is dominated by the pre-deploy majority — split before quoting one.
This is the GGENIE-1095 rule applied to the card itself: the filename names the build that was LIVE WHEN THE CAPTURE RAN, not the build that produced the rows.
An earlier draft of this card graded each category on a different metric. That was wrong, and we would rather say so than quietly change it: if one surface is scored on task success and another on groundedness, the two letters are not comparable, and whoever chooses the metric can choose a flattering one. We now follow the design of Stanford’s HELM — apply one shared rubric to every category and report a cell wherever it applies. 47 of 56 cells are graded on this build; the rest read NA with the reason stated. NA is a published result here, not a gap we hide.
We report the observed result as a letter, and how much weight it can bear as a separate certainty rating, following GRADE — the evidence-grading standard used by Cochrane, the WHO and NICE. Certainty starts at High and is downgraded for:
A cell can therefore read A / Very Low: an excellent result you should not yet lean on. That is more useful, and more honest, than folding the uncertainty into the letter and publishing a quietly deflated grade with no explanation of what deflated it.
| # | Dimension | Question asked of every category | Modelled on |
|---|---|---|---|
| 1 | Productive work | Did it actually produce substantive output, rather than terminating empty? | Modelled on agentic task-success rate (tau-bench, Yao et al. 2024; WebArena, Zhou et al. 2023) — the unit is work that achieved its goal, not work that merely finished. We compute a success rate over our own runs; we do not run those benchmarks. |
| 2 | Substantiation | Can every claim be traced to evidence the system actually holds? | Modelled on AIS (Rashkin et al., TACL 2023) and RAGAS faithfulness (Es et al., 2023). NOTE: AIS is a human-evaluation protocol and RAGAS scores claim-level entailment; we do neither. We check structural traceability — whether a claim has evidence attached at all. |
| 3 | Reference integrity | Do the references resolve, and point at the right record? | Modelled on ALCE citation precision (Gao et al., EMNLP 2023). NOTE: ALCE judges whether a citation SUPPORTS its statement, via entailment. We do not do that. We check that a reference is well-formed and specific — a weaker property, and we would rather name the gap than imply the stronger one. |
| 4 | Specificity | Is the output actionable and particular, rather than boilerplate? | Modelled on RAGAS answer relevance (Es et al., 2023). We use pattern-based detectors for boilerplate rather than RAGAS’s model-scored relevance. |
| 5 | Instruction adherence | Does it honour the machine-checkable output contract it was given? | IFEval-style strict accuracy (Zhou et al., 2023) — verifiable instructions scored pass/fail with no partial credit. This is the closest genuine match in the table: our contracts are machine-checkable, so the method really applies. |
| 6 | Calibration | Does stated confidence match the evidence actually behind the claim? | Modelled on the QUESTION HELM’s calibration metric asks — does stated confidence match reality. We do NOT compute expected calibration error: our outputs are discrete states and grades, not probabilities, so there is no calibration curve to score. |
| 7 | Non-redundancy | Is the same thing emitted more than once? | Modelled on the MQM error typology (themqm.org), which scores redundancy by user impact. We apply the idea, not MQM’s weighted scoring formula. |
The rubric is modelled on published evaluation work; it does not reimplement it. That distinction matters in a document arguing for honest measurement, so here it is explicitly.
Borrowed as structure, used faithfully: HELM’s one-suite-across-every-scenario design; GRADE’s separation of estimate from certainty (three of its five downgrade domains; the sample-size thresholds are ours); IFEval’s strict pass/fail scoring of machine-checkable instructions, which is the closest genuine match here.
Computed exactly: the Wilson score interval behind every certainty rating.
Status of each gap we named. Two are now closed on one surface, two are not:
v3-sources only.periodrelatedplaces carries an ordinal 1–5 grade on every row, so a reliability diagram over
those bins is constructible as soon as entailment runs there; v3-sources — the one surface we
have judged — records no confidence field at all (all 81 rows null), so there is nothing to
calibrate. That is a product gap, not a metric gap, and it is the blocker to report.If a number here matters to you, the definition behind it is the one in this document — not the one in the cited paper.
| Category | Productive work | Substantiation | Reference integrity | Specificity | Instruction adherence | Calibration | Non-redundancy | Overall |
|---|---|---|---|---|---|---|---|---|
| Research runs | A n=5 | A n=5 | NA | NA | A n=5 | A n=1 | NA | A |
| Research work log | A n=43 | A n=43 | A n=43 | A n=43 | A n=43 | A n=43 | A n=43 | A |
| Sources & citations | A n=11 | A n=11 | A n=11 | A n=11 | B n=11 | NA | A n=11 | B |
| Person identification | A n=32 | A n=32 | A n=32 | A n=32 | A1 n=32 | NA | A n=32 | A |
| Jev screen decisions | A2 n=98 | A n=4 | NA | A3 n=14 | A4 n=66 | NA | NA | A |
| Insights | A n=5 | A n=5 | A n=5 | A n=5 | A n=5 | A n=5 | A n=5 | A |
| Historical places | A n=34 | A n=34 | A n=34 | A n=34 | A n=34 | A n=20 | A n=34 | A |
| AI QA | A n=4 | A n=4 | NA | A n=4 | A n=4 | F n=1 | A n=4 | F |
1 Person identification — Instruction adherence: 32 of 32 rows carry a lead score (written from GGENIE-1382 on); 1 flagged as a hedge by the model, 1 weak lead(s) — counted, not graded
2 Jev screen decisions — Productive work: who decided, of 98: 50 reached Jev; 32 went to the deep judge after Jev; 16 settled by code on their data (GGENIE-1809: code and timing bypasses are by design; a Jev error is not)
3 Jev screen decisions — Specificity: grades the deep judge’s attach prose (14 attach(es) routed escalated / contradiction-review / jev-unavailable), not Jev’s own decisions (GGENIE-1807)
4 Jev screen decisions — Instruction adherence: who decided, of 66: 50 reached Jev; 16 settled by code on their data (GGENIE-1809: code and timing bypasses are by design; a Jev error is not)
Letters are the observed rate: A ≥ 95% · B ≥ 85% · C ≥ 70% · D ≥ 50% · F < 50%.
n is the number of records the cell was computed over, and it is on every graded cell on purpose: a letter without its denominator is not a result. Treat any cell with a single-digit n as a reading, not a rate — the certainty column beside it in the per-category tables says how far it can be leaned on.
Overall is the worst cell in the row, not an average — a system that substantiates nothing is not
redeemed by tidy formatting. Per-cell certainty follows.
What this sample cannot contain. Runs are counted from JobActivity, which carries a split TTL (jobactivity.model.ts:133-150): a completed run is retained 10 days, but failed, canceled, waiting, active, delayed and prioritized runs are retained 24 hours. Beyond that window a failed run is not merely unsampled — it has been deleted. 19 of 26 dated rows are older than 24 hours, and the status mix is 25 completed, 1 canceled — the non-completed rows that survive are the recent ones, which is the censoring visible in miniature. So the only failure this category can observe is a run that completed and produced nothing; a run that failed is structurally invisible to it, and the success rates below should be read as rates among survivors rather than among all runs. The capture is not at fault — it filters on job name alone and carries signals for every failure state; the deletion happens upstream of it. We publish the grades because the check does still fire — across the wider capture window, 5 of 22 genuine runs completed having written nothing — but the letter and the certainty below both rest on a sample that cannot include the worst case.
Records from this build: 7 (capture holds 26).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 5/5 = 100.0% | A | Very Low | risk of bias: the run population is TTL-censored before capture, so failed and canceled runs are absent; serious imprecision: only 5 records in scope |
| Substantiation | 5/5 = 100.0% | A | Very Low | risk of bias: the run population is TTL-censored before capture, so failed and canceled runs are absent; serious imprecision: only 5 records in scope |
| Reference integrity | — | NA | — | A run is not itself an authored claim; its references are graded on the sources and insights it wrote. |
| Specificity | — | NA | — | Not applicable to a run record. |
| Instruction adherence | 5/5 = 100.0% | A | Very Low | risk of bias: the run population is TTL-censored before capture, so failed and canceled runs are absent; serious imprecision: only 5 records in scope |
| Calibration | 1/1 = 100.0% | A | Very Low | risk of bias: the run population is TTL-censored before capture, so failed and canceled runs are absent; serious imprecision: only 1 records in scope |
| Non-redundancy | — | NA | — | Run ids are deterministic and rows are reused by design, so a duplicate run cannot occur. |
Records from this build: 104 (capture holds 432).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 43/43 = 100.0% | A | High | no downgrades |
| Substantiation | 42/43 = 97.7% | A | High | no downgrades |
| Reference integrity | 43/43 = 100.0% | A | High | no downgrades |
| Specificity | 43/43 = 100.0% | A | High | no downgrades |
| Instruction adherence | 42/43 = 97.7% | A | High | no downgrades |
| Calibration | 43/43 = 100.0% | A | High | no downgrades |
| Non-redundancy | 43/43 = 100.0% | A | High | no downgrades |
Records from this build: 19 (capture holds 128; 1 test-fixture rows from this build excluded, 18 in the whole capture).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 11/11 = 100.0% | A | Moderate | imprecision: 11 records in scope, short of the 30 a tight interval needs |
| Substantiation | 11/11 = 100.0% | A | Low | imprecision: 11 records in scope, short of the 30 a tight interval needs; indirectness: the presence of a source URL is checked, not whether the record at it supports the claim |
| Reference integrity | 11/11 = 100.0% | A | Low | imprecision: 11 records in scope, short of the 30 a tight interval needs; indirectness: link syntax and specificity are checked, but the URLs are never fetched, so reachability is unverified |
| Specificity | 11/11 = 100.0% | A | Moderate | imprecision: 11 records in scope, short of the 30 a tight interval needs |
| Instruction adherence | 10/11 = 90.9% | B | Moderate | imprecision: 11 records in scope, short of the 30 a tight interval needs |
| Calibration | — | NA | — | A source’s only confidence signal is provisional, which asserts that its URL was found in the research corpus. The capture cannot see corpus membership, so the claim cannot be checked against evidence. |
| Non-redundancy | 11/11 = 100.0% | A | Moderate | imprecision: 11 records in scope, short of the 30 a tight interval needs |
Records from this build: 32 (capture holds 120; 0 test-fixture rows from this build excluded, 11 in the whole capture).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 32/32 = 100.0% | A | High | no downgrades |
| Substantiation | 32/32 = 100.0% | A | High | no downgrades |
| Reference integrity | 32/32 = 100.0% | A | High | no downgrades |
| Specificity | 32/32 = 100.0% | A | High | no downgrades |
| Instruction adherence | 32/32 = 100.0% | A | High | 32 of 32 rows carry a lead score (written from GGENIE-1382 on); 1 flagged as a hedge by the model, 1 weak lead(s) — counted, not graded |
| Calibration | — | NA | — | A roster row’s only confidence signal is its reason prose, and the prompt asks for a terse relationship label with evidence carried structurally (sourceCount, urls) rather than asserted in the text. There is no stated-confidence claim to weigh against evidence strength, and the one evidence-strength check the capture could support — a definite kinship claim with zero sources — never fires, because no row has zero sources. |
| Non-redundancy | 32/32 = 100.0% | A | High | no downgrades |
Records from this build: 98 (capture holds 125).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 98/98 = 100.0% | A | High | who decided, of 98: 50 reached Jev; 32 went to the deep judge after Jev; 16 settled by code on their data (GGENIE-1809: code and timing bypasses are by design; a Jev error is not) |
| Substantiation | 4/4 = 100.0% | A | Low | serious imprecision: only 4 records in scope |
| Reference integrity | — | NA | — | A Jev decision carries no reference of its own to resolve. An implausible year in an age-check reason (the implausibleYear lead) is the candidate’s source data, and the age check that parked it decided correctly — so it is reported as a lead, not charged to the decision. |
| Specificity | 14/14 = 100.0% | A | Moderate | imprecision: 14 records in scope, short of the 30 a tight interval needs; grades the deep judge’s attach prose (14 attach(es) routed escalated / contradiction-review / jev-unavailable), not Jev’s own decisions (GGENIE-1807) |
| Instruction adherence | 66/66 = 100.0% | A | High | who decided, of 66: 50 reached Jev; 16 settled by code on their data (GGENIE-1809: code and timing bypasses are by design; a Jev error is not) |
| Calibration | — | NA | — | Every Jev route is composed from Jev’s own answer by code — judgeLinks derives a link’s route and attach from the screen’s outcome, composeMergeRoute folds only at p(same) >= 0.8 and stores it rounded, and keptReason gives every confident same-person keep a documented override — so a decision cannot disagree with Jev’s answer on app output. The three checks (routeScreenMismatch, foldBelowThreshold, keepAgainstAnswer) are kept as ungraded regression signals; one firing means that composition code changed, not that Jev misjudged. |
| Non-redundancy | — | NA | — | A Jev decision records the candidate’s NAME but no profile id, so two different same-name candidates in one run are indistinguishable from one decided twice. The capture keeps duplicateDecision as a lead; it is not graded until decisions carry a profile id. |
Records from this build: 5 (capture holds 61).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 5/5 = 100.0% | A | Low | serious imprecision: only 5 records in scope |
| Substantiation | 5/5 = 100.0% | A | Low | serious imprecision: only 5 records in scope |
| Reference integrity | 5/5 = 100.0% | A | Low | serious imprecision: only 5 records in scope |
| Specificity | 5/5 = 100.0% | A | Low | serious imprecision: only 5 records in scope |
| Instruction adherence | 5/5 = 100.0% | A | Low | serious imprecision: only 5 records in scope |
| Calibration | 5/5 = 100.0% | A | Low | serious imprecision: only 5 records in scope |
| Non-redundancy | 5/5 = 100.0% | A | Low | serious imprecision: only 5 records in scope |
What this sample is made of. 23,706 of the 23,851 places this capture drew from (99.4%) have lost their originating event; and the capture emits 2,000 of them. ⚠️ Of the 34 post-deploy rows, 34 carry a promptInput and are therefore gradeable against what the model was shown; they span only 1 distinct individuals, of which 0 of 34 rows (0%) were written for a test fixture (none). Those rows are now EXCLUDED from every rate below (GGENIE-1366), matching the harness exclusion v3-sources has carried since GGENIE-1279. What remains is 34 rows across 1 real subjects (Georgius Brasfield) — read the letters below against that, and note the certainty column, which carries the imprecision. These are debris from events deleted by past test resets, never cleaned up with them. That figure describes THIS ORGANIZATION, which is what the capture is scoped to — the whole-collection rate is lower, and conflating the two is the org trap, so quote each against its own denominator. Since GGENIE-1196 those rows are RETAINED and flagged rather than silently dropped, which is what makes the sample current at all. Name, reason, citation, grade and URL are judgeable on them and are graded below. Non-redundancy is not: a place cannot duplicate another on an event that no longer exists, so that dimension is scoped to the rows that still have one and its result speaks only for those. Read every other cell as the whole corpus and that one as a fraction of it.
Records from this build: 34 (capture holds 2000).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 34/34 = 100.0% | A | Moderate | risk of bias: the corpus is overwhelmingly orphaned, so the event-dependent dimension has almost no scope |
| Substantiation | 34/34 = 100.0% | A | Moderate | risk of bias: the corpus is overwhelmingly orphaned, so the event-dependent dimension has almost no scope |
| Reference integrity | 34/34 = 100.0% | A | Low | risk of bias: the corpus is overwhelmingly orphaned, so the event-dependent dimension has almost no scope; indirectness: URL presence is checked; the URLs are never fetched |
| Specificity | 34/34 = 100.0% | A | Moderate | risk of bias: the corpus is overwhelmingly orphaned, so the event-dependent dimension has almost no scope |
| Instruction adherence | 34/34 = 100.0% | A | Moderate | risk of bias: the corpus is overwhelmingly orphaned, so the event-dependent dimension has almost no scope |
| Calibration | 20/20 = 100.0% | A | Low | risk of bias: the corpus is overwhelmingly orphaned, so the event-dependent dimension has almost no scope; imprecision: 20 records in scope, short of the 30 a tight interval needs |
| Non-redundancy | 34/34 = 100.0% | A | Moderate | risk of bias: the corpus is overwhelmingly orphaned, so the event-dependent dimension has almost no scope |
Records from this build: 4 (capture holds 14).
| Dimension | Result | Grade | Certainty | Why not higher |
|---|---|---|---|---|
| Productive work | 4/4 = 100.0% | A | Low | serious imprecision: only 4 records in scope |
| Substantiation | 4/4 = 100.0% | A | Low | serious imprecision: only 4 records in scope |
| Reference integrity | — | NA | — | A priority flag is written onto an existing event; it carries no reference of its own to resolve. |
| Specificity | 4/4 = 100.0% | A | Low | serious imprecision: only 4 records in scope |
| Instruction adherence | 4/4 = 100.0% | A | Low | serious imprecision: only 4 records in scope |
| Calibration | 0/1 = 0.0% | F | Low | serious imprecision: only 1 records in scope |
| Non-redundancy | 4/4 = 100.0% | A | Low | serious imprecision: only 4 records in scope |