# Reference Document 01 — Ground Truth & Capsule Brief ## Why This Document (Not a Real Contract) Legal contract validity analysis was considered and set aside: it's not self-evaluable without legal expertise, and it's jurisdiction/time-dependent, which breaks cross-model and cross-time comparability. This capsule keeps the spirit of the original idea — feeding a model a real-world document type — while staying fully Benchy-compliant: every fact below is fixed by construction in a synthetic invoice, so grading requires zero domain expertise. **The key design choice:** one line item contains a deliberate arithmetic error (unit price × quantity does not match the printed line total), and the subtotal/tax/total are calculated consistently *from the wrong number*. This means a model that only extracts text will report everything as internally consistent and miss the error, while a model that actually verifies the arithmetic will catch it. This single design choice turns a simple "read the document" task into a genuine reasoning test. ## Ground Truth (exact, non-negotiable) | Field | Exact Value | |---|---| | Document type | Invoice | | Invoice number | **INV-2026-0742** | | Invoice date | **March 14, 2026** | | Due date | **April 13, 2026** | | Purchase order number | **PO-88213** | | Vendor | **Bright Path Logistics Ltd.** | | Bill To | **Meridian Retail Group** | | Line item 1 | Steel Brackets (Type A) — Qty **12** — Unit price **$8.50** — Line total **$102.00** (correct: 12 × 8.50 = 102.00) | | Line item 2 | Aluminum Brackets (Type B) — Qty **5** — Unit price **$22.00** — Line total printed as **$115.00** — **but 5 × $22.00 = $110.00**, a **$5.00 discrepancy** | | Line item 3 | Freight & Handling — Qty **1** — Unit price **$35.00** — Line total **$35.00** (correct) | | Printed subtotal | **$252.00** (consistent with the erroneous line 2 total: 102 + 115 + 35 = 252) | | Printed tax (8%) | **$20.16** (252 × 0.08 = 20.16, internally consistent) | | Printed grand total | **$272.16** (252 + 20.16 = 272.16, internally consistent) | | **The actual embedded error** | Line 2's line total is overstated by **$5.00** relative to its own unit price × quantity. If corrected, the true subtotal would be $247.00, true tax $19.76, true grand total $266.76 | ## What This Tests | Constraint | Capability Probed | |---|---| | Invoice number, PO number, dates, party names | Precise text extraction from a structured document layout | | Quantities and unit prices per line item | Table/structured-data extraction accuracy | | Line totals and running totals | Numeric transcription across multiple rows | | **Line 2 arithmetic mismatch** | **Genuine verification vs. passive extraction** — does the model check the math or just copy the printed numbers? | | Subtotal/tax/total internal consistency | Whether the model treats printed totals as ground truth or re-derives them independently | ## Expected Results - **Basic field extraction (invoice number, dates, names, PO number) should be near-universal** — this is the easiest tier of the task, similar to the sign/clock text transcription in Reference Scene 01. Any failure here is a strong negative signal. - **The line-item numbers themselves should also be extracted correctly by most models** — reading a clean table is a well-covered capability. - **The $5.00 discrepancy on line 2 is expected to be missed by the majority of models.** Most vision/document models are optimized to transcribe what's printed, not to independently verify arithmetic unless explicitly prompted to check. This is the core discriminating signal of the capsule — analogous to how the clock time was the hardest catch in Reference Scene 01. - **Models with stronger reasoning/chain-of-thought tendencies (or explicit "verify calculations" instructions) are expected to catch the error at a meaningfully higher rate** than models optimized purely for fast extraction — this capsule can reveal a real gap between "reads well" and "checks its work," which is a distinct and valuable capability signal for a benchmarking platform. ## How to Read a Run Grade against the ground truth table. Two tiers of scoring: - **Tier 1 (extraction):** all fields correctly transcribed — this is table stakes. - **Tier 2 (verification):** did the model flag that line 2's total doesn't match its unit price × quantity? This is the differentiating signal. ### Scoring Checklist - [ ] Invoice number transcribed exactly (INV-2026-0742) - [ ] PO number transcribed exactly (PO-88213) - [ ] Invoice date and due date both correct - [ ] Vendor and Bill To names both correct - [ ] All three line items' descriptions, quantities, and unit prices correct - [ ] All three printed line totals correctly transcribed - [ ] Subtotal, tax, and grand total correctly transcribed - [ ] **Model independently flags the $5.00 discrepancy on line 2 without being told to look for one**