[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"capsule:50c4b7d3-9ff8-494d-8a2e-021d4c4c2bef":3},["Reactive",4],{"id":5,"user_id":6,"title":7,"prompt_content":8,"prompt_type":9,"is_locked":10,"is_public":10,"fork_of":11,"created_at":12,"updated_at":13,"category":14,"brief":15,"attachments":16,"expected_markers":23,"user":25},"50c4b7d3-9ff8-494d-8a2e-021d4c4c2bef","695c8e6c-9654-4858-b2d3-925adefacc35","Reference Document 01","Extract all information from this invoice: the invoice number, dates, purchase order number, vendor and bill-to names, every line item with its quantity, unit price, and line total, and the subtotal, tax, and grand total. Then verify all the arithmetic — check that each line total equals quantity times unit price, and that the subtotal, tax, and grand total are calculated correctly. Report any discrepancy you find.","text",true,null,"2026-07-04T17:42:19.675252+00:00","2026-07-04T17:46:06.144976+00:00","research","# Reference Document 01 — Ground Truth & Capsule Brief\n\n## Why This Document (Not a Real Contract)\n\nLegal contract validity analysis was considered and set aside: it's not self-evaluable without legal expertise, and it's jurisdiction\u002Ftime-dependent, which breaks cross-model and cross-time comparability. This capsule keeps the spirit of the original idea — feeding a model a real-world document type — while staying fully Benchy-compliant: every fact below is fixed by construction in a synthetic invoice, so grading requires zero domain expertise.\n\n**The key design choice:** one line item contains a deliberate arithmetic error (unit price × quantity does not match the printed line total), and the subtotal\u002Ftax\u002Ftotal are calculated consistently *from the wrong number*. This means a model that only extracts text will report everything as internally consistent and miss the error, while a model that actually verifies the arithmetic will catch it. This single design choice turns a simple \"read the document\" task into a genuine reasoning test.\n\n## Ground Truth (exact, non-negotiable)\n\n| Field | Exact Value |\n|---|---|\n| Document type | Invoice |\n| Invoice number | **INV-2026-0742** |\n| Invoice date | **March 14, 2026** |\n| Due date | **April 13, 2026** |\n| Purchase order number | **PO-88213** |\n| Vendor | **Bright Path Logistics Ltd.** |\n| Bill To | **Meridian Retail Group** |\n| Line item 1 | Steel Brackets (Type A) — Qty **12** — Unit price **$8.50** — Line total **$102.00** (correct: 12 × 8.50 = 102.00) |\n| Line item 2 | Aluminum Brackets (Type B) — Qty **5** — Unit price **$22.00** — Line total printed as **$115.00** — **but 5 × $22.00 = $110.00**, a **$5.00 discrepancy** |\n| Line item 3 | Freight & Handling — Qty **1** — Unit price **$35.00** — Line total **$35.00** (correct) |\n| Printed subtotal | **$252.00** (consistent with the erroneous line 2 total: 102 + 115 + 35 = 252) |\n| Printed tax (8%) | **$20.16** (252 × 0.08 = 20.16, internally consistent) |\n| Printed grand total | **$272.16** (252 + 20.16 = 272.16, internally consistent) |\n| **The actual embedded error** | Line 2's line total is overstated by **$5.00** relative to its own unit price × quantity. If corrected, the true subtotal would be $247.00, true tax $19.76, true grand total $266.76 |\n\n## What This Tests\n\n| Constraint | Capability Probed |\n|---|---|\n| Invoice number, PO number, dates, party names | Precise text extraction from a structured document layout |\n| Quantities and unit prices per line item | Table\u002Fstructured-data extraction accuracy |\n| Line totals and running totals | Numeric transcription across multiple rows |\n| **Line 2 arithmetic mismatch** | **Genuine verification vs. passive extraction** — does the model check the math or just copy the printed numbers? |\n| Subtotal\u002Ftax\u002Ftotal internal consistency | Whether the model treats printed totals as ground truth or re-derives them independently |\n\n## Expected Results\n\n- **Basic field extraction (invoice number, dates, names, PO number) should be near-universal** — this is the easiest tier of the task, similar to the sign\u002Fclock text transcription in Reference Scene 01. Any failure here is a strong negative signal.\n- **The line-item numbers themselves should also be extracted correctly by most models** — reading a clean table is a well-covered capability.\n- **The $5.00 discrepancy on line 2 is expected to be missed by the majority of models.** Most vision\u002Fdocument models are optimized to transcribe what's printed, not to independently verify arithmetic unless explicitly prompted to check. This is the core discriminating signal of the capsule — analogous to how the clock time was the hardest catch in Reference Scene 01.\n- **Models with stronger reasoning\u002Fchain-of-thought tendencies (or explicit \"verify calculations\" instructions) are expected to catch the error at a meaningfully higher rate** than models optimized purely for fast extraction — this capsule can reveal a real gap between \"reads well\" and \"checks its work,\" which is a distinct and valuable capability signal for a benchmarking platform.\n\n## How to Read a Run\n\nGrade against the ground truth table. Two tiers of scoring:\n- **Tier 1 (extraction):** all fields correctly transcribed — this is table stakes.\n- **Tier 2 (verification):** did the model flag that line 2's total doesn't match its unit price × quantity? This is the differentiating signal.\n\n### Scoring Checklist\n- [ ] Invoice number transcribed exactly (INV-2026-0742)\n- [ ] PO number transcribed exactly (PO-88213)\n- [ ] Invoice date and due date both correct\n- [ ] Vendor and Bill To names both correct\n- [ ] All three line items' descriptions, quantities, and unit prices correct\n- [ ] All three printed line totals correctly transcribed\n- [ ] Subtotal, tax, and grand total correctly transcribed\n- [ ] **Model independently flags the $5.00 discrepancy on line 2 without being told to look for one**",[17],{"url":18,"kind":19,"mime":20,"name":21,"size":22},"https:\u002F\u002Fqoyuqtiafuqqyarrnsxr.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcapsule-files\u002F695c8e6c-9654-4858-b2d3-925adefacc35\u002F41c853f9-4982-40a2-9482-3ae0b2f68767.jpg","image","image\u002Fjpeg","reference-document-01-scanned.jpg",337063,{"items":24,"matchAll":10},[],{"username":26,"avatar_url":27},"Ezarwebmaster","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F42354978?v=4"]