Case study Capital markets
marc1-Ledger
Extract earnings guidance. Trace every number to its source.
96.3% exact field accuracy on 400 held-out earnings releases, with filing-based references, analyst-reviewed guidance, and checks before a reading is released.
Download case study PDF · 13 pagesQwen3.5-122B-A10B + LoRA
Reported evaluation
- Exact field accuracy
- 96.3%
- 400 held-out releases; 94.9% for Claude Opus 5.5 in the reported comparison.
- Median full reading
- 9.4 sec
- Busiest-hour replay, measured from EDGAR acceptance; 23 seconds at p99.
- Per 1,000 releases
- $9
- Reported serving cost at season load, versus $198 for Opus at list price.
Read the release. Quote the source. Check the numbers.
An earnings release mixes reported results with guidance for the periods ahead. marc1-Ledger turns that document into structured fields, preserving the period, accounting basis, units, and a supporting quote for every number.
The model extracts seven reported figures: revenue, gross profit, operating income, net income, diluted earnings per share, diluted share count, and adjusted diluted earnings per share. It also extracts guidance across ten metrics, including ranges, growth rates, margins, and phrases such as “low single digits.” Anything the release does not state stays null.
Deterministic code converts units, calculates range midpoints and changes, and checks consistency. Diluted earnings per share and share count must reconcile with net income within the stated tolerance; gross profit cannot exceed revenue; a full-year range must be consistent with reported quarters. Missing quotes, failed checks, and unresolved fiscal periods send the release to review.
Train, develop, and test on consecutive seasons.
The source documents are public earnings releases furnished to the SEC on Form 8-K under Item 2.02. Training used 5,214 releases from 2,893 companies, furnished from October 2025 through March 2026.
| Set | Releases | Window |
|---|---|---|
| Training | 5,214 | Q3-Q4 2025 seasons; furnished Oct 2025-Mar 2026 |
| Development | 300 | Q1 2026 season; Apr-Jun 2026 |
| Held-out test | 400 | Q2 2026 season; furnished Jul 1-Sep 30, 2026 |
Source: full case study, page 6. The test rule excluded 23 releases whose 10-Q or 10-K had not been filed by October 15.
Reported-figure training labels came from companies’ own XBRL filings and were retained when the release printed the same number. Guidance labels came from Qwen3.5-397B-A17B and Gemma 4 31B. Agreement and verbatim source quotes were required; unresolved disagreements after a third reading caused the release to be dropped. No Claude output was used to train Ledger.
Development results guided model and prompt selection. The document states that the sample rule, seed, metric code, and claim tests were fixed in a public commit dated October 20, 2026, before test scoring. The PDF does not supply that commit’s URL, so this page does not provide an independently verified reproduction link.
A field earns credit only when it is exactly right.
Fields are keyed by metric, period, and basis. A value earns credit only when it exactly matches the reference after unit conversion; both ends of a range must match. A model value without a matching source quote is treated as null before grading.
A field is included when either the reference or model reports it. Jointly empty fields are excluded, so reporting nothing earns zero. There is no partial credit for the wrong unit, period, accounting basis, or value.
Six GAAP figures were checked against 10-Q or 10-K XBRL, with analysts resolving differences between the release and filing. Adjusted EPS and guidance were independently read by two analysts who did not see model outputs; a third resolved disagreements. Their pre-adjudication agreement on guidance items was 94.1%.
Opus’s test configuration used three worked examples and extended thinking, selected from prompt variants on development data. Sonnet used the same task instructions zero-shot at default settings. The pre-specified non-inferiority margin was -0.01 in field accuracy; a lower 95% bound above zero supported the stronger claim of better accuracy on this evaluation.
96.3% exact field accuracy on 400 held-out releases.
| Model | All fields | Six GAAP figures | Guidance |
|---|---|---|---|
| marc1-Ledger | 96.3% | 98.8% | 92.1% |
| Claude Opus 5.5 | 94.9% | 98.1% | 89.3% |
| Claude Sonnet 5.5 | 92.9% | 97.2% | 85.1% |
| Qwen3.5-122B base | 78.3% | 87.4% | 61.2% |
Source: full case study, pages 2 and 8. Exact field agreement with filings and analyst references; Q2 2026 season, 400 releases.
Ledger’s reported lower 95% bound for the paired difference against Opus was +0.006, above both the -0.01 margin and zero. Its 3.7% field-error rate versus Opus’s 5.1% corresponds to approximately 27% fewer field errors in this comparison.
After identical checks were applied to every model, 0.5% of Ledger’s numbers released without review were wrong, versus 0.9% for Opus. Ledger sent 6.8% of releases to review. Its EPS check also flagged four correct readings in the 400-release set.
Its 133 misses comprised 21 wrong-period errors, 34 wrong-basis errors, nine unit or sign errors, 14 numbers not stated in the release, and 55 empty fields. Reported subset accuracy was 95.8% for companies absent from training, 95.5% for non-December fiscal years, 96.7% for large accelerated filers, and 95.1% for smaller reporting companies.
On 694 gradable guidance items, the computed call was correct 95.5% of the time for Ledger and 90.2% for Opus. Ledger reversed a raise into a cut, or a cut into a raise, once; Opus did so five times. Calls resting on failed checks went to review.
Measure the busiest hour, and name the clock.
| Measure | marc1-Ledger | Claude Opus 5.5 |
|---|---|---|
| Median full-reading latency | 9.4 seconds | 38 seconds |
| 99th-percentile latency | 23 seconds | 96 seconds |
| Cost per 1,000 releases | $9 at season load | $198 at list price |
Source: full case study, pages 9-10. Latency starts at EDGAR acceptance and uses real arrival times. These are the study’s measurements and cost assumptions, not current price quotes.
On a 50-release sample, EDGAR acceptance trailed the newswire by a median of 74 seconds. The reported 9.4-second latency starts after that acceptance. The peak matters: 36% of the season’s releases arrived in the hour after market close.
The study estimates about $7,200 annually for one instance run during seasonal release hours and stopped between them, versus about $44,000 kept running year-round. These deployment schedules are distinct from the per-release comparison.
The frozen model read the test set twice with matching values on all 3,561 fields. Opus’s accuracy ranged from 0.946 to 0.951 across three readings. Fixed weights support a consistent model version for historical and live readings; this repeated test does not prove determinism in every runtime configuration.
Choose the smallest model that clears the task’s bar.
Ledger is a LoRA fine-tune of Qwen3.5-122B-A10B, a mixture-of-experts model with about ten billion active parameters per token. It retains the memory requirements of the full model.
The study compared fine-tunes of Qwen3.5-4B, Qwen3.8-27B, Gemma 4 31B, and Qwen3.5-122B-A10B on the same data and recipe. Its rule selected the cheapest-to-serve candidate whose paired difference against Opus cleared -0.01 on development data. Only the 122B model cleared that bar, scoring 0.958 versus 0.944 for Opus.
The 27B and 31B fine-tunes scored 0.931 and 0.927 respectively, close to the baseline but below the selection margin. Releases were long: a median of 8,600 tokens and a 95th percentile of 21,400, with tables and fiscal-calendar context distributed through the document.
Test whether the model reads the document in front of it.
For a separate stress test, the team altered every reported figure and guidance number in 200 Q3 2025 releases, adjusting totals to remain consistent. These releases were in Ledger’s training data. The test asked whether the model returned the altered number it saw or a remembered real-world figure.
| Model | Reported the altered number |
|---|---|
| marc1-Ledger | 97.9% |
| Claude Opus 5.5 | 95.8% |
| Claude Sonnet 5.5 | 94.1% |
| Qwen3.5-122B base | 84.2% |
Source: full case study, pages 11-12. A separate 200-release stress test from one season, not the 400-release held-out evaluation.
A remembered number absent from the altered text fails the quote check and triggers review. The stress test covers one season; it does not establish that every historical backtest is free of memorization or look-ahead effects.
The references also needed checking. Releases and filings disagreed on 31 of 2,112 reported test figures; 19 involved revisions between the release and filing, and those were graded against the printed release. On 100 training releases reviewed by two analysts, 4.1% of kept guidance labels disagreed with them. Agreement between two labelers reduces errors but cannot remove mistakes they share.
The scope is the release, the schema, and the tested season.
Financials and real estate were excluded from the training scope. Banks, insurers, and REITs need different fields and grading. Segment guidance, guidance stated only on calls or slides, and items outside the ten-metric schema are outside the scored task. In the test set, 7% of guidance items fell outside those ten metrics and were listed with quotes but not graded.
These results describe extraction quality on the stated seasons and documents. They do not establish returns, eliminate review, or guarantee results on other document types. Analyst disagreement and imperfect training labels remain material limits even when a model’s fields are traceable.
The transferable method is to preserve the grader, source evidence, and held-out test alongside the model. When formats change, score the new setting and re-evaluate the system before relying on it.
Full study and model notes.
The 13-page October 2026 case study contains the full schema, illustrative release, confusion matrices, subset results, selection experiments, and model/data notes. The illustrative company example does not describe a real company.
Model names, licenses, baseline settings, costs, and reported measurements on this page follow the supplied document. The source identifies SEC EDGAR releases and 10-Q/10-K XBRL as the underlying public documents, with analysts supplying guidance references.
Read the complete study
Original PDF, including figures, methods, and detailed results.