LaneMint Public Leaderboard

held-out setlm-public-v1 · 6 tasks · scored Oct 15, 2026

statusscored · single-response published

modesingle-response | agentic (Harbor), reported separately

gradingdeterministic · sandboxed · 5 hidden seeds per task · pass = 100/100

methodology/methods changelog /leaderboard#changelog

Model Provider Version / snapshot Date tested Mode Pass rate 95% CI (Wilson) Passes / attempts Attempts per task Mean partial score Cost per attempt Provider errors

pub_06: no model passed. Kept and labelled hard.

Sorted by pass rate once scored. Where 95% intervals overlap, the ranking marker is greyed and marked "overlapping CIs". Provider errors count as failures and are shown in their own column. Modes are never mixed in one table.

Per-category misses

cell = missed / attempts where present · single-hue intensity · hatched where n < 10

model survivorship unadjusted_split unadjusted_dividends stale_prices trading_halt ticker_rename currency_switch

claude-opus-5-53/304/309/242/304/362/126/36

gpt-6-astra3/291/2910/242/293/3511/1227/35

grok-4.710/292/288/237/287/346/1223/34

A category counts as missed if the attempt failed it on any hidden seed.

Sanitized failure examples

gpt-6-astra · ticker_rename · score 52/100 · FAIL Holdings diverge from the reference after a symbol change. Every other category passed.

claude-opus-5-5 · currency_switch · score 60/100 · FAIL Holdings diverge from the reference after a listing changed currency.

grok-4.7 · stale_prices · score 93.6/100 · FAIL The closest miss in the set: every category passed except stale prices. One missed trap fails the task.

contamination

All data is synthetic, with fictional tickers. Hidden grading data is generated from seeds that never ship. The public set is separate from the tasks we sell and carries canary strings. If a public task leaks, it is rotated out and replaced with a fresh variant from the generator, and the rotation is logged.

reproducibility

Grading is deterministic: the same code on the same seed gets the same grade. Five hidden seeds per task. The verifier runs in a separate container and receives only /app/solution.py. Pinned environment: Python 3.11, numpy, pandas, pyarrow. We re-grade a sample of saved responses on every release to check that grades don't drift. Oct 14: 36 of 36 re-graded identically.

grader soundness

Before any task is scored, it passes acceptance: reference = 100/100, ignoring any single trap fails that trap on 5/5 seeds, and shortcut controls (stubs, random, fake holdings, naive filters) score 0.

limitations

Sample size per task is small. Single-response and agentic results can differ and are reported separately. One configuration per model.

Changelog

Oct 19, 2026: Private calibrated set expanded from 10 to 50 tasks. The 40 new tasks add four strategy families (earnings-yield value, volatility-scaled residual reversal, low volatility, 52-week-high proximity), and fundamentals look-ahead is now in the calibrated set. Every model now has 6 single-response attempts per task. grok-4.7 was re-run on the original 10 tasks at 6 attempts, replacing its Oct 14 3-attempt run. Calibration aggregates on the homepage and /methods updated. Free sample pack refreshed to one easy, one in-band and one hard task. The public held-out set and this leaderboard are unchanged.

Oct 15, 2026: Public set v1 scored. claude-opus-5-5, gpt-6-astra and grok-4.7, 6 tasks × 6 attempts, single-response. grok-4.7 was run with deferred API requests after earlier streaming timeouts. 3 provider errors in total, counted as fails.

Logged: every model added, every re-run on a new snapshot, every task rotated, every grader or harness version change.