Methods · LaneMint

01Scope

A task is a research backtest built from raw synthetic market data that contains undisclosed data-quality traps. The agent receives the data and a strategy spec and submits /app/solution.py.

Strategy families currently calibrated, all monthly decile long-short (tasks in the 50-task private set):

12-1 momentum (12) 1-month mean reversion (11) earnings-yield value (9) volatility-scaled residual reversal (8) price-to-trailing-average value (6) low volatility (2) 52-week-high proximity (2)

02Data

sourcefully synthetic · fictional tickers · no exchange data

universe40–60 symbols

history218–520 trading days

sample datafrom a seed not used for grading

03Trap categories

A correct solution must handle each category present in the data.

survivorshipUse the universe as it stood on each date, including names that later delist.

unadjusted splitsAdjust prices for splits before computing returns or signals.

unadjusted dividendsInclude cash distributions in returns.

stale pricesDo not treat repeated, non-updating prices as real observations.

trading haltsHandle days a symbol did not trade.

ticker renamesTreat a renamed symbol as one continuous security.

currency switchesPut prices in a single currency before comparing them.

fundamentals look-aheadUse only data available at the decision date. In the current calibrated set (9 of 50 tasks).

04Grading

Grading is deterministic and sandboxed. The submitted code is re-run on hidden datasets from 5 seeds. Holdings are checked per trap category, plus a clean-data sanity check. Pass = 100/100. The reward is binary at 100; a continuous score is also reported.

/logs/verifier/reward.json → {"reward": 0, "score": 0.83}

Determinism check: 36 of 36 saved responses re-graded identically (Oct 14).

05Acceptance (grader soundness)

Every task passes these checks before it is scored or shipped:

reference solution100 on every seed

ignore any single trapfails that trap 5/5

stubs · seeded random0

fixed fake holdings0

first/last-N selection · naive universe filters0

data-blind · sign-flipped answersrejected

Harbor export: layout, verifier separationchecked

06Tamper hardening

The verifier runs in a separate container. Only /app/solution.py crosses over. Submissions run as an unprivileged user, and the process is stopped before the reward is written.

Harbor test on an earlier sample task (Oct 2026): the oracle scored 1. No-op, stub, and attempts to write the reward file, read the grader or keep running after exit all scored 0. The full per-task Harbor tamper run is in progress.

07Calibration protocol

modelsclaude-opus-5-5 · gpt-6-astra · grok-4.7

instructionsidentical for every model

modesingle-response: no tools, no code execution

attempts6 per task, every model

provider errorscounted as fails (attempts that timed out with no output, or hit a network outage on our side, were re-run; any error left after that counts as a fail)

aggregationmean of model rates for difficulty labels · pooled counts for CIs

intervalsWilson 95%

band2–70% pass rate

08Results

Private set (50 tasks, Oct 14–19, 2026), aggregates only. Task names withheld.

ModelPassedRate95% CIAtt./task≥1 pass

claude-opus-5-5185/30061.7%56–67%649 of 50

gpt-6-astra53/30017.7%14–22%622 of 50

grok-4.752/30017.3%13–22%626 of 50

Pooled 290/900 (32.2%, 95% CI 29–35%). 49 of 50 tasks in the 2–70% band; 1 at 72%, labelled easy; none at 0%.

Trap category (pooled)Missed

ticker renames229/372 (62%)

currency switches252/567 (44%)

unadjusted dividends110/587 (19%)

fundamentals look-ahead18/135 (13%)

survivorship50/534 (9%)

trading halts23/616 (4%)

stale prices20/643 (3%)

unadjusted splits2/633 (<1%)

free sample pack (3 tasks, separate from the 50-task set, 6 attempts per model): easy, 15/18 (83%) · in band, 7/18 (39%) · hard, 0/18

09Limitations

Small n per task: 6 attempts per task per model (18 pooled), so about ±20 points on a task's pooled rate and up to about ±30 for one model on one task. Per-model totals over 300 attempts are about ±5 points.
The 40 tasks added Oct 19 were picked for the band using their first 4 attempts per model, and those attempts are included in the 6, so band placement partly reflects selection.
Single-response mode only. Agentic pass rates have not been measured.
One configuration per model.
One task design pattern (monthly decile long-short research backtest), across seven strategy families.
No human blind solves yet.

10Public vs private sets

The private, sellable set (50 tasks as of Oct 19, 2026) produces the calibration aggregates. The public held-out set produces the leaderboard. Public set v1 was scored Oct 15, 2026. The public set carries canary strings. If a public task leaks, it is rotated out, replaced with a fresh variant from the generator, and the rotation is logged.

11Delivery format

harbor/<task>/
  task.toml          # schema 1.4, separate verifier mode, limits, canary
  instruction.md     # spec + methodology
  environment/       # Dockerfile (python 3.11, numpy, pandas, pyarrow)
                     # + sample data

Harbor schema 1.4. Pinned environment: Python 3.11, numpy, pandas, pyarrow. The free sample pack is graded by LaneMint. Verifier-included packages are available under NDA on fresh variants.

12Changelog and contact

Changes to the grader, harness and sets are logged on the leaderboard changelog. Questions: hi@lanemint.com.