The agent gets raw market data and a strategy spec and has to build a working backtest. The data contains undisclosed traps, like the ones real data has. Grading is deterministic and sandboxed, and runs on hidden synthetic data across 5 seeds. A pass means every trap was handled on every seed.
Models write backtests that run, return plausible Sharpe ratios and are wrong. In our calibration runs, most failures were near-misses: in 341 of 511 trap-handling failures, the code handled every data issue but one. Unit tests on clean data don't catch that. A grader that re-runs the code on hidden, messy data does.
Calibration figures on this site cover the research-backtest family only.
# 12-1 Momentum Long-Short: Research Task ## Objective Build a monthly decile long-short strategy from a raw synthetic market dataset, following standard quantitative research methodology. ## Dataset (/app/data/) prices.csv date, symbol, open, high, low, close, volume, currency universe.csv symbol, snapshot_date corporate_actions.csv symbol, date, action_type, ratio, amount All data is synthetic. Hidden grading datasets share the schema but use different symbols, sizes and dates. ## Signal Score = close[t-21] / close[t-273] - 1 (trading-day offsets). Long top decile, short bottom decile. Rebalance first trading day of month. Equal weight; longs sum to +1, shorts to -1. ## Deliverable /app/solution.py defining run_strategy(data_dir: str) -> dict sharpe_ratio, annual_return, annual_volatility, max_drawdown, total_return, num_trades, turnover, holdings_sample ## Grading run_strategy is executed on hidden datasets. Holdings must follow the spec and hold up against the realities of raw market data.
The submitted code is re-run on hidden synthetic datasets across 5 seeds. Pass = 100/100. The same code gets the same grade: we re-graded 36 saved responses and every score came out identical.
Every task passes an acceptance check before it ships: the reference scores 100, a solution that ignores any single trap fails that trap on 5/5 seeds, and stubs, random answers, fake holdings and naive filters score 0.
On the current private set, 49 of 50 tasks fall in the 2–70% pass band across three frontier models. One sits just above it at 72% and is labelled easy. No task in the set is at 0%.
All data is synthetic, with fictional tickers and no exchange data, not drawn from real price history. Grading data is generated from seeds that never ship. A procedural generator makes fresh variants (strategy, trap mix, universe, history length) when a task leaks or when you need an exclusive batch.
Synthetic markets with the problems real data has: corporate actions, listings and delistings, stale quotes, halts, symbol and currency changes.
Each task must pass an acceptance check first: reference = 100/100, every single-trap omission fails, every shortcut control scores 0.
Frontier models attempt each task on identical instructions. Each task ships with its per-model pass rate, attempt count and difficulty label.
Harbor-format environments with a pinned Docker image, sample data and a verifier that runs in a separate container.
Aggregates from the private set we sell, so you can see how hard each task is before you buy. Task names are withheld. The public leaderboard runs on a separate held-out set (see below).
Share of well-formed attempts that missed the category, where the category was present.
For post-training and eval teams at AI labs. Also useful to quant funds and finance-agent startups testing their own agents.
License tasks from the calibrated set, each with a calibration record (per-model passes, attempts, difficulty label).
Fresh variants from the generator (new strategy, trap mix, universe and history length) that no one else has seen. Use them to refresh tasks that have leaked or to buy batches exclusive to your lab.
Saved attempts with per-category feedback and partial scores, for analysing failures or building reward models. Graded responses with feedback are already saved for every calibration attempt.
harbor/<task>/
task.toml # schema 1.4, separate verifier mode, limits, canary
instruction.md # spec + methodology
environment/ # Dockerfile (python 3.11, numpy, pandas, pyarrow)
# + sample data
The verifier runs in its own container and receives only /app/solution.py. Submissions run as an unprivileged user.
Delivered in Harbor format (schema 1.4): a pinned Python 3.11 image and a verifier in a separate container. Other formats on request.
Three tasks: one easy, one in the 2–70% band and one hard that no model has passed. Includes sample data, Harbor environments and the calibration report. Run your agent, send us the solution.py files and we return graded results with per-category feedback. Evaluation use only.
Licensed per task set.
Scoped to your target skills and difficulty band.
Verifier-included packages, for rewards inside your own RL loop, are available under NDA on fresh variants.
Real price history is in pretraining data, so a model can recall outcomes instead of working them out. We use synthetic markets with fictional tickers, generated from seeds that never ship. No exchange data, so there are no data licences to clear.
Every task passes an acceptance check before it ships: the reference solution scores 100/100 on every hidden seed, a solution that ignores any single trap fails that trap on all 5 seeds, and shortcut controls (stubs, random answers, fake holdings, naive filters) score 0. Grading is deterministic and runs in a separate sandbox.
The current set was calibrated on claude-opus-5-5, gpt-6-astra and grok-4.7, with identical instructions and 6 attempts per task for each model. Each task ships with its per-model passes, attempt counts and difficulty label.
Harbor format (schema 1.4) by default. Other packaging on request.
Yes. The generator produces fresh variants that can be licensed to one buyer. Terms are agreed per engagement.
The free sample is graded by us: send solution.py and we return the reward and per-category feedback. Verifier-included packages for your own RL loop are available under NDA on fresh variants.
Calibration numbers come from the private set we sell and are shown only as aggregates. The leaderboard uses a separate held-out public set.
Probably. Calibration was single-response, with no tools and no code execution. We'll publish agentic results separately.
Three tasks, Harbor environments, sample data and the calibration report. We reply from hi@lanemint.com.
We'll reply to the email you entered.