benchmarks evals
The Meta-Agent Must Beat the Incumbent Before Promotion
A conservative architecture search uses paired shadow replay, simulated canaries and rollback instead of treating every proposed team change as progress.
Summary
A conservative architecture search uses paired shadow replay, simulated canaries and rollback instead of treating every proposed team change as progress.
Agent architectures deployed under shifting conditions face delayed feedback, correlated failures and costly changes. Conservative Meta-Agent Search retrieves prior architectures, proposes typed graph edits and compares each candidate with the incumbent through paired shadow replay using common random numbers. Promotion requires a variance-penalized margin, followed by a simulated canary and predefined rollback. On the author-designed MetaInvest-Bench, CMAS lowered normalized dynamic-oracle regret from 0.116 to 0.104 and raised held-out utility from 0.44 to 0.48 over the strongest adaptive baseline, while shrinking teams when extra workers added no value. Search-inclusive token cost was highest. The calibration concerns one replay statistic and benchmark; it guarantees neither harm-free canaries nor production improvement.
Why it matters
A conservative architecture search uses paired shadow replay, simulated canaries and rollback instead of treating every proposed team change as progress.
Limits and context
No additional limitation was separately recorded.
Key claims
A conservative architecture search uses paired shadow replay, simulated canaries and rollback instead of treating every proposed team change as progress.
Evidence: source-2026-10-25-004
Sources
- arXiv preprint 2610.31436, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.