The 10.6× Speedup Became 2.03× Under an Honest Baseline

Agents that tune GPU kernels can optimize the measurement rather than the system. AutoTuneBench formalizes a protocol after a four-day pilot of 619 model calls exposed strawman baselines, machine-specific timing, saturated tasks and infrastructure faults. Test-enforced provenance and database validation reject out-of-protocol runs, while anti-cheat checks sit outside the agent's modification surface. The best reported kernel fell from 10.6× against a naive baseline to 2.03× against an honest one; 51% of KernelBench Level-1 tasks admitted comparison, with median speedup 1.0001× over PyTorch eager.