developer tools
The Website Benchmark Gets a Dockyard
WebHarbor packages 153 executable website reconstructions and 3,518 tasks, including multi-step and authenticated workflows, under deterministic control.
Summary
WebHarbor packages 153 executable website reconstructions and 3,518 tasks, including multi-step and authenticated workflows, under deterministic control.
Live websites drift, block access and make repeatable agent evaluation difficult; shallow synthetic sites miss the workflows agents must actually complete. WebHarbor reconstructs 153 website environments as executable Docker packages and builds 3,518 tasks around them. Coding agents extend each environment as new tasks demand more functionality, while people audit visual fidelity, behavior and task validity. The collection includes multi-step and post-authentication workflows and also serves as a reinforcement-learning environment. A roughly four-billion-parameter GUI agent trained in the suite generalized to unseen tasks and competed with tested open-source agents across four benchmarks. Fidelity remains an audited reconstruction claim, not identity with the live services.
Why it matters
WebHarbor packages 153 executable website reconstructions and 3,518 tasks, including multi-step and authenticated workflows, under deterministic control.
Limits and context
- Fidelity remains an audited reconstruction claim, not identity with the live services.
Key claims
WebHarbor packages 153 executable website reconstructions and 3,518 tasks, including multi-step and authenticated workflows, under deterministic control.
Qualification: Fidelity remains an audited reconstruction claim, not identity with the live services.
Evidence: source-2026-11-03-008
Sources
- arXiv preprint 2611.03634, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.