benchmarks evals
A Benchmark Asked Which Pig Made the Trace
A provenance audit finds repurposed tasks, toy setups and flawed splits inside a standard time-series archive.
Summary
A provenance audit finds repurposed tasks, toy setups and flawed splits inside a standard time-series archive.
PigAirwayPressure began as a hemorrhage-detection experiment, yet its UCR Archive task asks a classifier to identify which of 52 pigs produced a trace. The authors trace this and other datasets back to their origins, finding repurposed objectives, toy setups, questionable splits and ordered non-temporal sequences presented as time-series tasks. They also question whether decomposing multivariate problems preserves the deployment setting. The audit does not invalidate every archive-wide comparison, but it shows why aggregate benchmark ranks can reward performance on tasks that differ materially from the practical questions the source data were collected to answer.
Why it matters
A provenance audit finds repurposed tasks, toy setups and flawed splits inside a standard time-series archive.
Limits and context
- The audit does not invalidate every archive-wide comparison, but it shows why aggregate benchmark ranks can reward performance on tasks that differ materially from the practical questions the source data were collected to answer.
Key claims
A provenance audit finds repurposed tasks, toy setups and flawed splits inside a standard time-series archive.
Qualification: The audit does not invalidate every archive-wide comparison, but it shows why aggregate benchmark ranks can reward performance on tasks that differ materially from the practical questions the source data were collected to answer.
Evidence: source-2026-11-09-014
Sources
- arXiv preprint 2611.09843, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.