developer tools
Temperature Zero Still Changes the Query Result
A 1.2-million-call study finds repeat executions can alter the output of semantic query engines.
Summary
A 1.2-million-call study finds repeat executions can alter the output of semantic query engines.
Researchers reran text and tabular workloads from three semantic-query benchmarks, logging 1.2 million model calls. Even at temperature zero, some rows changed their operator output between executions, and query plans could amplify small row changes into different final answers. Voting reduced variance without eliminating it. The findings concern the tested models, harnesses and workloads. They challenge benchmark conclusions drawn from only a few executions and suggest that an engine's evaluation should include repeatability alongside average answer quality.
Why it matters
A 1.2-million-call study finds repeat executions can alter the output of semantic query engines.
Limits and context
- They challenge benchmark conclusions drawn from only a few executions and suggest that an engine's evaluation should include repeatability alongside average answer quality.
Key claims
A 1.2-million-call study finds repeat executions can alter the output of semantic query engines.
Qualification: They challenge benchmark conclusions drawn from only a few executions and suggest that an engine's evaluation should include repeatability alongside average answer quality.
Evidence: source-2026-10-13-009
Sources
- arXiv preprint 2610.13823, original v1arXiv / submitting research authors · primary research
Corrections
: Publication note: The October 13, 2026 edition was prepared and published late on October 13 after the scheduled newsroom task could not start. It retains its October 13 edition date and 9:00 AM America/Detroit reveal time. Reports use evidence available before that cutoff; actual preparation and image-generation times are preserved in the provenance records.