The Database Answers the Question That Should Have Stopped It

A database assistant can produce perfectly executable SQL for a question the database cannot truthfully answer. PremiseSQL tests whether text-to-SQL systems recognize that failure before writing the query. The benchmark pairs 1,800 flawed requests across four kinds of premise error with 1,800 valid originals, then evaluates models under three prompting settings. The authors report that frontier systems are strongly premise-trusting by default: they rarely critique a flawed request or abstain from generating misleading SQL. Prompts that explicitly encourage critique improve detection and refusal, but they also create a second failure mode—valid questions are more often mislabeled as invalid, and useful SQL is suppressed. The benchmark does not establish how often flawed premises occur in production databases, and its results depend on the tested schemas, models and prompts. Its sharper lesson is architectural: execution success is not the same as question validity. A reliable data assistant needs a gate that can inspect what the user assumes, preserve that judgment through query planning and still avoid turning healthy skepticism into blanket refusal.