The Hardest Compositional Test Fell to 15.8%

TranSGrid tests whether models can recombine known pieces when the task requires deductive, inductive and abductive reasoning together. Seven Transformers performed much worse than on an ordinary held-out set: the largest solved 79.6% of that set, 55.3% of TranSGrid and 15.8% of its hardest subset. Making actions nearly linear or making goals action-explicit returned scores toward the simpler held-out level. The result argues that familiar systematic-generalization tasks can hide the reasoning demands they claim to measure.