infrastructure
Optimizer State Fell From 27.7 Gigabytes to 0.16
TACO keeps one signed maximum-magnitude gradient component per column for full-parameter language-model tuning.
Summary
TACO keeps one signed maximum-magnitude gradient component per column for full-parameter language-model tuning.
TACO follows an operator-norm steepest-descent view but selects only the sign of the largest-magnitude entry in each column of two-dimensional weights. On OPT-13B, the authors report 0.16 gigabytes of persistent optimizer state versus 27.7 gigabytes for AdamW8bit, with peak training memory falling from 80.6 to 27.5 gigabytes at comparable accuracy and runtime. The method also enabled full-parameter tuning of tested 30–32-billion-parameter models on one 80-gigabyte H100; those source-reported results do not establish the same tradeoff for every architecture or task.
Why it matters
TACO keeps one signed maximum-magnitude gradient component per column for full-parameter language-model tuning.
Limits and context
- TACO follows an operator-norm steepest-descent view but selects only the sign of the largest-magnitude entry in each column of two-dimensional weights.
- The method also enabled full-parameter tuning of tested 30–32-billion-parameter models on one 80-gigabyte H100; those source-reported results do not establish the same tradeoff for every architecture or task.
Key claims
TACO keeps one signed maximum-magnitude gradient component per column for full-parameter language-model tuning.
Qualification: TACO follows an operator-norm steepest-descent view but selects only the sign of the largest-magnitude entry in each column of two-dimensional weights.
Evidence: source-2026-10-03-006
Sources
- arXiv preprint 2610.02199arXiv · primary research
Corrections
No corrections have been recorded for this story.