Prefill and Decode Stop Sharing the Same Number Format

Transformer prefill and decode stress hardware differently, so FuTransformer gives them different kernels and precision. On Hopper GPUs, prefill uses FP8 with fused epilogues and direct cache placement; decode uses INT4 projections and a rotated INT4 key-value cache to reduce outlier error. On one H100, the author reports prefill up to 1.48 times faster than compiled PyTorch FP8 and INT4 decode 1.3 to 2.4 times faster than the best evaluated PyTorch path across tested shapes and batches. Full Llama-3.1-8B decoding reached 2.8 milliseconds per step at batch one in the reported setup, with a perplexity cost from quantization. These are implementation-specific comparisons, not universal model-serving gains.