research
A Model Learned Language Without Seeing One
A byte-level transformer trained only on synthetic causal processes inferred real languages from context with frozen weights.
Summary
A byte-level transformer trained only on synthetic causal processes inferred real languages from context with frozen weights.
The 300-million-parameter Prior-Fitted Language Model never saw a real word during pretraining. Given long Wikipedia prefixes in six languages, it reduced bits per byte to between 0.9 and 2.4 at one million bytes of context, and it also learned patterns in numerals, code and speech. The experiment tests whether learning how to infer a generative process can substitute for memorizing one language in advance.
Why it matters
A byte-level transformer trained only on synthetic causal processes inferred real languages from context with frozen weights.
Limits and context
No additional limitation was separately recorded.
Key claims
A byte-level transformer trained only on synthetic causal processes inferred real languages from context with frozen weights.
Evidence: source-2026-10-06-019
Sources
- arXiv preprint 2610.05879arXiv · primary research
Corrections
No corrections have been recorded for this story.