Parallel Language Models via Variational Decoding

Language models that write several coordinated tokens per step, trained from scratch with one loss.

2.3-5.7×

faster

Inference speedup after post-training

1.3B model matches or beats an identically trained AR model

How it works

One token per pass
Autoregressive models run a full forward pass for every token. Bigger models make this latency worse.
Random codes carry dependencies
Each position gets a random auxiliary vector. A shared-weight decoder maps a block of them to consistent tokens in one call.
Faster at AR quality
From 80M to 3B parameters, loss stays near AR. Tokens per call grow with size. Post-trained models answer several times faster.

From the paper