Autoregressive Models

2 articles

Artificial Intelligence 24 Sep 2026 6 min read

Speculative Decoding Verifies Draft Tokens Without Changing the Target Sampling Distribution

Autoregressive decoding normally commits one token after each target-model evaluation. A sequence of K generated tokens therefore creates a serial dependency chain: token t+1 cannot be sampled until token t is fixed and becomes part of the prefix. Speculative decoding changes the amount of useful work obtained from a target-model call without removing that causal dependency. A cheaper draft model first proposes several continuation tokens. The target model then evaluates those candidate positions together. Tokens whose draft probabilities are compatible with the target distribution can be accepted, while the first rejected position is corrected using a residual distribution. The resulting samples follow the target model’s distribution when the acceptance and correction procedure is implemented as specified.

Artificial Intelligence LLM Inference

Artificial Intelligence 16 Sep 2026 6 min read

Account for Exposure Bias in Autoregressive Decoding

An autoregressive model can receive cleaner context during training than it receives during generation. Under teacher forcing, the next-token prediction is conditioned on a reference prefix from the training sequence. During free-running decoding, the model instead conditions on tokens it generated itself. Once a generated token differs from the intended continuation, later predictions operate on a prefix that training may have represented less often. This mismatch is commonly called exposure bias. It is not simply a claim that autoregressive models make errors. The specific issue is that the distribution of prefixes presented to the model can change between optimization and generation, and an early deviation can change every subsequent conditional prediction.

Artificial Intelligence LLM