Text Generation

Artificial Intelligence 24 Sep 2026 5 min read

Contrastive Search Penalizes Hidden-State Repetition During Decoding

Autoregressive generation exposes a distribution over the next token, but a decoder still has to choose which candidate to append. Contrastive search changes that choice by combining model probability with a penalty for candidates whose resulting hidden state is too similar to hidden states already present in the generated prefix. The mechanism operates only at inference time. It does not alter model parameters or the next-token distribution itself. Instead, it changes the ranking used to select a token from a restricted candidate set.

Artificial Intelligence Language Models

Artificial Intelligence 16 Sep 2026 6 min read

Control Repetition with Contrastive Search Decoding

Greedy decoding can keep selecting locally probable tokens even when the resulting continuation becomes repetitive. Sampling can break that pattern, but it does so by introducing randomness. Contrastive search takes a different route: it remains deterministic for fixed inputs and settings while scoring likely next-token candidates against a representation-level repetition penalty. The method combines two signals that describe different properties of a candidate. The language-model probability favors tokens that fit the current prefix. A degeneration penalty disfavors candidates whose new hidden representation is too similar to representations already present in the generated context.

Artificial Intelligence Large Language Models

Artificial Intelligence 12 Sep 2026 6 min read

Contrast Language Model Logits with Expert-Amateur Decoding

A language model can assign high probability to tokens that are fluent but generic. Contrastive decoding changes token selection by comparing a stronger expert model with a weaker amateur model at the same generation position. A token becomes attractive when the expert favors it more strongly than the amateur does. The comparison is not an unrestricted subtraction across the vocabulary. The original method also keeps candidate tokens inside a plausibility set defined by the expert. That constraint matters because a large expert-amateur score gap can otherwise promote a token that both models consider implausible.

Artificial Intelligence Large Language Models

Artificial Intelligence 05 Sep 2026 12 min read

Contrastive Decoding with Expert and Amateur Models

A language model can assign high probability to text that is fluent but unhelpfully generic, repetitive, or too close to an easy pattern. Changing temperature or top-p changes how tokens are sampled from one model’s distribution, but it does not ask a different question: which candidate tokens are especially characteristic of a stronger model rather than a weaker one? Contrastive decoding asks exactly that. It uses two language models at inference time: a stronger expert and a weaker amateur. A candidate is favored when the expert scores it well relative to the amateur, while a plausibility constraint prevents the decoder from choosing bizarre tokens merely because the amateur dislikes them even more.

Artificial Intelligence LLM