Tokenization

Artificial Intelligence / 23 Sep 2026 / 5 min read

Repetition Penalty Rewrites Logits for Seen Token IDs

A repetition penalty can act before sampling by changing the logits of token IDs that already occur in a selected token history. The operation does not need to compare words, phrases, or rendered strings. Its unit can be the tokenizer’s integer ID, which gives the mechanism a narrower meaning than its name may suggest. That distinction matters when a decoder emits subword tokens. Two strings that appear similar to a person can map to different token sequences, while a token reused inside unrelated words can still be marked as previously seen.

Artificial Intelligence Language Models

Artificial Intelligence / 16 Sep 2026 / 6 min read

Mask Padding Tokens in Language Model Loss

Variable-length text batches are commonly padded into rectangular tensors. The extra positions simplify batching, but they are not ordinary training targets. If padded target positions contribute to cross-entropy, the optimizer receives gradients for synthetic symbols that were introduced only to align tensor shapes. Preventing that signal requires a loss mask. An attention mask can stop selected positions from participating in attention, but that does not by itself remove their target terms from the objective.

Artificial Intelligence Language Models

Artificial Intelligence / 08 Sep 2026 / 10 min read

Avoid Tokenization Boundary Failures in LLM Generation

A language model application usually treats a prompt as text: provide a prefix, then ask the model to continue it. The model sees something more specific. Its tokenizer first converts that text into tokens, and the end of the prompt forces the last token to end at exactly that position. That detail can matter when the prompt ends at a character position that would normally fall inside a larger token if the prompt and its continuation were tokenized together. The resulting tokenization boundary problem, also called the partial token problem, can make an otherwise natural continuation unexpectedly unlikely.

Artificial Intelligence LLM

Artificial Intelligence / 07 Sep 2026 / 12 min read

Measure Tokenization Efficiency Across Languages

Two prompts can communicate roughly the same amount of information and still consume very different numbers of model tokens. The difference can appear between languages, writing systems, domains, or even formatting styles. That matters because language-model systems usually operate on tokens rather than characters or words. A context window is measured in tokens. Many hosted APIs account for usage in tokens. Longer token sequences can also increase inference work, although the exact latency and compute effect depends on the model, serving stack, batching, caching, and whether the tokens belong to the input or generated output.

Artificial Intelligence LLM

Artificial Intelligence / 03 Sep 2026 / 7 min read

Tokenization in Large Language Models

Large language models do not read text as words or characters in the way people do. Before text reaches the model, a tokenizer converts it into a sequence of discrete units called tokens and maps those tokens to numerical identifiers. Tokenization is easy to overlook because most model APIs perform it automatically. Yet token boundaries affect context-window usage, inference cost, truncation, multilingual behavior, and even whether two visually similar strings are represented in similar ways. These details explain many LLM behaviors that otherwise look inconsistent.

Artificial Intelligence LLM

Artificial Intelligence / 03 Sep 2026 / 10 min read

Tokenization in Language Models

Language models do not read text as a sequence of words. Before a model can process a prompt, a tokenizer converts the text into a sequence of token IDs from a fixed vocabulary. The model operates on those IDs, and generated IDs are later converted back into text. This extra layer is easy to ignore because most model APIs accept ordinary strings. But tokenization affects how much text fits in a context window, how usage-based costs are calculated, how text is truncated or split, and why seemingly small formatting changes can alter model behavior.

Artificial Intelligence Language Models