Representation Learning

2 articles

Artificial Intelligence 06 Sep 2026 9 min read

Choose Pooling Strategies for Text Embeddings

A transformer usually produces one contextual representation for every input token. Many applications, however, need one vector for an entire sentence, query, or document. Semantic search, clustering, and similarity systems commonly compare these fixed-size vectors rather than every token representation separately. The operation that turns a variable number of token vectors into one vector is called pooling. It can look like a minor implementation detail, but changing it changes the representation being compared. Averaging every meaningful token, selecting a designated token, or emphasizing particular positions encodes different assumptions about where useful information lives.

Artificial Intelligence Embeddings

Artificial Intelligence 04 Sep 2026 10 min read

Negative Sampling for Representation Learning

Some representation-learning problems have an awkward shape: each training example has one observed target, but the model could choose from thousands or millions of alternatives. Computing a score and normalization term for every alternative on every update can become a major training cost. Negative sampling changes the training problem. Instead of comparing the observed target with every possible alternative, the model learns from the observed positive pair and a small set of deliberately sampled negative pairs. The update becomes much cheaper, but it also optimizes a sampled discrimination objective rather than the exact full-class objective.

Artificial Intelligence Machine Learning