Regularization

Artificial Intelligence / 13 Sep 2026 / 6 min read

Regularize Classifier Targets with Label Smoothing

A classifier trained with one-hot targets is rewarded for pushing the target class probability toward one and every other class probability toward zero. Cross-entropy keeps applying pressure in that direction even after the predicted class is already correct. Label smoothing changes that pressure by replacing the exact one-hot target with a distribution that reserves some mass for other classes. That small change affects more than the target tensor. It changes the gradient on every output logit, limits the incentive for extreme class separation, and alters how predicted probabilities should be interpreted.

Artificial Intelligence Classification

Artificial Intelligence / 12 Sep 2026 / 7 min read

Soften Classification Targets with Label Smoothing

A classifier trained with one-hot targets receives a strong signal to push the target class probability toward one and every other class probability toward zero. Cross-entropy supports that behavior even after the predicted class is already correct: making the target probability more extreme can still reduce the loss. Label smoothing changes the target distribution before cross-entropy is computed. Instead of assigning all target mass to one class, it reserves a small amount for the remaining classes. This alters the gradient applied to the logits and reduces pressure toward extreme output distributions.

Artificial Intelligence Neural Networks

Artificial Intelligence / 11 Sep 2026 / 9 min read

Use Token Dropout to Train Robust Sequence Models

Use Token Dropout to Train Robust Sequence Models A sequence model can become too dependent on a few easy input clues. Remove one field, truncate a message, or corrupt a token at inference time, and a prediction that looked reliable on clean validation data may change sharply. Token dropout is a simple training-time corruption technique: randomly hide some input tokens while keeping the learning target unchanged. The model is forced to solve some training examples without every usual clue. Used carefully, this can reduce brittle dependence on individual tokens. Used carelessly, it can destroy information the task genuinely requires.

Artificial Intelligence Machine Learning

Artificial Intelligence / 11 Sep 2026 / 9 min read

Control Neural Network Weight Scale with Spectral Normalization

Control Neural Network Weight Scale with Spectral Normalization A neural network layer can amplify a small change in its input into a much larger change in its output. Large amplification isn’t automatically a defect, but it can make some models harder to control during training, especially when one network is reacting to another as in a generative adversarial network. Spectral normalization puts a direct constraint on that amplification for a linear transformation. It rescales a weight matrix using its largest singular value, called the spectral norm. The result is a simple mechanism with a precise local meaning: under the Euclidean norm, the normalized linear map cannot stretch a vector by more than the chosen scale.

Artificial Intelligence Neural Networks

Artificial Intelligence / 10 Sep 2026 / 9 min read

Regularize Neural Networks with Mixup

Regularize Neural Networks with Mixup A neural network can fit its training examples while behaving unpredictably in the space between them. If two nearby inputs belong to different classes, standard training tells the model what to do at the endpoints but often says little about intermediate points. Mixup changes that training signal. Instead of training only on individual examples, it creates synthetic examples by interpolating pairs of inputs and their labels. The model is then asked to make a correspondingly mixed prediction. This acts as a regularizer because it constrains how predictions may change between training examples.

Artificial Intelligence Machine Learning