Uncertainty

Artificial Intelligence / 17 Sep 2026 / 6 min read

Use Selective Classification to Trade Coverage for Error Rate

A classifier usually returns a label for every input, even when its score distribution is nearly tied or the input sits far from familiar data. Selective classification changes that interface: the system may return a prediction or abstain. The acceptance rule then determines both how many inputs receive predictions and how often those accepted predictions are wrong. This is not the same as making the classifier intrinsically more accurate. Abstention moves some cases out of the automatic-decision set. Its value depends on whether the selection score ranks difficult cases well enough for rejected inputs to contain a disproportionate share of errors.

Artificial Intelligence Classification

Artificial Intelligence / 17 Sep 2026 / 6 min read

Conformal Prediction Sets Need Exchangeable Calibration Data

A classifier that emits 0.93 for one class does not automatically provide a statistical statement that the class is correct with probability 0.93. Conformal prediction takes a different route: it uses held-out labeled examples to construct a set of candidate labels with a target marginal coverage level. For split conformal classification, the base model can remain fixed. The guarantee comes from ranking a test example’s conformity or nonconformity score against scores computed on an exchangeable calibration sample. That assumption is the part that gives the coverage statement its scope.

Artificial Intelligence Conformal Prediction

Artificial Intelligence / 13 Sep 2026 / 7 min read

Build Classification Sets with Split Conformal Prediction

A classifier normally returns one label or a vector of class scores. Neither output directly states how many labels should remain plausible when the system needs a controlled error rate. Split conformal prediction adds a calibration layer that turns those scores into prediction sets. The useful property is not that every individual set has a fixed probability of containing the correct label. Under the standard exchangeability assumption, split conformal methods target marginal coverage across new examples. That distinction shapes both implementation and interpretation.

Artificial Intelligence Classification

Artificial Intelligence / 12 Sep 2026 / 10 min read

Route Uncertain Classifier Predictions with Selective Classification

Route Uncertain Classifier Predictions with Selective Classification A classifier does not have to answer every request. In systems where a bad prediction is costly, forcing a label on every input can be a poor product decision even when the model has strong average accuracy. Selective classification adds a reject option. The system returns a model prediction only when an acceptance rule considers the case suitable; otherwise it abstains and sends the case to a fallback such as human review, a second model, or a request for more information.

Artificial Intelligence Model Evaluation

Artificial Intelligence / 11 Sep 2026 / 10 min read

Build Prediction Sets with Conformal Prediction

Build Prediction Sets with Conformal Prediction A classifier usually returns one label even when several labels are plausible. That is convenient for software interfaces, but it can hide uncertainty exactly where mistakes are expensive. A document router might be unsure between billing and account, yet an argmax still emits one of them. Conformal prediction offers another interface: return a set of labels sized according to the evidence. Easy inputs can produce one label. Ambiguous inputs can produce several. Under specific assumptions, the procedure also gives a finite-sample coverage guarantee.

Artificial Intelligence Model Evaluation

Artificial Intelligence / 09 Sep 2026 / 9 min read

Use Predictive Entropy to Detect Uncertain Classifications

Use Predictive Entropy to Detect Uncertain Classifications A classifier can return the same predicted label for two inputs while being much less certain about one of them. If an application only keeps the winning label, that difference disappears. Predictive entropy gives you a compact way to preserve it. It summarizes how spread out a classifier’s predicted probability distribution is: concentrated probability produces low entropy, while probability spread across several classes produces higher entropy.

Artificial Intelligence Machine Learning

Artificial Intelligence / 09 Sep 2026 / 10 min read

Select Active Learning Examples with BALD

When labels are expensive, training on every available example may be impractical. An active learning system tries to spend its labeling budget selectively: train a model on the labels already available, score unlabeled examples, request labels for useful examples, then retrain. A common first idea is to label the examples with the highest predictive entropy. That can help, but entropy mixes together two different reasons for uncertainty. The model may be uncertain because it does not yet know enough, or because the input itself is genuinely ambiguous. More labels are most valuable for the first case.

Artificial Intelligence Active Learning

Artificial Intelligence / 08 Sep 2026 / 8 min read

Estimate LLM Uncertainty with Semantic Entropy

A language model can produce a fluent answer even when it is uncertain. Token probabilities help describe uncertainty during generation, but they can be misleading at the answer level because many different strings can express the same meaning. Consider a question whose correct answer is Paris. A model might generate Paris, The answer is Paris, and France's capital is Paris. These strings differ, yet they represent the same answer. Treating them as three unrelated outcomes exaggerates the apparent uncertainty.

Artificial Intelligence LLM

Artificial Intelligence / 06 Sep 2026 / 9 min read

Inspect Language Model Uncertainty with Token Entropy

A language model can produce fluent text even when several continuations look similarly plausible to the model. Looking only at the selected token hides that ambiguity: a token chosen with probability 0.90 and one chosen from a nearly even 0.51 versus 0.49 split both appear as a single output token. Token entropy summarizes how spread out the model’s next-token probability distribution is. It can help developers inspect uncertain generation steps, compare decoding behavior under controlled conditions, and build diagnostic signals for evaluation. But entropy is not a probability that the model is correct, and using it as one leads to unreliable decisions.

Artificial Intelligence Large Language Models

Artificial Intelligence / 05 Sep 2026 / 12 min read

Split Conformal Classification for Prediction Sets

A classifier usually returns one label or a vector of scores. That is convenient when the application must choose one answer, but it hides an important distinction: some inputs strongly support one class, while others leave several classes plausible. Conformal prediction provides a way to expose that ambiguity. For classification, it can return a prediction set containing one or more labels instead of forcing every input into a single choice. With an appropriate calibration procedure and statistical assumptions, the method can target a long-run coverage level such as 90%: roughly speaking, the true label should appear in the prediction set for at least that proportion of future examples.

Artificial Intelligence Machine Learning

Artificial Intelligence / 05 Sep 2026 / 12 min read

Estimate Neural Network Uncertainty with Monte Carlo Dropout

A neural network can produce a confident-looking prediction even when the input is unlike the data it learned from. A single output such as 0.93 tells you what one forward pass predicts; by itself, it does not tell you how sensitive that prediction is to uncertainty in the learned model. Monte Carlo dropout is a practical way to obtain an additional uncertainty signal from some neural networks that were trained with dropout. Instead of disabling dropout at inference time, you keep it active, run the same input through the network multiple times, and inspect how much the predictions vary.

Artificial Intelligence Machine Learning