What data format does the checker accept?

Paste or upload CSV or TSV data with a header row. The probability column must be named probability or p, and the binary outcome column must be outcome, y, or correct. Outcomes can be 0/1 or true/false.

What does the Brier score measure?

The Brier score is the mean squared difference between each predicted probability and its binary outcome. A lower score indicates smaller probability errors for this dataset.

Why can ECE change when I change the number of bins?

Expected calibration error groups observations into probability intervals and takes a count-weighted average of the gaps between mean probability and outcome rate. Different bin widths group observations differently, so ECE can change.

Does a softmax score mean a model is calibrated?

No. Softmax converts model scores into values that sum to one, but that normalization does not show whether predictions at a given confidence level are correct at the same rate. Calibration must be checked against observed outcomes.

Research workbench

Probability Calibration Checker

Check predicted probabilities against observed outcomes

Input data

Probability and outcome data

Choose a CSV/TSV fileBins

CSV and TSV are supported; outcomes may be 0/1 or true/false. Pasted and uploaded data is processed only in your browser.

Results

Paste or load data with a header row to see calibration results.

Understanding probability calibration

A probability forecast is calibrated when predictions made at a given probability correspond to the observed outcome rate over repeated cases. For example, among cases assigned a probability near 0.7, the outcome should occur about 70% of the time. Calibration describes agreement between probabilities and outcomes; it does not by itself measure whether a model separates positive from negative cases.

Brier score

The Brier score is the mean of (p − y)², where p is the predicted probability and y is the binary outcome. It is a proper scoring rule for probability forecasts: lower values indicate smaller squared probability errors on the supplied observations. Interpret it with the outcome prevalence and the intended comparison in mind.

Expected calibration error

Expected calibration error (ECE) divides the probability range into equal-width bins. For every non-empty bin, it compares the mean predicted probability with the observed outcome rate, then weights the absolute gap by that bin's share of observations. Because grouping affects the result, report the bin count alongside ECE. This checker uses half-open intervals, with the final interval including 1.0.

Reading the reliability diagram

The horizontal axis shows mean predicted probability in each bin and the vertical axis shows its observed outcome rate. Points near the diagonal indicate closer agreement. The points are sized by the number of rows in each bin, so a point based on few observations is visibly smaller. The diagram is a descriptive summary; it does not show uncertainty intervals.

Softmax confidence and calibration

A softmax layer turns a set of model scores into normalized class probabilities. Normalization alone does not guarantee calibration: a model can produce probabilities that are systematically too high or too low. Compare predictions with outcomes on data that represents the use case, and keep any calibration procedure separate from the data used for final evaluation.