dew.eval.calibration
Calibration of a classifier’s forecasts over K options, and the scores of it: the math any model’s calibration reads.
A forecast is a row of logits, or of probabilities, over its real options. Each function reads
held-out forecasts as arrays, so a model keeps its own policy of which forecasts it calibrates together
(dew.decision.calibration’s buckets of a question type and its option count):
fit_temperatureis the temperature that minimizes the held-out log loss.histogram_mapmaps a top probability to the held-out accuracy of its bin (histogram_mapped).abstention_thresholdis the lowest top probability whose accepted forecasts err at most a target.reliability_binscounts forecasts, confidence and right answers per bin of the top probability, additive over passes, andexpected_calibration_errorreads them.risk_coverage_areais the area under the risk-coverage curve, overtied_prefixes.
The two binnings are their references’: a histogram map’s bin is floor(top * bins), as a binning
calibrator’s (Laya’s fit_binning_map), and a reliability bin is closed on the right, as the expected
calibration error’s (Guo et al. 2017, Laya’s ece_score), so a top probability on an edge falls in the
upper bin of one and the lower of the other.
| Name | Summary |
|---|---|
abstention_threshold | The smallest top probability at which the forecasts at or above it err at most target_error, a group of tied forecasts counted as a whole (tied_prefixes); with conservative, the error is estimated as (errors + 1) / (accepted + 1). |
expected_calibration_error | The gap between confidence and accuracy, per bin weighted by its forecasts (reliability_bins). |
fit_temperature | The temperature minimizing mean log loss of logits [N, K] against the gold distributions targets [N, K] over the real options [N, K], by SciPy’s bounded L-BFGS-B on its inverse, at least 1e-3, in which the loss is convex: its derivative is the mean over forecasts of E_p[z] - E_gold[z]. |
histogram_map | The held-out accuracy of each of bins equal bins of the top probability on [0, 1], a forecast in the bin floor(top * bins) (the last holds 1); an empty bin keeps its midpoint. |
histogram_mapped | top mapped through a histogram’s bins (histogram_map). |
reliability_bins | Per bin of bins equal bins on (0, 1], closed on the right, the forecast count, the summed top probability tops and the number correct, [3, bins]: additive, so passes merge by sum. |
risk_coverage_area | The area under the risk-coverage curve: the error rate among the forecasts at or above each confidence, weighted by how many forecasts that confidence adds, so a group of equal confidences counts as a whole in any order. |
softmax | The options’ probabilities of logits at temperature, over the last axis, in float64; an option of logit -inf takes no mass. |
tied_prefixes | Each distinct confidence, highest first, with how many forecasts are at or above it and how many of those are correct: a group of tied forecasts counts as a whole, in any order. |
abstention_threshold
Section titled “abstention_threshold”def abstention_threshold( tops: np.ndarray, correct: np.ndarray, *, target_error: float, conservative: bool = True,) -> floatThe smallest top probability at which the forecasts at or above it err at most target_error,
a group of tied forecasts counted as a whole (tied_prefixes); with conservative, the error is
estimated as (errors + 1) / (accepted + 1). REJECT_ALL where no threshold meets the target.
expected_calibration_error
Section titled “expected_calibration_error”def expected_calibration_error(binned: np.ndarray) -> floatThe gap between confidence and accuracy, per bin weighted by its forecasts (reliability_bins).
fit_temperature
Section titled “fit_temperature”def fit_temperature(logits: np.ndarray, targets: np.ndarray, real: np.ndarray) -> floatThe temperature minimizing mean log loss of logits [N, K] against the gold distributions
targets [N, K] over the real options [N, K], by SciPy’s bounded L-BFGS-B on its inverse, at
least 1e-3, in which the loss is convex: its derivative is the mean over forecasts of
E_p[z] - E_gold[z]. Each gold row is read as a distribution, to unit mass: a batch packs it in
float32, a little short of unit mass, and the shortfall would scale with the logits’ common
offset.
histogram_map
Section titled “histogram_map”def histogram_map(tops: np.ndarray, correct: np.ndarray, bins: int) -> tuple[float, ...]The held-out accuracy of each of bins equal bins of the top probability on [0, 1], a forecast in
the bin floor(top * bins) (the last holds 1); an empty bin keeps its midpoint. A top probability
maps through it as histogram_mapped reads it.
histogram_mapped
Section titled “histogram_mapped”def histogram_mapped(values: tuple[float, ...], top: float) -> floattop mapped through a histogram’s bins (histogram_map).
reliability_bins
Section titled “reliability_bins”def reliability_bins(tops: np.ndarray, correct: np.ndarray, bins: int) -> np.ndarrayPer bin of bins equal bins on (0, 1], closed on the right, the forecast count, the summed top
probability tops and the number correct, [3, bins]: additive, so passes merge by sum.
risk_coverage_area
Section titled “risk_coverage_area”def risk_coverage_area(tops: np.ndarray, correct: np.ndarray) -> floatThe area under the risk-coverage curve: the error rate among the forecasts at or above each confidence, weighted by how many forecasts that confidence adds, so a group of equal confidences counts as a whole in any order. Lower is better; it rewards a confidence that ranks right answers above wrong ones.
softmax
Section titled “softmax”def softmax(logits: np.ndarray, temperature: float = 1.0) -> np.ndarrayThe options’ probabilities of logits at temperature, over the last axis, in float64; an option
of logit -inf takes no mass.
tied_prefixes
Section titled “tied_prefixes”def tied_prefixes( confidence: np.ndarray, correct: np.ndarray,) -> tuple[np.ndarray, np.ndarray, np.ndarray]Each distinct confidence, highest first, with how many forecasts are at or above it and how many
of those are correct: a group of tied forecasts counts as a whole, in any order. Abstention’s
thresholds and the risk-coverage area read their accepted forecasts off these.