Bootstrap confidence interval for a calibration metric

cal_ci() returns a percentile bootstrap confidence interval for a calibration-error metric by resampling prediction-label pairs with replacement and recomputing the metric. It works for the binned errors (ece(), mce(), ace()), the kernel error mmce(), and the squared kernel calibration error skce().

Usage

cal_ci(
  p,
  y,
  metric = c("ece", "skce", "mmce", "mce", "ace"),
  conf_level = 0.95,
  n_boot = 999,
  bins = 10,
  ...
)

Arguments

p: Predicted probabilities. A numeric vector in [0, 1] for binary problems, or a numeric matrix with one column per class for multiclass problems. Matrix inputs must have finite entries in [0, 1], at least two columns, and rows summing to one within absolute tolerance 1e-6.
y: Outcome labels. A vector coded as 0 and 1 for binary problems, or a factor or vector of integer class codes in 1:K for multiclass problems.
metric: Which metric to bootstrap: one of "ece", "skce", "mmce", "mce", or "ace".
conf_level: Confidence level for the two-sided interval. A single number in (0, 1).
n_boot: Number of bootstrap resamples. A single positive integer.
bins: Number of bins for the binned metrics ("ece", "mce", "ace").
...: Additional arguments passed to the underlying metric (for example type for multiclass inputs, estimator for "skce", or debiased and strategy for "ece").

Value

An object of class cal_ci, a list with estimate, lower, upper, conf_level, metric, and method, with a print() method.

Details

For n_boot bootstrap resamples, the observations are sampled with replacement and the chosen metric is recomputed. The two-sided conf_level percentile interval uses the empirical quantiles of the bootstrap distribution at \(\alpha/2\) and \(1 - \alpha/2\) with \(\alpha = 1 - \) conf_level. The point estimate is the metric on the full sample. Because the percentile interval of a biased metric need not bracket the point estimate, the reported interval is widened when necessary so that it always contains the estimate. The interval is clamped at zero from below for the nonnegative metrics (ece, mmce, mce, ace); the unbiased skce estimators can be negative, so their interval is not clamped.

Percentile bootstrap intervals make no distributional assumption, but Sun et al. (2024) show they can undercover in finite samples, most severely for models with small calibration error. Treat the interval as approximate, especially near zero.

References

Sun, Y., Chaudhari, P., Barnett, I. J., & Dobriban, E. (2024). A confidence interval for the l2 expected calibration error. arXiv:2408.08998.

Examples

set.seed(42)
p <- stats::runif(300)
y <- rbinom(300, 1, p)

cal_ci(p, y, metric = "ece", bins = 10, n_boot = 199)
#> 
#> ── Calibration metric confidence interval ──────────────────────────────────────
#> Metric: ECE
#> Estimate: 0.04587
#> Confidence level: 95.0%
#> Interval: [0.04587, 0.1089]
#> Method: percentile bootstrap (199 resamples)
cal_ci(p, y, metric = "skce", n_boot = 199)
#> 
#> ── Calibration metric confidence interval ──────────────────────────────────────
#> Metric: SKCE
#> Estimate: -0.0001383
#> Confidence level: 95.0%
#> Interval: [-0.000219, 0.001565]
#> Method: percentile bootstrap (199 resamples)