Skip to contents

probcal

probcal calibrates predicted class probabilities after a classifier has already been fitted. Many classifiers rank observations well but assign probabilities that are too high, too low, or distorted in different parts of the range. If observations predicted near 0.80 do not occur about 80 percent of the time, the scores may still be useful for ranking, but they are unreliable for decisions that use the probability itself.

The package is designed for workflows where model fitting and probability calibration are separate steps. Fit a binary or multiclass classifier with any R modeling tool, reserve calibration data, and pass the resulting scores, logits, or probability matrices to probcal. Fitted calibrators are S3 objects: inspect them with print() or summary() and apply them to new predictions with predict().

Calibration is useful when predictions feed risk thresholds, triage rules, forecasts, or decision analyses with unequal costs. probcal provides post-hoc calibration methods, calibration metrics, reliability diagrams, and out-of-fold calibration for settings where a separate calibration split is small. Beyond these point summaries, it adds a calibration-inference layer: debiased calibration-error estimators, bootstrap confidence intervals, and a kernel calibration test that returns a p-value, so “is this model calibrated?” can be answered with a test instead of a single number.

What It Does

  • Fits post-hoc calibrators for binary and multiclass probabilities.
  • Works with scores, logits, probability vectors, and probability matrices.
  • Uses an S3 interface: fit with cal_*(), inspect with print() or summary(), and apply with predict().
  • Includes parametric methods, nonparametric methods, calibration metrics, reliability diagrams, and cross-validated calibration.
  • Tests calibration: cal_test() implements the kernel calibration tests of Widmann et al. (2019) for binary and multiclass predictions, including the strong (canonical) case in which the full probability vector is assessed.
  • Quantifies uncertainty: skce() provides unbiased kernel calibration-error estimators, cal_ci() bootstrap confidence intervals, and ece(debiased = TRUE) the debiased estimator of Kumar et al. (2019).
  • Runs as a native R package with no Python or Julia runtime. To our knowledge, probcal is the first package to combine binary and multiclass recalibration with this calibration inference in native R.

Installation

From CRAN:

install.packages("probcal")

Development version from GitHub:

# install.packages("remotes")
remotes::install_github("prdm0/probcal")

Binary Workflow

This example simulates a classifier that is too confident. Calibration is fitted on a calibration split and evaluated on a test split.

library(probcal)
library(dplyr)

set.seed(42)
n <- 600
predictions <- data.frame(x = rnorm(n)) |>
  mutate(
    true_p = inv_logit(-0.4 + 1.1 * x),
    y = rbinom(n(), 1, true_p),
    raw_logits = 1.8 * (-0.4 + 1.1 * x),
    raw_p = inv_logit(raw_logits),
    split = sample(rep(c("calibration", "test"), each = n / 2))
  )

calibration <- predictions |>
  filter(split == "calibration")

test <- predictions |>
  filter(split == "test")

fit <- cal_beta(calibration$raw_p, calibration$y)

test <- test |>
  mutate(calibrated = predict(fit, raw_p))

test |>
  summarise(
    raw_ece = ece(raw_p, y, bins = 10),
    calibrated_ece = ece(calibrated, y, bins = 10)
  )

reliability_diagram(test$calibrated, test$y, bins = 10)

Is the Model Calibrated? Test It

The metrics above are point summaries. The calibration-inference layer turns the question into a hypothesis test and an interval. The raw overconfident probabilities are rejected; the beta-calibrated ones are not.

set.seed(1)
cal_test(test$raw_p, test$y)$p.value
cal_test(test$calibrated, test$y)$p.value

cal_ci(test$calibrated, test$y, metric = "ece", bins = 10)

cal_test() also assesses strong (canonical) calibration of a multiclass probability matrix, and skce() exposes the unbiased kernel calibration-error estimators behind the test.

Multiclass Workflow

For multiclass calibration, pass a matrix with one column per class. Temperature and vector scaling use logits. Dirichlet calibration and one-vs-rest calibration use probability matrices.

set.seed(2024)
n <- 600
k <- 3

true_prob <- matrix(runif(n * k), ncol = k)
true_prob <- true_prob / rowSums(true_prob)
labels <- apply(true_prob, 1, function(row) sample.int(k, 1, prob = row))

raw_prob <- true_prob^2
raw_prob <- raw_prob / rowSums(raw_prob)

split <- sample(rep(c("calibration", "test"), each = n / 2))

fit <- cal_dirichlet(
  raw_prob[split == "calibration", ],
  labels[split == "calibration"]
)

calibrated <- predict(fit, raw_prob[split == "test", ])

ece(raw_prob[split == "test", ], labels[split == "test"], type = "classwise")
ece(calibrated, labels[split == "test"], type = "classwise")

Included Methods

Binary calibrators:

Multiclass calibrators:

Metrics and diagnostics:

  • ece(): Expected Calibration Error, with debiased, strategy (equal-width or equal-mass bins), and norm ("l1" or "l2") options.
  • mce(): Maximum Calibration Error.
  • ace(): Average Calibration Error.
  • mmce(): Maximum Mean Calibration Error.
  • reliability_diagram(): reliability diagram returned as a ggplot object.

Calibration inference:

  • skce(): squared kernel calibration error of Widmann et al. (2019), with unbiased quadratic ("uq"), unbiased linear-time ("ul"), and biased estimators, for binary, confidence, classwise, and strong (canonical) multiclass targets.
  • cal_test(): kernel calibration hypothesis test (wild bootstrap by default, asymptotic normal as a fast alternative), returning an object of class c("cal_test", "htest") with a p-value.
  • cal_ci(): percentile bootstrap confidence interval for ece(), mce(), ace(), mmce(), or skce().

For small calibration sets, cal_cv() provides out-of-fold calibration. It works on supplied scores, logits, or probabilities and does not train the underlying classifier.

Choosing a Calibrator

Method Input scale Typical use
cal_platt() score or probability simple logistic calibration
cal_temperature() logits, vector or matrix overconfident logit-based models
cal_beta() probability asymmetric binary probability distortion
cal_isotonic() probability flexible monotone calibration with enough data
cal_histogram() probability auditable bin-level calibration
cal_vector_scaling() logit matrix per-class scale and bias for several classes
cal_dirichlet() probability matrix multiclass generalization of beta calibration
cal_ovr() matrix on the base-method scale apply a binary method to each class

See vignettes/choosing-a-calibrator.Rmd and vignettes/multiclass.Rmd for longer examples.

Numerical Validation

The test suite contains optional checks against external implementations. These checks are used during development and are skipped when the optional software is not available. They do not make Python or any external calibration package a runtime dependency.

Target Coverage Dependency
External confidence metrics selected ece(), mce(), ace(), and multiclass ECE cases reticulate and optional Python software
External histogram binning selected equal-width binning cases reticulate and optional Python software
R beta calibration implementation selected cal_beta() predictions optional R package

Scope

probcal covers post-hoc probability calibration for binary and multiclass classification. Bayesian binning, near-isotonic ensembles, object detection calibration, regression uncertainty calibration, and neural calibration methods are not part of the current API.

Citation

Use citation("probcal") to cite the package. The citation metadata includes the package author, ORCID, version, and project URL.