Kernel calibration test

cal_test() tests the null hypothesis that a probabilistic classifier is calibrated, using the squared kernel calibration error of Widmann, Lindsten and Zachariah (2019) as the test statistic. Two tests are available: a bootstrap test for the quadratic unbiased estimator (method = "bootstrap", the default) and an asymptotic normal test based on the linear-time unbiased estimator (method = "asymptotic"). The bootstrap test is the default because the quadratic estimator is more powerful than the linear one (Widmann et al. 2019, Section 7.2). Both build on the kernel machinery of skce() and mmce().

Usage

cal_test(
  p,
  y,
  method = c("bootstrap", "asymptotic"),
  bandwidth = 0.2,
  n_boot = 999,
  type = c("canonical", "confidence"),
  ...
)

Arguments

p: Predicted probabilities. A numeric vector in [0, 1] for binary problems, or a numeric matrix with one column per class for multiclass problems. Matrix inputs must have finite entries in [0, 1], at least two columns, and rows summing to one within absolute tolerance 1e-6.
y: Outcome labels. A vector coded as 0 and 1 for binary problems, or a factor or vector of integer class codes in 1:K for multiclass problems.
method: Test to perform: "bootstrap" (the default) for the wild-bootstrap test based on the more powerful quadratic estimator \(\widehat{\mathrm{SKCE}}_{uq}\), or "asymptotic" for the faster \(O(n)\) normal test based on the linear estimator \(\widehat{\mathrm{SKCE}}_{ul}\).
bandwidth: Bandwidth of the Laplacian kernel. Either a single positive number (the default 0.2) or the string "median", which sets the bandwidth to the median of the positive pairwise distances (the median heuristic of Widmann et al. 2019, Section 7). The median heuristic is recommended for the canonical multiclass form, where the scale of the simplex distances depends on the number of classes; the fixed 0.2 is a reproducible default for the binary and confidence forms, whose confidences lie in [0, 1].
n_boot: Number of bootstrap resamples for method = "bootstrap". A single positive integer.
type: Multiclass calibration target tested, one of "canonical" (strong calibration of the full probability vector, the default) or "confidence" (top-label confidence). Ignored for binary vector inputs. The classwise average is not a valid test target; use skce() with type = "classwise" for a point estimate.
...: Unused; present for future extension.

Value

An object of class c("cal_test", "htest") with components statistic, p.value, method, data.name, and estimate (the SKCE estimate used). It prints in the style of base R hypothesis tests.

Details

For binary inputs the test assesses binary calibration. For a multiclass probability matrix, type selects the target: "canonical" tests strong calibration of the full probability vector with the matrix-valued kernel, and "confidence" tests calibration of the top-label probability. The classwise (one-vs-rest) average is available as a point estimate from skce() but not as a test, because averaging per-class kernels does not yield a single positive-definite kernel with a valid null distribution for the disjoint-pair and U-statistic constructions.

The asymptotic test uses the linear estimator \(\widehat{\mathrm{SKCE}}_{ul}\) over the disjoint pairs \((1, 2), (3, 4), \ldots\). Under \(H_0\) it is asymptotically normal (Widmann et al. 2019, Lemma 3): with \(\hat\sigma\) the sample standard deviation of the disjoint-pair terms \(h_{2i-1,2i}\), the standardised statistic is \(\sqrt{\lfloor n/2 \rfloor}\, \widehat{\mathrm{SKCE}}_{ul} / \hat\sigma\), and the one-sided p-value is \(1 - \Phi(\cdot)\). This test is \(O(n)\) and needs no resampling.

The bootstrap test targets the quadratic estimator \(\widehat{\mathrm{SKCE}}_{uq}\), which is a degenerate \(U\)-statistic under strong calibration with no closed-form limit (Widmann et al. 2019, Theorem G.2). The null distribution is approximated by a wild bootstrap of the centred kernel terms \(h_{ij}\) with Rademacher multipliers (Arcones and Giné 1992), the analogue of the bootstrap for the maximum mean discrepancy two-sample test. The p-value is the fraction of bootstrap statistics that equal or exceed the observed estimator, with the customary add-one correction. This test is more powerful but costs \(O(B n^2)\) for n_boot resamples.

Two caveats are worth stating. Consistency-resampling tests for binned calibration error tend to over-reject calibrated models (Widmann et al. 2019, Section 7.2); the kernel tests here are the recommended replacement. Resampling procedures can also undercover or misstate uncertainty for models with small calibration error in finite samples (Sun et al. 2024); the asymptotic test is preferable when its normal approximation is adequate.

References

Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. Advances in Neural Information Processing Systems 32. arXiv:1910.11385.

Sun, Y., Chaudhari, P., Barnett, I. J., & Dobriban, E. (2024). A confidence interval for the l2 expected calibration error. arXiv:2408.08998.

Examples

set.seed(40)
p <- stats::runif(300)
y <- rbinom(300, 1, p)

# Calibrated by construction: large p-value expected.
cal_test(p, y)
#> 
#> ── Kernel calibration test ─────────────────────────────────────────────────────
#> Method: Bootstrap kernel calibration test (SKCE_uq, 999 resamples)
#> Target: binary calibration
#> Data: p and y
#> Statistic: SKCE_uq = 9.020202e-05
#> Estimate: SKCE_uq = 9.020202e-05
#> p-value: 0.284
#> Alternative hypothesis: the model is not calibrated.

# Miscalibrated (overconfident) predictions: small p-value expected.
set.seed(41)
p2 <- stats::runif(300)
y2 <- rbinom(300, 1, pmin(pmax(p2 - 0.25, 0), 1))
cal_test(p2, y2)
#> 
#> ── Kernel calibration test ─────────────────────────────────────────────────────
#> Method: Bootstrap kernel calibration test (SKCE_uq, 999 resamples)
#> Target: binary calibration
#> Data: p2 and y2
#> Statistic: SKCE_uq = 0.02973914
#> Estimate: SKCE_uq = 0.02973914
#> p-value: 0.001
#> Alternative hypothesis: the model is not calibrated.