Squared Kernel Calibration Error

skce() returns an estimate of the squared kernel calibration error (SKCE) of Widmann, Lindsten and Zachariah (2019). The SKCE is a binning-free, kernel-based measure of calibration error built from a residual and a positive-definite kernel, generalising the maximum mean calibration error of mmce(). Three estimators are available: the biased plug-in $V$-statistic that keeps the diagonal terms, and two unbiased estimators that remove the diagonal-induced upward bias.

Usage

skce(
  p,
  y,
  estimator = c("uq", "ul", "biased"),
  bandwidth = 0.2,
  type = c("canonical", "confidence", "classwise")
)

Arguments

p: Predicted probabilities. A numeric vector in [0, 1] for binary problems, or a numeric matrix with one column per class for multiclass problems. Matrix inputs must have finite entries in [0, 1], at least two columns, and rows summing to one within absolute tolerance 1e-6.
y: Outcome labels. A vector coded as 0 and 1 for binary problems, or a factor or vector of integer class codes in 1:K for multiclass problems.
estimator: Which estimator to return: "uq" for the unbiased $O(n^2)$ $U$-statistic (the default), "ul" for the unbiased linear-time estimator, or "biased" for the biased $V$-statistic that matches mmce()^2.
bandwidth: Bandwidth of the Laplacian kernel. Either a single positive number (the default 0.2) or the string "median", which sets the bandwidth to the median of the positive pairwise distances (the median heuristic of Widmann et al. 2019, Section 7). The median heuristic is recommended for the canonical multiclass form, where the scale of the simplex distances depends on the number of classes; the fixed 0.2 is a reproducible default for the binary and confidence forms, whose confidences lie in [0, 1].
type: Multiclass calibration target, one of "canonical" (strong calibration of the full probability vector, the default for matrix inputs), "confidence" (top-label confidence), or "classwise" (mean over one-vs-rest columns). Ignored for binary vector inputs.

Value

A single numeric value, the squared kernel calibration error estimate. For estimator = "biased" the value is nonnegative; the unbiased estimators may be negative in finite samples.

Details

For a binary input the residual compares the event indicator y with the predicted event probability p, and the kernel is the scalar Laplacian kernel on p. For a multiclass probability matrix, type selects the calibration target. The "confidence" form reduces to the top-label probability and whether the predicted class is correct, exactly as in mmce(). The "canonical" (strong) form uses the full probability vector with a matrix-valued kernel and characterises calibration of the entire forecast distribution. The "classwise" form averages the binary SKCE over the one-vs-rest columns.

In the binary or confidence case skce(p, y, estimator = "biased") equals mmce(p, y, bandwidth)^2; that is, mmce() is the square root of the biased SKCE estimator.

Let $e_i$ be the residual for observation $i$ and $k$ the kernel. In the confidence and binary cases the residual is the scalar $e_i = c_i - \rho_i$, where $\rho_i$ is the scalar confidence (the predicted event probability for binary inputs, or the top-label probability for the confidence reduction) and $c_i$ the corresponding binary target. The kernel is the scalar Laplacian kernel $k(\rho_i, \rho_j) = \exp(-|\rho_i - \rho_j| / h)$ with bandwidth $h$, and the kernel term is $h_{ij} = e_i e_j k(\rho_i, \rho_j)$.

In the canonical multiclass case the residual is the vector $e_i = \mathbf{1}_{y_i} - p_i$ in $\mathbb{R}^K$, where $\mathbf{1}_{y_i}$ is the one-hot encoding of the label. The matrix-valued kernel is $k(s, t) = \tilde k(s, t) I_K$ with $\tilde k$ the Laplacian kernel $\tilde k(s, t) = \exp(-\lVert s - t \rVert_2 / h)$ on the probability vectors, using the Euclidean norm on the simplex. The kernel term is then $h_{ij} = \tilde k(p_i, p_j)\, \langle e_i, e_j \rangle$.

Writing $H = [h_{ij}]$, the three estimators of Widmann et al. (2019, Table 1) are

$$\widehat{\mathrm{SKCE}}_b = \frac{1}{n^2} \sum_{i,j} h_{ij},$$

$$\widehat{\mathrm{SKCE}}_{uq} = \binom{n}{2}^{-1} \sum_{i < j} h_{ij},$$

$$\widehat{\mathrm{SKCE}}_{ul} = \lfloor n/2 \rfloor^{-1} \sum_{i = 1}^{\lfloor n/2 \rfloor} h_{2i-1,\,2i}.$$

The biased estimator "biased" keeps the diagonal terms $i = j$ and is the $V$-statistic underlying mmce(); the diagonal contributes $n^{-2}\sum_i \lVert e_i \rVert^2 > 0$, so it is biased upward and need not vanish even under perfect finite-sample calibration. The unbiased estimator "uq" drops the diagonal and divides by $n(n-1)$; it costs $O(n^2)$. The unbiased estimator "ul" sums over the disjoint pairs $(1, 2), (3, 4), \ldots$ and costs $O(n)$. The unbiased estimators can be negative in finite samples because they are unbiased estimates of a nonnegative quantity; only the biased estimator is guaranteed nonnegative.

References

Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. Advances in Neural Information Processing Systems 32. arXiv:1910.11385.

Kumar, A., Sarawagi, S., & Jain, U. (2018). Trainable calibration measures for neural networks from kernel mean embeddings. Proceedings of the 35th International Conference on Machine Learning.

Examples

set.seed(31)
p <- stats::runif(200)
y <- rbinom(200, 1, p)

# Unbiased U-statistic estimate of the squared kernel calibration error.
skce(p, y)
#> [1] 9.74756e-05

# The biased estimator equals mmce()^2.
all.equal(skce(p, y, estimator = "biased"), mmce(p, y)^2)
#> [1] TRUE

# Canonical (strong) multiclass calibration from a probability matrix.
set.seed(32)
prob <- matrix(stats::runif(150 * 3), ncol = 3)
prob <- prob / rowSums(prob)
labels <- max.col(prob)
skce(prob, labels, type = "canonical")
#> [1] 0.01539506