Question: for each finalist, how does the WIS change split across quantile levels, how large are the offsets next to the base forecast’s spread, and how much of the change comes from moving the median versus changing widths?

Setup:

  • Diseases: flu and covid, one section each.
  • Base forecaster: the clean windowed_seasonal replay, on the (round, location, horizon, level) rows the submitted ensemble also has, h0–h3 (E10’s replay runs).
  • Learning truth: exact (re-run at every round on the data published by then; E00), one week of extra delay.
  • Scores against finalized truth over 2024-25 and 2025-26, pooled; states only (US excluded).
  • Configs, E10’s finalists:
    • REF-op warm: flu prod; warm start from the 2023-24 burn-in season.
    • REF-op cold: covid prod; no burn-in season.
    • sqrt 0.018: constant learning rate 0.018 on the square root of admissions per 100k (E16), cold.
    • sqrt 0.018 warm: the same with the burn-in season and a warm start.

WIS reduction % = 100 × (base − calibrated) / base, so positive = calibrated WIS lower than base.

Three ways to cut the WIS reduction.

  1. By quantile level. WIS is the average pinball loss over the 23 quantile levels, so the total change splits exactly into one piece per level. A level’s piece is its change in pinball loss as a share of the base WIS; the pieces add up to the total.
  2. Offset size against the forecast’s own spread. For each forecast, the offset at a level is calibrated quantile − base quantile (in admissions, after every projection step). It is compared with the base forecast’s 90% interval width (q95 − q05) and 50% interval width (q75 − q25). A ratio of 0.1 means the quantile moved by a tenth of the interval width.
  3. Median shift vs width change. Each calibrated forecast is split into two parts and each part is scored on its own:
    • median shift only: every base quantile moves by the same amount, the calibrated median minus the base median. The base forecast’s shape and widths are kept.
    • width change only: the calibrated forecast is moved so that its median sits on the base median. The calibrated widths and shape are kept, the base centre is kept.
    The interaction is the full change minus the two parts. Both parts are floored at zero, as the calibrated forecast is.
source(here::here("scripts/calibration/calibration_ws_replay.R"))
pinball <- function(y, q, tau) ifelse(y >= q, tau * (y - q), (1 - tau) * (q - y))
FIN_COLS <- c(`REF-op warm` = "#000000", `REF-op cold` = "#D55E00", `sqrt 0.018` = "#0072B2", `sqrt 0.018 warm` = "#56B4E9")
fin_fc <- function(d) {
  f <- fin_runs(d, bases = "replay", workers = params$workers)
  purrr::imap(f$cals, \(cal, nm) cal$forecasts %>% mutate(config = sub("^replay, ", "", nm))) %>%
    bind_rows() %>%
    mutate(season = season_of(reference_date), config = factor(config, levels = names(FIN_COLS))) %>%
    filter(season %in% LIVE, !is_burn_in, !is.na(truth), !is.na(value_base), location != "US")
}
FC <- purrr::map(c(flu = "flu", covid = "covid"), fin_fc)
# 1. Each level's change in pinball loss as a share of the base WIS.
by_level <- function(fc) {
  fc %>%
    group_by(config, horizon, level) %>%
    summarize(base = sum(pinball(truth, value_base, level)), cal = sum(pinball(truth, value_cal, level)), .groups = "drop") %>%
    group_by(config, horizon) %>%
    mutate(contrib = 100 * (base - cal) / sum(base), total = sum(contrib)) %>%
    ungroup()
}
level_plot <- function(bl) {
  d <- bl %>% mutate(panel = sprintf("h%d, total %+.1f%%", horizon, total))
  ggplot(d, aes(factor(level), contrib, fill = config)) +
    geom_hline(yintercept = 0, colour = "grey50") +
    geom_col(width = 0.75) +
    facet_wrap(~ paste(config, panel, sep = ": "), ncol = 4) +
    scale_fill_manual(values = FIN_COLS, guide = "none") +
    scale_x_discrete(labels = function(x) ifelse(x %in% c("0.01", "0.05", "0.25", "0.5", "0.75", "0.95", "0.99"), x, "")) +
    labs(x = "quantile level", y = "share of WIS reduction, % of base WIS\n(positive = calibrated WIS lower than base)") +
    theme(axis.text.x = element_text(angle = 90, vjust = 0.5, size = 7))
}
group_tbl <- function(bl) {
  grouped <- bl %>%
    mutate(
      band = case_when(level <= 0.05 | level >= 0.95 ~ "tails", level <= 0.3 | level >= 0.7 ~ "shoulders", TRUE ~ "centre"),
      side = case_when(level < 0.5 ~ "lower", level > 0.5 ~ "upper", TRUE ~ "median")
    )
  bind_rows(
    grouped %>% group_by(config, horizon, part = band) %>% summarize(v = sum(contrib), .groups = "drop"),
    grouped %>% group_by(config, horizon, part = paste(side, "side")) %>% summarize(v = sum(contrib), .groups = "drop"),
    grouped %>% group_by(config, horizon) %>% summarize(v = sum(contrib), .groups = "drop") %>% mutate(part = "total")
  ) %>%
    mutate(part = factor(part, levels = c("tails", "shoulders", "centre", "lower side", "median side", "upper side", "total"),
      labels = c("tails", "shoulders", "centre", "below the median", "the median level", "above the median", "total"))) %>%
    transmute(config, part, horizon = paste0("h", horizon), v = sprintf("%+.2f", v)) %>%
    tidyr::pivot_wider(names_from = horizon, values_from = v) %>%
    arrange(config, part)
}
# 2. Signed offset ÷ the base 90% interval width, per forecast and level.
offset_plot <- function(fc) {
  ref <- levels(droplevels(fc$config))[1]
  widths <- fc %>%
    filter(config == ref, level %in% c(0.05, 0.95)) %>%
    select(location, horizon, reference_date, level, value_base) %>%
    tidyr::pivot_wider(names_from = level, values_from = value_base, names_prefix = "q") %>%
    transmute(location, horizon, reference_date, w90 = q0.95 - q0.05)
  off_q <- fc %>%
    inner_join(widths, by = c("location", "horizon", "reference_date")) %>%
    filter(w90 > 0) %>%
    mutate(r90 = (value_cal - value_base) / w90) %>%
    group_by(config, horizon, level) %>%
    summarize(med = median(r90), lo = quantile(r90, 0.1), hi = quantile(r90, 0.9), .groups = "drop")
  ggplot(off_q, aes(level, med, colour = config, fill = config)) +
    geom_hline(yintercept = 0, colour = "grey50") +
    geom_ribbon(aes(ymin = lo, ymax = hi), alpha = 0.12, colour = NA) +
    geom_line() +
    geom_point(size = 0.8) +
    facet_wrap(~ paste0("h", horizon), nrow = 2) +
    scale_colour_manual(values = FIN_COLS, name = NULL, aesthetics = c("colour", "fill")) +
    labs(x = "quantile level", y = "offset / base (q95 − q05)") +
    theme(legend.position = "bottom")
}
# 3. Score the median shift and the width change on their own.
split_tbl <- function(fc) {
  med <- fc %>%
    filter(level == 0.5) %>%
    select(config, location, horizon, reference_date, med_base = value_base, med_cal = value_cal)
  scores <- fc %>%
    left_join(med, by = c("config", "location", "horizon", "reference_date")) %>%
    mutate(median_only = pmax(value_base + (med_cal - med_base), 0), width_only = pmax(med_base + (value_cal - med_cal), 0)) %>%
    group_by(config, horizon) %>%
    summarize(
      base = sum(pinball(truth, value_base, level)),
      full = sum(pinball(truth, value_cal, level)),
      `median shift only` = sum(pinball(truth, median_only, level)),
      `width change only` = sum(pinball(truth, width_only, level)),
      .groups = "drop"
    ) %>%
    tidyr::pivot_longer(c(full, `median shift only`, `width change only`), names_to = "variant", values_to = "wis") %>%
    mutate(pct = 100 * (base - wis) / base)
  bind_rows(
    scores %>% select(config, variant, horizon, pct),
    scores %>%
      group_by(config, horizon) %>%
      summarize(pct = pct[variant == "full"] - pct[variant == "median shift only"] - pct[variant == "width change only"], .groups = "drop") %>%
      mutate(variant = "interaction (full − the two parts)")
  ) %>%
    mutate(variant = factor(variant, levels = c("full", "median shift only", "width change only", "interaction (full − the two parts)"))) %>%
    transmute(config, variant, horizon = paste0("h", horizon), pct = sprintf("%+.2f", pct)) %>%
    tidyr::pivot_wider(names_from = horizon, values_from = pct) %>%
    arrange(config, variant)
}
for (d in names(FC)) {
  fc <- FC[[d]]
  bl <- by_level(fc)
  cat("\n\n# ", d, "\n\n", sep = "")
  cat("\n\n## WIS reduction by quantile level\n\n")
  cat("Each bar is one level's share of the total WIS reduction %; the bars in a panel add up to the panel's total, printed in the strip.\n\n")
  print(level_plot(bl))
  cat("\n\nThe same pieces summed over groups of levels: **tails** are the six levels at or beyond 0.05/0.95, **shoulders** the ten levels 0.1–0.3 and 0.7–0.9, **centre** the seven levels 0.35–0.65. Below and above the median are split, to show whether the gain comes from moving the lower or the upper side.\n\n")
  print(knitr::kable(group_tbl(bl), caption = paste0(d, ": WIS reduction % by group of levels (positive = calibrated WIS lower than base)")))
  cat("\n\n## Offset size against the forecast's spread\n\n")
  cat("Offsets are signed: positive means the calibrated quantile is above the base one. The line is the median over forecasts (location × round) and the band the middle 80%, offset ÷ base 90% interval width. Forecasts with a zero-width base interval are left out.\n\n")
  print(offset_plot(fc))
  cat("\n\n## Median shift vs width change\n\n")
  print(knitr::kable(split_tbl(fc), caption = paste0(d, ": WIS reduction % (positive = calibrated WIS lower than base)")))
}

flu

WIS reduction by quantile level

Each bar is one level’s share of the total WIS reduction %; the bars in a panel add up to the panel’s total, printed in the strip.

The same pieces summed over groups of levels: tails are the six levels at or beyond 0.05/0.95, shoulders the ten levels 0.1–0.3 and 0.7–0.9, centre the seven levels 0.35–0.65. Below and above the median are split, to show whether the gain comes from moving the lower or the upper side.

flu: WIS reduction % by group of levels (positive = calibrated WIS lower than base)
config part h0 h1 h2 h3
REF-op warm tails +0.02 -0.14 -0.27 -0.18
REF-op warm shoulders +1.27 +0.60 +0.45 +0.53
REF-op warm centre +1.53 +0.62 +0.67 +0.83
REF-op warm below the median +2.43 +1.52 +1.30 +1.20
REF-op warm the median level +0.21 +0.07 +0.09 +0.12
REF-op warm above the median +0.18 -0.52 -0.55 -0.14
REF-op warm total +2.82 +1.08 +0.85 +1.18
REF-op cold tails +0.10 +0.06 +0.05 +0.03
REF-op cold shoulders +0.92 +0.36 +0.02 -0.25
REF-op cold centre +1.00 +0.16 +0.02 +0.04
REF-op cold below the median +1.36 +0.72 +0.59 +0.58
REF-op cold the median level +0.13 +0.01 -0.01 +0.00
REF-op cold above the median +0.54 -0.14 -0.48 -0.76
REF-op cold total +2.03 +0.59 +0.09 -0.17
sqrt 0.018 tails +0.08 +0.05 +0.04 +0.04
sqrt 0.018 shoulders +0.96 +0.38 +0.20 +0.17
sqrt 0.018 centre +1.16 +0.34 +0.36 +0.39
sqrt 0.018 below the median +1.59 +0.75 +0.57 +0.49
sqrt 0.018 the median level +0.17 +0.05 +0.05 +0.07
sqrt 0.018 above the median +0.45 -0.03 -0.02 +0.04
sqrt 0.018 total +2.20 +0.77 +0.61 +0.61
sqrt 0.018 warm tails -0.00 -0.28 -0.62 -0.37
sqrt 0.018 warm shoulders +1.62 +0.86 +0.87 +1.43
sqrt 0.018 warm centre +1.74 +0.98 +1.24 +1.62
sqrt 0.018 warm below the median +3.21 +2.15 +1.89 +1.77
sqrt 0.018 warm the median level +0.25 +0.12 +0.18 +0.23
sqrt 0.018 warm above the median -0.11 -0.72 -0.58 +0.67
sqrt 0.018 warm total +3.36 +1.55 +1.49 +2.68
NA tails -0.06 -0.34 -0.68 -0.43
NA shoulders +1.50 +0.83 +0.95 +1.60
NA centre +1.82 +1.22 +1.52 +1.91
NA below the median +3.15 +2.23 +2.01 +1.89
NA the median level +0.27 +0.16 +0.23 +0.28
NA above the median -0.16 -0.68 -0.44 +0.91
NA total +3.25 +1.71 +1.79 +3.08

Offset size against the forecast’s spread

Offsets are signed: positive means the calibrated quantile is above the base one. The line is the median over forecasts (location × round) and the band the middle 80%, offset ÷ base 90% interval width. Forecasts with a zero-width base interval are left out.

Median shift vs width change

flu: WIS reduction % (positive = calibrated WIS lower than base)
config variant h0 h1 h2 h3
REF-op warm full +2.82 +1.08 +0.85 +1.18
REF-op warm median shift only +3.56 +1.89 +2.05 +2.16
REF-op warm width change only -1.27 -1.49 -2.13 -2.27
REF-op warm interaction (full − the two parts) +0.53 +0.67 +0.93 +1.29
REF-op cold full +2.03 +0.59 +0.09 -0.17
REF-op cold median shift only +2.29 +0.80 +0.47 +0.39
REF-op cold width change only -0.59 -0.65 -0.86 -1.14
REF-op cold interaction (full − the two parts) +0.33 +0.44 +0.49 +0.57
sqrt 0.018 full +2.20 +0.77 +0.61 +0.61
sqrt 0.018 median shift only +2.52 +0.99 +0.85 +0.91
sqrt 0.018 width change only -0.68 -0.47 -0.52 -0.49
sqrt 0.018 interaction (full − the two parts) +0.36 +0.25 +0.28 +0.19
sqrt 0.018 warm full +3.36 +1.55 +1.49 +2.68
sqrt 0.018 warm median shift only +4.17 +2.48 +3.26 +3.99
sqrt 0.018 warm width change only -1.37 -1.32 -2.85 -2.70
sqrt 0.018 warm interaction (full − the two parts) +0.56 +0.40 +1.09 +1.39
NA full +3.25 +1.71 +1.79 +3.08
NA median shift only +4.20 +2.82 +3.79 +4.72
NA width change only -1.57 -1.45 -3.01 -2.83
NA interaction (full − the two parts) +0.63 +0.34 +1.01 +1.20

covid

WIS reduction by quantile level

Each bar is one level’s share of the total WIS reduction %; the bars in a panel add up to the panel’s total, printed in the strip.

The same pieces summed over groups of levels: tails are the six levels at or beyond 0.05/0.95, shoulders the ten levels 0.1–0.3 and 0.7–0.9, centre the seven levels 0.35–0.65. Below and above the median are split, to show whether the gain comes from moving the lower or the upper side.

covid: WIS reduction % by group of levels (positive = calibrated WIS lower than base)
config part h0 h1 h2 h3
REF-op warm tails +0.07 +0.38 +0.83 +1.57
REF-op warm shoulders -0.01 -0.71 -0.20 +1.06
REF-op warm centre -0.20 -1.44 -1.03 +0.18
REF-op warm below the median +0.06 -1.16 -0.90 -0.37
REF-op warm the median level -0.01 -0.23 -0.17 +0.03
REF-op warm above the median -0.19 -0.37 +0.67 +3.15
REF-op warm total -0.14 -1.76 -0.40 +2.80
REF-op cold tails +0.09 +0.06 +0.07 +0.08
REF-op cold shoulders +0.55 -0.31 -0.57 -0.48
REF-op cold centre +0.62 -0.81 -1.16 -1.19
REF-op cold below the median +0.58 -0.53 -0.88 -0.96
REF-op cold the median level +0.12 -0.14 -0.19 -0.19
REF-op cold above the median +0.57 -0.39 -0.59 -0.44
REF-op cold total +1.27 -1.06 -1.66 -1.59
sqrt 0.018 tails +0.17 +0.11 +0.10 +0.10
sqrt 0.018 shoulders +0.29 -0.64 -0.41 +0.34
sqrt 0.018 centre +0.19 -1.24 -1.15 -0.37
sqrt 0.018 below the median +0.46 -0.70 -0.61 -0.06
sqrt 0.018 the median level +0.03 -0.20 -0.18 -0.08
sqrt 0.018 above the median +0.17 -0.88 -0.67 +0.21
sqrt 0.018 total +0.66 -1.77 -1.47 +0.07
sqrt 0.018 warm tails +0.20 +0.85 +1.39 +1.47
sqrt 0.018 warm shoulders -0.41 -1.86 -1.54 -0.34
sqrt 0.018 warm centre -0.66 -2.36 -1.65 +0.18
sqrt 0.018 warm below the median -0.44 -2.33 -2.23 -1.79
sqrt 0.018 warm the median level -0.10 -0.34 -0.20 +0.11
sqrt 0.018 warm above the median -0.34 -0.69 +0.62 +3.00
sqrt 0.018 warm total -0.87 -3.36 -1.80 +1.31
NA tails +0.06 +0.77 +1.30 +1.08
NA shoulders -1.01 -2.40 -2.21 -1.24
NA centre -1.23 -3.02 -2.33 -0.61
NA below the median -1.11 -3.18 -3.10 -2.80
NA the median level -0.20 -0.44 -0.27 +0.00
NA above the median -0.87 -1.03 +0.13 +2.03
NA total -2.18 -4.65 -3.24 -0.77

Offset size against the forecast’s spread

Offsets are signed: positive means the calibrated quantile is above the base one. The line is the median over forecasts (location × round) and the band the middle 80%, offset ÷ base 90% interval width. Forecasts with a zero-width base interval are left out.

Median shift vs width change

covid: WIS reduction % (positive = calibrated WIS lower than base)
config variant h0 h1 h2 h3
REF-op warm full -0.14 -1.76 -0.40 +2.80
REF-op warm median shift only -0.43 -2.42 -1.71 +1.22
REF-op warm width change only -0.31 +0.38 +1.19 +1.86
REF-op warm interaction (full − the two parts) +0.61 +0.29 +0.12 -0.29
REF-op cold full +1.27 -1.06 -1.66 -1.59
REF-op cold median shift only +0.89 -1.93 -2.64 -2.35
REF-op cold width change only -0.41 -0.05 -0.33 -0.84
REF-op cold interaction (full − the two parts) +0.79 +0.92 +1.31 +1.60
sqrt 0.018 full +0.66 -1.77 -1.47 +0.07
sqrt 0.018 median shift only -0.72 -3.09 -2.31 -0.45
sqrt 0.018 width change only -0.54 +0.10 +0.22 -0.10
sqrt 0.018 interaction (full − the two parts) +1.92 +1.22 +0.62 +0.62
sqrt 0.018 warm full -0.87 -3.36 -1.80 +1.31
sqrt 0.018 warm median shift only -2.02 -4.40 -2.18 +1.67
sqrt 0.018 warm width change only -0.73 +0.47 +2.67 +4.92
sqrt 0.018 warm interaction (full − the two parts) +1.87 +0.56 -2.29 -5.28
NA full -2.18 -4.65 -3.24 -0.77
NA median shift only -2.13 -4.17 -2.32 +1.20
NA width change only -0.15 +1.15 +3.16 +5.31
NA interaction (full − the two parts) +0.10 -1.63 -4.09 -7.28