Question: do the warm start and the leak change WIS and coverage in the same way on the adaptive learning rate and on the constant one, and on both diseases? E04 and E05 asked this on the flu ensemble with the adaptive rate only; E18 found that the leak’s effect depends on how large the adaptive step gets late in the season, so the two anchors may answer differently.

Setup:

  • Diseases: flu and covid. Base forecaster: the clean windowed_seasonal replay (53 locations, the hub rounds). Horizons h0–h3.
  • Learning truth: exact (re-run at every round on the data published by then; E00), one week of extra delay.
  • Warm configs run with the 2023-24 season present (the HHS-stitched backfill from E13) and start the offset from that season’s error quantiles; cold configs start at zero with no burn-in rounds, so they play the base through the first live rounds.
  • Scores over the two live seasons, each on its own rounds; states only (US excluded).

The grid is 2 × 2 × 2 (E18 step 2 shows the slow term does nothing, so it is left out), plus prod’s two configs:

factor levels
anchor adaptive 0.03: sqrt scale, eta = 0.03 × the 90th percentile of the last 20 rounds’ absolute errors, floor 0.001 (E01’s elbow). constant 0.018: sqrt of admissions per 100k, eta = 0.018 (E16’s elbow, picked on flu)
start cold: offsets start at zero. warm: slow_init = "burn_in_quantile" from the 2023-24 season
leak no leak. leak 0.1: the offset shrinks by 10% each round
references REF-op warm (flu prod: adaptive, warm, leak, plus a slow term); REF-op cold (covid prod)

WIS reduction % = 100 × (base − calibrated) / base, so positive = calibrated WIS lower than base. L1 coverage bias: lower is better. “Month-avg” coverage bias is computed within each calendar month and then averaged.

Every WIS cell carries a 90% interval from a round bootstrap: the season’s rounds are resampled with replacement 1000 times and the WIS reduction recomputed on each draw (cal_wis_boot()). The same draws score every config, so a gap between two configs has a paired interval; the “Paired gaps” tables show it for the cells the ledger compares. Outcomes within a round move together, so the round is the resampling unit; this ignores dependence between neighbouring rounds, so the intervals are if anything too narrow.

source(here::here("scripts/calibration/calibration_ws_replay.R"))
DISEASES <- c(flu = "flu", covid = "covid")
RUNS <- purrr::map(DISEASES, function(d) {
  wi <- ws_inputs(d)
  fc_burn_in <- bind_rows(ws_burn_in_forecasts(d), wi$fc)
  specs <- c(ws_grid_configs(wi$rate_scales), ws_prod_specs(wi$rate_scales))
  list(wi = wi, specs = specs, cals = purrr::map(specs, ws_run_spec, wi = wi, fc_burn_in = fc_burn_in, workers = params$workers))
})
grid_names <- names(ws_grid_configs(RUNS$flu$wi$rate_scales))
split_name <- function(d) {
  d %>%
    mutate(config = as.character(config)) %>%
    tidyr::separate_wider_delim(config, " | ", names = c("anchor", "start", "leak"), cols_remove = FALSE, too_few = "align_start")
}
ANCHOR_COLS <- c(`adaptive 0.03` = "#E69F00", `constant 0.018` = "#0072B2")
h_levels <- paste0("h", 0:3)
M <- purrr::map(RUNS, function(r) {
  m <- ws_wis_by_season(r$cals)
  mv <- cal_month_view(r$cals, seasons = LIVE)
  cov <- mv$cov %>% transmute(config, season, h = factor(paste0("h", horizon), levels = h_levels), pooled = pooled_cal, mavg = mavg_cal)
  bt <- cal_wis_boot(r$cals, seasons = LIVE)
  list(m = m, mv = mv, cov = cov, bt = bt)
})

WIS by season and horizon

for (d in names(M)) {
  cat("\n\n", cal_h(2), " ", d, "\n\n", sep = "")
  cal_wis_boot_table(M[[d]]$bt) %>%
    knitr::kable(caption = paste0(d, ": ", cal_wis_boot_caption(M[[d]]$bt))) %>%
    print()
}

flu

flu: WIS reduction % (positive = calibrated WIS lower than base), states only, with a 90% interval from 1000 round bootstrap draws (rounds resampled within each season; both pools the seasons’ draws).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
adaptive 0.03 | cold | no leak +1.8 [-0.2, +3.5] +0.0 [-2.7, +2.0] -1.4 [-5.1, +0.9] -2.6 [-6.9, -0.1] +2.2 [-3.2, +6.5] -0.6 [-6.0, +3.4] -0.7 [-6.9, +3.2] -0.2 [-7.6, +4.0] +1.9 [-0.6, +3.9] -0.2 [-2.8, +1.8] -1.1 [-4.2, +1.1] -1.7 [-5.2, +0.7]
adaptive 0.03 | cold | leak 0.1 +2.0 [+0.5, +3.3] +0.9 [-0.7, +2.4] +0.1 [-2.0, +1.5] -0.5 [-2.9, +0.9] +1.5 [-0.6, +3.3] +0.1 [-2.1, +1.8] +0.4 [-1.6, +2.1] +0.9 [-0.8, +2.3] +1.8 [+0.6, +2.9] +0.6 [-0.7, +1.8] +0.2 [-1.2, +1.3] -0.0 [-1.4, +1.1]
adaptive 0.03 | warm | no leak +2.8 [+1.0, +4.4] +1.2 [-1.0, +3.1] +0.4 [-3.1, +2.8] -0.0 [-4.6, +2.7] +2.4 [-2.7, +6.7] -0.7 [-5.3, +3.0] -0.8 [-6.1, +2.7] -0.2 [-6.2, +3.3] +2.7 [+0.3, +4.6] +0.4 [-1.8, +2.3] -0.1 [-2.9, +2.0] -0.1 [-3.5, +2.2]
adaptive 0.03 | warm | leak 0.1 +3.1 [+1.6, +4.4] +2.0 [+0.5, +3.5] +1.8 [-0.3, +3.4] +1.9 [-1.0, +3.8] +2.1 [-0.7, +5.2] +0.1 [-3.0, +2.8] +0.1 [-3.1, +2.9] +0.4 [-3.0, +3.2] +2.7 [+1.3, +4.0] +1.2 [-0.4, +2.6] +1.1 [-0.8, +2.6] +1.3 [-0.9, +3.0]
constant 0.018 | cold | no leak +1.7 [+0.6, +2.7] +0.6 [-0.3, +1.4] +0.3 [-0.6, +0.9] +0.1 [-0.7, +0.6] +3.0 [-0.9, +6.6] +1.0 [-2.1, +3.5] +1.1 [-1.9, +3.3] +1.5 [-1.4, +3.6] +2.2 [+0.5, +3.7] +0.8 [-0.6, +1.9] +0.6 [-0.7, +1.6] +0.6 [-0.5, +1.5]
constant 0.018 | cold | leak 0.1 +1.5 [+0.6, +2.2] +0.6 [+0.0, +1.2] +0.4 [-0.2, +0.8] +0.2 [-0.3, +0.5] +1.3 [-0.1, +2.7] +0.4 [-0.7, +1.3] +0.4 [-0.3, +1.2] +0.7 [+0.0, +1.3] +1.4 [+0.6, +2.2] +0.5 [-0.0, +1.0] +0.4 [-0.1, +0.8] +0.4 [-0.0, +0.7]
constant 0.018 | warm | no leak +3.4 [+1.9, +4.8] +2.1 [+0.7, +3.7] +2.2 [+0.2, +4.0] +3.3 [+0.7, +5.2] +3.4 [-1.0, +7.7] +0.9 [-2.8, +4.4] +0.8 [-3.6, +4.5] +1.2 [-4.6, +5.2] +3.4 [+1.5, +5.3] +1.6 [-0.2, +3.3] +1.6 [-0.5, +3.5] +2.5 [+0.0, +4.5]
constant 0.018 | warm | leak 0.1 +3.6 [+2.3, +4.9] +2.5 [+1.2, +3.9] +2.7 [+1.1, +4.4] +4.0 [+1.9, +5.6] +2.9 [+0.2, +6.3] +0.8 [-1.7, +3.8] +0.8 [-2.3, +4.1] +1.2 [-3.0, +4.5] +3.3 [+2.0, +4.8] +1.8 [+0.4, +3.3] +1.9 [+0.3, +3.6] +2.9 [+0.9, +4.6]
REF-op warm +3.2 [+1.6, +4.6] +1.9 [+0.2, +3.6] +1.6 [-0.8, +3.4] +1.6 [-1.6, +3.7] +2.2 [-0.9, +5.6] -0.1 [-3.6, +2.9] -0.0 [-3.7, +3.1] +0.2 [-3.6, +3.2] +2.8 [+1.3, +4.3] +1.1 [-0.7, +2.7] +0.9 [-1.2, +2.6] +1.1 [-1.4, +2.9]
REF-op cold +2.2 [+0.5, +3.6] +0.9 [-1.1, +2.5] -0.0 [-2.5, +1.6] -0.8 [-3.7, +0.9] +1.8 [-0.8, +4.1] +0.1 [-2.7, +2.4] +0.3 [-2.5, +2.5] +0.9 [-1.6, +2.8] +2.0 [+0.6, +3.3] +0.6 [-1.0, +2.0] +0.1 [-1.7, +1.4] -0.2 [-2.0, +1.1]

covid

covid: WIS reduction % (positive = calibrated WIS lower than base), states only, with a 90% interval from 1000 round bootstrap draws (rounds resampled within each season; both pools the seasons’ draws).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
adaptive 0.03 | cold | no leak +2.3 [+1.1, +3.5] -0.7 [-1.7, +0.2] -2.1 [-3.8, -0.5] -2.8 [-5.8, -0.3] -0.8 [-4.8, +3.5] -4.5 [-6.9, -2.1] -4.0 [-5.9, -2.1] -1.2 [-5.3, +2.6] +0.7 [-1.4, +2.9] -2.7 [-4.1, -1.3] -3.1 [-4.4, -1.8] -1.9 [-4.5, +0.3]
adaptive 0.03 | cold | leak 0.1 +1.7 [+0.8, +2.6] +0.4 [-0.5, +1.2] +0.3 [-0.7, +1.4] +0.4 [-1.1, +1.9] +0.5 [-1.2, +2.3] -2.0 [-3.7, -0.2] -2.8 [-4.3, -1.2] -3.1 [-4.8, -1.1] +1.1 [+0.1, +2.1] -0.8 [-1.8, +0.2] -1.4 [-2.3, -0.3] -1.4 [-2.7, -0.1]
adaptive 0.03 | warm | no leak +0.5 [-0.7, +1.8] -2.5 [-5.5, +0.8] -2.7 [-8.4, +2.8] -1.0 [-8.9, +6.5] -0.7 [-4.3, +3.1] -3.9 [-5.7, -1.9] -2.8 [-6.2, +0.3] +1.0 [-5.2, +6.7] -0.1 [-2.0, +1.9] -3.2 [-5.0, -1.3] -2.7 [-6.1, +0.3] +0.1 [-4.9, +4.8]
adaptive 0.03 | warm | leak 0.1 -0.9 [-1.8, +0.0] -2.5 [-5.6, +0.8] -1.1 [-7.0, +4.6] +1.7 [-6.3, +9.2] -0.4 [-1.5, +0.6] -1.3 [-4.7, +1.6] +0.8 [-5.1, +5.9] +4.9 [-3.1, +11.8] -0.6 [-1.3, +0.0] -1.9 [-4.1, +0.3] -0.1 [-4.3, +3.6] +3.4 [-2.3, +8.4]
constant 0.018 | cold | no leak +2.2 [+1.2, +3.3] -0.2 [-0.7, +0.4] -0.8 [-1.7, -0.0] -0.8 [-2.2, +0.3] -0.9 [-5.1, +3.7] -3.3 [-5.5, -1.0] -2.1 [-3.1, -0.8] +0.9 [-1.3, +3.0] +0.7 [-1.6, +3.0] -1.8 [-3.0, -0.6] -1.5 [-2.2, -0.7] +0.1 [-1.3, +1.3]
constant 0.018 | cold | leak 0.1 +1.5 [+0.8, +2.2] +0.3 [-0.2, +0.8] +0.2 [-0.3, +0.7] +0.3 [-0.4, +1.0] +0.6 [-1.3, +2.6] -1.4 [-2.8, +0.1] -1.7 [-2.5, -0.6] -1.2 [-2.1, -0.2] +1.1 [+0.0, +2.1] -0.5 [-1.3, +0.3] -0.8 [-1.3, -0.2] -0.5 [-1.1, +0.1]
constant 0.018 | warm | no leak -1.3 [-2.8, +0.5] -5.1 [-10.1, +0.3] -4.6 [-13.6, +3.9] -2.8 [-15.5, +8.9] -0.5 [-3.9, +3.3] -1.7 [-3.6, +0.1] +0.8 [-6.1, +7.0] +5.0 [-7.0, +15.6] -0.9 [-2.8, +1.3] -3.4 [-6.1, -0.7] -1.8 [-7.6, +3.3] +1.3 [-7.4, +9.1]
constant 0.018 | warm | leak 0.1 -3.2 [-4.6, -1.8] -6.8 [-12.0, -1.5] -6.0 [-15.4, +2.7] -4.0 [-17.1, +7.9] -1.1 [-2.8, +0.2] -2.6 [-9.1, +3.1] -0.7 [-11.7, +8.9] +2.2 [-13.1, +15.4] -2.2 [-3.3, -1.1] -4.7 [-8.8, -0.8] -3.2 [-10.8, +3.3] -0.8 [-11.2, +8.3]
REF-op warm -0.5 [-1.4, +0.5] -2.3 [-5.4, +0.9] -1.2 [-7.0, +4.6] +1.4 [-6.6, +8.8] +0.2 [-1.0, +1.5] -1.2 [-3.8, +1.0] +0.3 [-4.7, +4.7] +4.1 [-3.2, +10.4] -0.1 [-0.9, +0.7] -1.8 [-3.8, +0.2] -0.4 [-4.3, +3.0] +2.8 [-2.6, +7.7]
REF-op cold +1.9 [+0.9, +2.9] +0.3 [-0.6, +1.2] +0.0 [-1.1, +1.2] +0.1 [-1.6, +1.9] +0.6 [-1.6, +2.9] -2.4 [-4.4, -0.3] -3.2 [-4.8, -1.4] -3.1 [-5.0, -0.8] +1.3 [+0.0, +2.5] -1.1 [-2.2, +0.1] -1.7 [-2.7, -0.5] -1.6 [-3.0, -0.1]

Paired gaps

The gap in WIS reduction % between two cells, with its paired bootstrap interval (same draws on both sides). An interval that excludes zero in a season means the ranking of the pair survived resampling that season’s rounds.

PAIRS <- list(
  flu = list(
    c("constant 0.018 | warm | leak 0.1", "constant 0.018 | warm | no leak"),
    c("constant 0.018 | warm | leak 0.1", "REF-op warm"),
    c("constant 0.018 | warm | leak 0.1", "adaptive 0.03 | warm | leak 0.1"),
    c("constant 0.018 | warm | leak 0.1", "constant 0.018 | cold | leak 0.1")
  ),
  covid = list(
    c("constant 0.018 | cold | leak 0.1", "REF-op cold"),
    c("constant 0.018 | cold | leak 0.1", "constant 0.018 | cold | no leak"),
    c("constant 0.018 | cold | leak 0.1", "constant 0.018 | warm | leak 0.1")
  )
)
for (d in names(M)) {
  cat("\n\n", cal_h(2), " ", d, "\n\n", sep = "")
  purrr::map(PAIRS[[d]], function(pr) {
    cal_wis_boot_diff(M[[d]]$bt, pr[1], pr[2]) %>% mutate(pair = paste(pr[1], "−", pr[2]), .before = 1)
  }) %>%
    bind_rows() %>%
    transmute(pair, col = paste(season, h), cell = sprintf("%+.1f [%+.1f, %+.1f]", diff, lo, hi)) %>%
    tidyr::pivot_wider(names_from = col, values_from = cell) %>%
    knitr::kable(caption = paste0(d, ": gap in WIS reduction % (first − second) with its 90% paired bootstrap interval")) %>%
    print()
}

flu

flu: gap in WIS reduction % (first − second) with its 90% paired bootstrap interval
pair 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
constant 0.018 | warm | leak 0.1 − constant 0.018 | warm | no leak +0.2 [-0.2, +0.7] +0.4 [+0.1, +0.9] +0.5 [+0.2, +1.1] +0.7 [+0.3, +1.4] -0.5 [-2.2, +1.9] -0.1 [-1.3, +1.8] +0.1 [-1.0, +1.9] -0.0 [-1.2, +1.9] -0.1 [-0.9, +0.8] +0.2 [-0.4, +0.9] +0.3 [-0.2, +1.0] +0.4 [-0.2, +1.2]
constant 0.018 | warm | leak 0.1 − REF-op warm +0.4 [-0.2, +1.3] +0.5 [-0.3, +1.9] +1.1 [+0.0, +2.7] +2.3 [+1.3, +4.0] +0.7 [-0.3, +1.9] +0.9 [-0.2, +2.8] +0.9 [+0.0, +2.4] +1.0 [+0.2, +1.8] +0.5 [-0.0, +1.2] +0.7 [-0.0, +1.7] +1.0 [+0.2, +2.1] +1.8 [+1.2, +2.7]
constant 0.018 | warm | leak 0.1 − adaptive 0.03 | warm | leak 0.1 +0.5 [+0.0, +1.2] +0.5 [-0.2, +1.6] +0.9 [+0.1, +2.3] +2.1 [+1.3, +3.4] +0.8 [+0.1, +1.8] +0.8 [-0.1, +2.2] +0.7 [+0.1, +1.9] +0.8 [-0.2, +1.7] +0.6 [+0.2, +1.2] +0.6 [+0.0, +1.5] +0.8 [+0.2, +1.7] +1.6 [+1.0, +2.3]
constant 0.018 | warm | leak 0.1 − constant 0.018 | cold | leak 0.1 +2.1 [+1.3, +3.3] +1.8 [+0.8, +3.2] +2.3 [+1.1, +3.9] +3.8 [+2.1, +5.3] +1.6 [+0.1, +3.6] +0.5 [-1.2, +2.6] +0.4 [-2.0, +2.9] +0.5 [-3.1, +3.3] +1.9 [+1.1, +2.9] +1.3 [+0.3, +2.4] +1.5 [+0.3, +2.9] +2.5 [+0.9, +4.0]

covid

covid: gap in WIS reduction % (first − second) with its 90% paired bootstrap interval
pair 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
constant 0.018 | cold | leak 0.1 − REF-op cold -0.4 [-0.6, -0.1] +0.0 [-0.4, +0.5] +0.1 [-0.6, +0.8] +0.2 [-0.9, +1.2] -0.0 [-0.5, +0.3] +1.0 [+0.3, +1.7] +1.6 [+0.8, +2.3] +1.9 [+0.6, +3.0] -0.2 [-0.5, +0.0] +0.5 [+0.1, +0.9] +0.9 [+0.3, +1.4] +1.1 [+0.2, +1.9]
constant 0.018 | cold | leak 0.1 − constant 0.018 | cold | no leak -0.6 [-1.1, -0.3] +0.5 [+0.2, +0.8] +1.0 [+0.5, +1.6] +1.1 [+0.5, +1.9] +1.4 [-1.4, +4.0] +2.0 [+0.7, +3.1] +0.4 [-0.4, +1.2] -2.1 [-3.8, -0.3] +0.4 [-1.1, +1.8] +1.2 [+0.6, +1.9] +0.7 [+0.2, +1.2] -0.6 [-1.5, +0.4]
constant 0.018 | cold | leak 0.1 − constant 0.018 | warm | leak 0.1 +4.8 [+3.0, +6.4] +7.1 [+1.6, +12.4] +6.1 [-2.5, +15.4] +4.3 [-7.5, +17.2] +1.7 [-0.9, +4.5] +1.3 [-5.3, +8.6] -0.9 [-10.8, +10.6] -3.4 [-16.9, +11.9] +3.2 [+1.6, +4.9] +4.1 [-0.0, +8.7] +2.5 [-4.2, +10.1] +0.3 [-8.7, +10.4]

What each factor changes, per anchor

The change from adding the warm start (warm − cold) at each leak level, and from adding the leak (leak − no leak) at each start, per anchor. WIS: change in WIS reduction % (positive = the factor lowers WIS). Coverage: change in month-avg L1 coverage bias (negative = the factor improves coverage).

contrast <- function(d, factor, hi, lo, held) {
  d %>%
    select(anchor, start, leak, season, h, wis_pct, mavg) %>%
    tidyr::pivot_wider(names_from = all_of(factor), values_from = c(wis_pct, mavg)) %>%
    transmute(
      anchor, season, h,
      factor = paste0("+ ", hi, " (", .data[[held]], ")"),
      d_wis = .data[[paste0("wis_pct_", hi)]] - .data[[paste0("wis_pct_", lo)]],
      d_mavg = .data[[paste0("mavg_", hi)]] - .data[[paste0("mavg_", lo)]]
    )
}
DELTAS <- purrr::imap(M, function(x, d) {
  g <- x$m %>%
    filter(config %in% grid_names) %>%
    select(config, season, h, wis_pct) %>%
    left_join(x$cov, by = c("config", "season", "h")) %>%
    split_name()
  bind_rows(contrast(g, "start", "warm", "cold", "leak"), contrast(g, "leak", "leak 0.1", "no leak", "start")) %>%
    mutate(disease = d)
})
for (d in names(DELTAS)) {
  cat("\n\n", cal_h(2), " ", d, "\n\n", sep = "")
  DELTAS[[d]] %>%
    transmute(anchor, factor, col = paste(season, h), cell = sprintf("%+.1f / %+.3f", d_wis, d_mavg)) %>%
    tidyr::pivot_wider(names_from = col, values_from = cell) %>%
    arrange(anchor, factor) %>%
    knitr::kable(caption = paste0(d, ": change in WIS reduction % / month-avg L1 coverage bias from adding the factor, per anchor and season")) %>%
    print()
}

flu

flu: change in WIS reduction % / month-avg L1 coverage bias from adding the factor, per anchor and season
anchor factor 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
adaptive 0.03 + leak 0.1 (cold) +0.2 / +0.008 +0.9 / +0.002 +1.5 / +0.002 +2.0 / -0.000 -0.7 / +0.003 +0.7 / -0.001 +1.1 / -0.003 +1.0 / -0.009 -0.1 / +0.006 +0.8 / +0.000 +1.4 / -0.001 +1.7 / -0.005
adaptive 0.03 + leak 0.1 (warm) +0.3 / +0.000 +0.8 / -0.003 +1.4 / +0.000 +1.9 / -0.019 -0.3 / -0.001 +0.8 / -0.001 +1.0 / -0.007 +0.6 / -0.005 +0.1 / -0.001 +0.8 / -0.002 +1.2 / -0.004 +1.4 / -0.011
adaptive 0.03 + warm (leak 0.1) +1.1 / -0.006 +1.1 / -0.007 +1.7 / -0.002 +2.5 / -0.005 +0.6 / -0.003 -0.0 / +0.007 -0.3 / +0.003 -0.5 / +0.009 +0.9 / -0.005 +0.6 / +0.000 +0.9 / +0.000 +1.3 / +0.002
adaptive 0.03 + warm (no leak) +1.0 / +0.002 +1.2 / -0.002 +1.8 / -0.000 +2.6 / +0.013 +0.2 / +0.002 -0.1 / +0.007 -0.1 / +0.007 -0.0 / +0.005 +0.7 / +0.002 +0.7 / +0.003 +1.0 / +0.003 +1.6 / +0.009
constant 0.018 + leak 0.1 (cold) -0.2 / +0.010 +0.0 / +0.005 +0.1 / +0.006 +0.1 / +0.005 -1.7 / +0.011 -0.6 / +0.011 -0.6 / +0.009 -0.8 / +0.003 -0.8 / +0.010 -0.2 / +0.008 -0.2 / +0.007 -0.2 / +0.004
constant 0.018 + leak 0.1 (warm) +0.2 / +0.003 +0.4 / -0.002 +0.5 / -0.004 +0.7 / -0.011 -0.5 / -0.000 -0.1 / -0.007 +0.1 / -0.014 -0.0 / -0.012 -0.1 / +0.001 +0.2 / -0.005 +0.3 / -0.009 +0.4 / -0.012
constant 0.018 + warm (leak 0.1) +2.1 / -0.007 +1.8 / -0.012 +2.3 / -0.007 +3.8 / -0.002 +1.6 / -0.004 +0.5 / +0.006 +0.4 / +0.015 +0.5 / +0.022 +1.9 / -0.005 +1.3 / -0.002 +1.5 / +0.005 +2.5 / +0.011
constant 0.018 + warm (no leak) +1.7 / -0.001 +1.5 / -0.005 +1.9 / +0.004 +3.3 / +0.014 +0.4 / +0.008 -0.1 / +0.024 -0.3 / +0.037 -0.3 / +0.037 +1.2 / +0.004 +0.8 / +0.011 +1.0 / +0.021 +1.9 / +0.026

covid

covid: change in WIS reduction % / month-avg L1 coverage bias from adding the factor, per anchor and season
anchor factor 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
adaptive 0.03 + leak 0.1 (cold) -0.6 / +0.011 +1.1 / -0.006 +2.4 / -0.008 +3.1 / -0.007 +1.3 / +0.007 +2.6 / -0.003 +1.2 / -0.002 -1.9 / +0.003 +0.4 / +0.009 +1.8 / -0.005 +1.8 / -0.004 +0.5 / -0.001
adaptive 0.03 + leak 0.1 (warm) -1.4 / +0.019 -0.0 / +0.014 +1.6 / +0.010 +2.7 / +0.008 +0.4 / +0.023 +2.7 / +0.019 +3.6 / +0.016 +3.9 / +0.009 -0.5 / +0.021 +1.3 / +0.017 +2.6 / +0.013 +3.3 / +0.008
adaptive 0.03 + warm (leak 0.1) -2.6 / +0.035 -2.9 / +0.035 -1.4 / +0.023 +1.4 / +0.020 -0.9 / +0.017 +0.7 / +0.021 +3.7 / +0.017 +8.0 / +0.005 -1.7 / +0.025 -1.0 / +0.027 +1.2 / +0.020 +4.8 / +0.012
adaptive 0.03 + warm (no leak) -1.8 / +0.027 -1.8 / +0.014 -0.6 / +0.005 +1.8 / +0.006 +0.1 / +0.001 +0.6 / -0.001 +1.3 / -0.000 +2.2 / -0.001 -0.9 / +0.012 -0.5 / +0.005 +0.4 / +0.002 +2.0 / +0.002
constant 0.018 + leak 0.1 (cold) -0.6 / +0.009 +0.5 / -0.004 +1.0 / -0.004 +1.1 / -0.003 +1.4 / +0.006 +2.0 / -0.005 +0.4 / +0.000 -2.1 / +0.002 +0.4 / +0.007 +1.2 / -0.005 +0.7 / -0.002 -0.6 / -0.000
constant 0.018 + leak 0.1 (warm) -2.0 / +0.016 -1.7 / +0.011 -1.3 / +0.007 -1.2 / +0.005 -0.6 / +0.025 -0.9 / +0.027 -1.6 / +0.025 -2.9 / +0.028 -1.3 / +0.021 -1.3 / +0.020 -1.4 / +0.017 -2.1 / +0.018
constant 0.018 + warm (leak 0.1) -4.8 / +0.036 -7.1 / +0.043 -6.1 / +0.032 -4.3 / +0.035 -1.7 / +0.019 -1.3 / +0.029 +0.9 / +0.029 +3.4 / +0.034 -3.2 / +0.026 -4.1 / +0.035 -2.5 / +0.030 -0.3 / +0.034
constant 0.018 + warm (no leak) -3.4 / +0.029 -5.0 / +0.027 -3.8 / +0.021 -2.0 / +0.028 +0.4 / +0.001 +1.6 / -0.003 +2.9 / +0.003 +4.2 / +0.008 -1.5 / +0.013 -1.6 / +0.010 -0.3 / +0.011 +1.2 / +0.016 
ggplot(bind_rows(DELTAS) %>% filter(season != "both"), aes(h, d_wis, colour = anchor, group = anchor)) +
  geom_hline(yintercept = 0, colour = "grey60") +
  geom_line() +
  geom_point() +
  facet_grid(disease + season ~ factor) +
  scale_colour_manual(values = ANCHOR_COLS) +
  labs(x = NULL, y = "change in WIS reduction %", colour = NULL) +
  theme(legend.position = "bottom")

Anchor against anchor, state by state

Per state, pooled over h0–h3: WIS reduction % of the adaptive anchor against the constant one at the same start and leak (above the line, adaptive is better).

STATE <- purrr::map(RUNS, function(r) {
  cal_state_wis(r$cals[grid_names]) %>%
    group_by(config, season, location) %>%
    summarize(pct = 100 * (sum(wis_base) - sum(wis_cal)) / sum(wis_base), .groups = "drop") %>%
    split_name() %>%
    select(-config) %>%
    tidyr::pivot_wider(names_from = anchor, values_from = pct) %>%
    mutate(cell = paste(start, leak, sep = ", "))
})
for (d in names(STATE)) {
  cat("\n\n", cal_h(2), " ", d, "\n\n", sep = "")
  print(
    ggplot(STATE[[d]], aes(`constant 0.018`, `adaptive 0.03`)) +
      geom_abline(slope = 1, intercept = 0, colour = "grey50") +
      geom_hline(yintercept = 0, colour = "grey85") +
      geom_vline(xintercept = 0, colour = "grey85") +
      geom_point(alpha = 0.8) +
      facet_grid(season ~ cell) +
      labs(title = d, x = "constant 0.018: WIS reduction %, h0–h3", y = "adaptive 0.03: WIS reduction %, h0–h3")
  )
  STATE[[d]] %>%
    group_by(cell, season) %>%
    summarize(adaptive_better = sum(`adaptive 0.03` > `constant 0.018`), states = n(), .groups = "drop") %>%
    transmute(cell, season, v = sprintf("%d / %d", adaptive_better, states)) %>%
    tidyr::pivot_wider(names_from = season, values_from = v) %>%
    knitr::kable(caption = paste0(d, ": states (of those scored) where the adaptive anchor beats the constant one, h0–h3 pooled")) %>%
    print()
}

flu

flu: states (of those scored) where the adaptive anchor beats the constant one, h0–h3 pooled
cell 2024-2025 2025-2026
cold, leak 0.1 19 / 52 23 / 52
cold, no leak 11 / 52 5 / 52
warm, leak 0.1 15 / 52 13 / 52
warm, no leak 8 / 52 10 / 52

covid

covid: states (of those scored) where the adaptive anchor beats the constant one, h0–h3 pooled
cell 2024-2025 2025-2026
cold, leak 0.1 31 / 52 13 / 52
cold, no leak 14 / 52 13 / 52
warm, leak 0.1 26 / 52 20 / 52
warm, no leak 19 / 52 5 / 52

Month views

The month plots show the no-leak grid and flu prod; the tables keep every config.

mv_plot_cfgs <- c(grid_names[c(1, 3, 5, 7)], "REF-op warm")
mv_cols <- setNames(c("#E69F00", "#D55E00", "#0072B2", "#56B4E9", "#000000"), mv_plot_cfgs)
for (d in names(M)) {
  mv <- M[[d]]$mv
  cal_month_section(d, level = 2L)
}

flu

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
adaptive 0.03 | cold | no leak +1.8 +0.0 -1.4 -2.6 +2.2 -0.6 -0.7 -0.2 +1.9 -0.2 -1.1 -1.7
adaptive 0.03 | cold | leak 0.1 +2.0 +0.9 +0.1 -0.5 +1.5 +0.1 +0.4 +0.9 +1.8 +0.6 +0.2 -0.0
adaptive 0.03 | warm | no leak +2.8 +1.2 +0.4 -0.0 +2.4 -0.7 -0.8 -0.2 +2.7 +0.4 -0.1 -0.1
adaptive 0.03 | warm | leak 0.1 +3.1 +2.0 +1.8 +1.9 +2.1 +0.1 +0.1 +0.4 +2.7 +1.2 +1.1 +1.3
constant 0.018 | cold | no leak +1.7 +0.6 +0.3 +0.1 +3.0 +1.0 +1.1 +1.5 +2.2 +0.8 +0.6 +0.6
constant 0.018 | cold | leak 0.1 +1.5 +0.6 +0.4 +0.2 +1.3 +0.4 +0.4 +0.7 +1.4 +0.5 +0.4 +0.4
constant 0.018 | warm | no leak +3.4 +2.1 +2.2 +3.3 +3.4 +0.9 +0.8 +1.2 +3.4 +1.6 +1.6 +2.5
constant 0.018 | warm | leak 0.1 +3.6 +2.5 +2.7 +4.0 +2.9 +0.8 +0.8 +1.2 +3.3 +1.8 +1.9 +2.9
REF-op warm +3.2 +1.9 +1.6 +1.6 +2.2 -0.1 -0.0 +0.2 +2.8 +1.1 +0.9 +1.1
REF-op cold +2.2 +0.9 -0.0 -0.8 +1.8 +0.1 +0.3 +0.9 +2.0 +0.6 +0.1 -0.2 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.127 / 0.130 0.129 / 0.144 0.148 / 0.179 0.169 / 0.211 0.090 / 0.097 0.081 / 0.112 0.084 / 0.130 0.086 / 0.146 0.108 / 0.112 0.105 / 0.127 0.115 / 0.153 0.127 / 0.177
adaptive 0.03 | cold | no leak 0.085 / 0.115 0.077 / 0.138 0.078 / 0.174 0.075 / 0.213 0.008 / 0.078 0.010 / 0.105 0.014 / 0.130 0.016 / 0.143 0.045 / 0.095 0.039 / 0.120 0.037 / 0.150 0.032 / 0.176
adaptive 0.03 | cold | leak 0.1 0.110 / 0.123 0.111 / 0.140 0.123 / 0.176 0.132 / 0.213 0.067 / 0.082 0.060 / 0.104 0.063 / 0.126 0.064 / 0.134 0.088 / 0.101 0.085 / 0.121 0.093 / 0.149 0.097 / 0.171
adaptive 0.03 | warm | no leak 0.069 / 0.116 0.057 / 0.136 0.048 / 0.173 0.051 / 0.227 0.012 / 0.080 0.017 / 0.112 0.017 / 0.136 0.018 / 0.148 0.035 / 0.097 0.030 / 0.123 0.026 / 0.153 0.027 / 0.185
adaptive 0.03 | warm | leak 0.1 0.089 / 0.116 0.081 / 0.133 0.082 / 0.173 0.071 / 0.208 0.042 / 0.079 0.036 / 0.111 0.031 / 0.129 0.030 / 0.144 0.065 / 0.096 0.053 / 0.121 0.048 / 0.150 0.041 / 0.173
constant 0.018 | cold | no leak 0.099 / 0.115 0.104 / 0.136 0.120 / 0.170 0.137 / 0.204 0.021 / 0.075 0.023 / 0.098 0.024 / 0.119 0.022 / 0.138 0.059 / 0.094 0.062 / 0.116 0.071 / 0.143 0.078 / 0.169
constant 0.018 | cold | leak 0.1 0.117 / 0.125 0.120 / 0.141 0.138 / 0.176 0.157 / 0.209 0.075 / 0.086 0.071 / 0.108 0.074 / 0.128 0.075 / 0.141 0.096 / 0.104 0.095 / 0.124 0.105 / 0.150 0.115 / 0.173
constant 0.018 | warm | no leak 0.076 / 0.115 0.070 / 0.131 0.065 / 0.174 0.057 / 0.218 0.018 / 0.083 0.027 / 0.122 0.035 / 0.156 0.051 / 0.175 0.040 / 0.098 0.034 / 0.126 0.032 / 0.164 0.040 / 0.195
constant 0.018 | warm | leak 0.1 0.093 / 0.118 0.086 / 0.129 0.085 / 0.169 0.075 / 0.207 0.048 / 0.083 0.040 / 0.115 0.034 / 0.142 0.044 / 0.163 0.070 / 0.099 0.055 / 0.121 0.046 / 0.155 0.042 / 0.183
REF-op warm 0.085 / 0.115 0.079 / 0.133 0.077 / 0.175 0.066 / 0.208 0.035 / 0.078 0.032 / 0.111 0.027 / 0.129 0.029 / 0.146 0.060 / 0.095 0.049 / 0.121 0.043 / 0.150 0.038 / 0.175
REF-op cold 0.106 / 0.121 0.107 / 0.140 0.117 / 0.175 0.124 / 0.213 0.057 / 0.078 0.050 / 0.100 0.049 / 0.121 0.047 / 0.127 0.081 / 0.098 0.077 / 0.119 0.082 / 0.146 0.085 / 0.167

covid

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
adaptive 0.03 | cold | no leak +2.3 -0.7 -2.1 -2.8 -0.8 -4.5 -4.0 -1.2 +0.7 -2.7 -3.1 -1.9
adaptive 0.03 | cold | leak 0.1 +1.7 +0.4 +0.3 +0.4 +0.5 -2.0 -2.8 -3.1 +1.1 -0.8 -1.4 -1.4
adaptive 0.03 | warm | no leak +0.5 -2.5 -2.7 -1.0 -0.7 -3.9 -2.8 +1.0 -0.1 -3.2 -2.7 +0.1
adaptive 0.03 | warm | leak 0.1 -0.9 -2.5 -1.1 +1.7 -0.4 -1.3 +0.8 +4.9 -0.6 -1.9 -0.1 +3.4
constant 0.018 | cold | no leak +2.2 -0.2 -0.8 -0.8 -0.9 -3.3 -2.1 +0.9 +0.7 -1.8 -1.5 +0.1
constant 0.018 | cold | leak 0.1 +1.5 +0.3 +0.2 +0.3 +0.6 -1.4 -1.7 -1.2 +1.1 -0.5 -0.8 -0.5
constant 0.018 | warm | no leak -1.3 -5.1 -4.6 -2.8 -0.5 -1.7 +0.8 +5.0 -0.9 -3.4 -1.8 +1.3
constant 0.018 | warm | leak 0.1 -3.2 -6.8 -6.0 -4.0 -1.1 -2.6 -0.7 +2.2 -2.2 -4.7 -3.2 -0.8
REF-op warm -0.5 -2.3 -1.2 +1.4 +0.2 -1.2 +0.3 +4.1 -0.1 -1.8 -0.4 +2.8
REF-op cold +1.9 +0.3 +0.0 +0.1 +0.6 -2.4 -3.2 -3.1 +1.3 -1.1 -1.7 -1.6 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.080 / 0.081 0.030 / 0.082 0.013 / 0.109 0.048 / 0.131 0.049 / 0.103 0.032 / 0.119 0.050 / 0.147 0.087 / 0.177 0.057 / 0.094 0.023 / 0.103 0.034 / 0.131 0.070 / 0.157
adaptive 0.03 | cold | no leak 0.056 / 0.062 0.033 / 0.094 0.013 / 0.121 0.019 / 0.142 0.023 / 0.091 0.021 / 0.125 0.020 / 0.148 0.024 / 0.172 0.022 / 0.079 0.016 / 0.112 0.014 / 0.136 0.017 / 0.159
adaptive 0.03 | cold | leak 0.1 0.071 / 0.073 0.036 / 0.088 0.012 / 0.113 0.025 / 0.136 0.039 / 0.098 0.027 / 0.121 0.033 / 0.146 0.059 / 0.175 0.047 / 0.087 0.023 / 0.107 0.018 / 0.132 0.044 / 0.158
adaptive 0.03 | warm | no leak 0.086 / 0.089 0.093 / 0.108 0.096 / 0.126 0.105 / 0.148 0.021 / 0.092 0.020 / 0.123 0.025 / 0.148 0.029 / 0.171 0.034 / 0.091 0.044 / 0.117 0.050 / 0.138 0.057 / 0.161
adaptive 0.03 | warm | leak 0.1 0.108 / 0.108 0.113 / 0.122 0.119 / 0.136 0.128 / 0.156 0.065 / 0.115 0.082 / 0.142 0.094 / 0.163 0.106 / 0.180 0.082 / 0.112 0.094 / 0.134 0.104 / 0.152 0.115 / 0.170
constant 0.018 | cold | no leak 0.060 / 0.065 0.029 / 0.089 0.008 / 0.114 0.037 / 0.135 0.025 / 0.092 0.025 / 0.125 0.032 / 0.147 0.049 / 0.172 0.025 / 0.080 0.015 / 0.110 0.021 / 0.133 0.043 / 0.156
constant 0.018 | cold | leak 0.1 0.073 / 0.074 0.032 / 0.085 0.008 / 0.110 0.039 / 0.132 0.040 / 0.098 0.028 / 0.120 0.040 / 0.147 0.073 / 0.174 0.048 / 0.088 0.022 / 0.105 0.026 / 0.131 0.059 / 0.156
constant 0.018 | warm | no leak 0.091 / 0.094 0.106 / 0.116 0.117 / 0.135 0.136 / 0.163 0.022 / 0.092 0.025 / 0.122 0.046 / 0.150 0.076 / 0.181 0.038 / 0.093 0.056 / 0.120 0.075 / 0.144 0.100 / 0.173
constant 0.018 | warm | leak 0.1 0.111 / 0.110 0.120 / 0.128 0.129 / 0.142 0.145 / 0.167 0.068 / 0.117 0.093 / 0.149 0.114 / 0.175 0.146 / 0.208 0.085 / 0.114 0.103 / 0.140 0.120 / 0.161 0.145 / 0.191
REF-op warm 0.104 / 0.104 0.110 / 0.120 0.116 / 0.135 0.125 / 0.155 0.052 / 0.107 0.067 / 0.133 0.078 / 0.156 0.090 / 0.176 0.073 / 0.106 0.084 / 0.128 0.093 / 0.147 0.104 / 0.167
REF-op cold 0.068 / 0.071 0.035 / 0.089 0.014 / 0.115 0.023 / 0.137 0.033 / 0.095 0.026 / 0.122 0.031 / 0.147 0.052 / 0.175 0.042 / 0.085 0.022 / 0.108 0.017 / 0.133 0.039 / 0.159