Question: how do three variants of the adaptive tracker, each changed alone from a reference, change WIS and coverage?

  • Switch off in spring (off_after): from a calendar date until July, play the base forecast and stop learning; the offsets carry unchanged into the next season. Dates: 1 April, 1 March, 15 February.
  • Per-level learning rate (lr_args$per_level): each of the 23 quantile levels gets its own eta, instead of one eta pooled over levels.
  • Seasonal window (lr_seasonal): eta is sized from the last 10 rounds plus the rounds within 5 weeks of the same point in earlier seasons. Run with carry and with reset at the summer. This also shortens the window from 20 to 10, so it changes two things at once.

This re-asks the original E02 (a pre-fix, count-scale comparison on the submitted ensemble with finalized truth, since deleted) on the current setup.

Setup:

  • Disease: flu. Base forecaster: the clean windowed_seasonal replay (all 53 locations, the 55 hub rounds). Horizons h0–h3.
  • Learning truth: exact (re-run at every round on the data published by then; E00), one week of extra delay.
  • Reference: E01’s center point, the sqrt single-term tracker with the adaptive rate (mult 0.03, floor 0.001, window 20, carry), cold start (no burn-in).
  • The seasonal window needs an earlier season to look back on, so it runs with the 2023-24 burn-in (the HHS-stitched backfill from E13), which only warms eta. The reference is re-run with the same burn-in (reference + burn-in) for comparison.
  • Scores: states only (US excluded).

WIS reduction % = 100 × (base − calibrated) / base, so positive = calibrated WIS lower than base. L1 coverage bias: lower is better.

source(here::here("scripts/calibration/calibration_ws_replay.R"))
wi <- ws_inputs("flu")
fc_burn_in <- bind_rows(ws_burn_in_forecasts("flu"), wi$fc)
# Same builder as E01, so the reference run is shared.
sqrt_cfg <- function(mult = 0.03, window = 20, policy = "carry") {
  list(
    ref = "paper", transform = "sqrt", lr_args = list(mult = mult, floor = 1e-3),
    burn_in_seasons = character(0), slow_init = NULL, lr_window = window, season_policy = policy
  )
}
ref <- sqrt_cfg()
burn <- list(burn_in_seasons = "2023-2024")
seasonal <- list(lr_window = 10, lr_seasonal = list(half_width_weeks = 5))
CFGS <- list(
  reference = list(cfg = ref),
  `off after 1 Apr` = list(cfg = c(ref, list(off_after = "04-01"))),
  `off after 1 Mar` = list(cfg = c(ref, list(off_after = "03-01"))),
  `off after 15 Feb` = list(cfg = c(ref, list(off_after = "02-15"))),
  `per-level eta` = list(cfg = utils::modifyList(ref, list(lr_args = list(mult = 0.03, floor = 1e-3, per_level = TRUE)))),
  `reference + burn-in` = list(cfg = utils::modifyList(ref, burn), burn_in = TRUE),
  `seasonal window, carry` = list(cfg = utils::modifyList(ref, c(burn, seasonal)), burn_in = TRUE),
  `seasonal window, reset` = list(cfg = utils::modifyList(ref, c(burn, seasonal, list(season_policy = "reset"))), burn_in = TRUE)
)
run <- function(spec) {
  fc <- if (isTRUE(spec$burn_in)) fc_burn_in else wi$fc
  do.call(cal_run_cached, c(list(fc, wi$truth), spec$cfg, list(learn = "exact", vintages = wi$vintages, workers = params$workers)))
}
cals <- purrr::map(CFGS, run)
CFG_COLS <- setNames(c("black", "#56B4E9", "#0072B2", "#003F6B", "#CC79A7", "grey55", "#E69F00", "#D55E00"), names(CFGS))
mv <- cal_month_view(cals, seasons = LIVE)
# The plots leave out the burn-in configs except the seasonal window with reset; the tables keep every config.
mv_plot_cfgs <- c("reference", "off after 1 Mar", "off after 15 Feb", "per-level eta", "seasonal window, reset")
mv_cols <- CFG_COLS

The seasonal-window configs compare with “reference + burn-in”, the others with “reference”.

Month view: WIS, coverage, median error

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
reference +1.8 +0.0 -1.4 -2.6 +2.2 -0.6 -0.7 -0.2 +1.9 -0.2 -1.1 -1.7
off after 1 Apr +1.5 +0.1 -0.9 -1.6 +1.9 -0.5 -0.3 +0.6 +1.6 -0.2 -0.7 -0.8
off after 1 Mar +2.3 +1.6 +1.4 +1.2 +3.6 +1.8 +2.8 +4.6 +2.8 +1.7 +2.0 +2.5
off after 15 Feb +2.0 +1.7 +1.6 +1.3 +3.8 +2.6 +3.8 +6.2 +2.7 +2.1 +2.5 +3.2
per-level eta +1.6 +0.6 -0.2 -0.9 +2.8 +0.5 +0.2 +0.7 +2.1 +0.5 -0.0 -0.3
reference + burn-in +1.9 +0.4 -0.6 -1.5 +2.3 -0.7 -0.7 -0.2 +2.0 -0.0 -0.6 -1.0
seasonal window, carry +2.0 +0.5 -0.6 -1.6 +2.0 -0.9 -1.1 -0.8 +2.0 -0.1 -0.8 -1.3
seasonal window, reset +2.0 +0.5 -0.6 -1.6 +0.5 -1.2 -1.4 -1.6 +1.4 -0.2 -0.9 -1.6 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.127 / 0.130 0.129 / 0.144 0.148 / 0.179 0.169 / 0.211 0.090 / 0.097 0.081 / 0.112 0.084 / 0.130 0.086 / 0.146 0.108 / 0.112 0.105 / 0.127 0.115 / 0.153 0.127 / 0.177
reference 0.085 / 0.115 0.077 / 0.138 0.078 / 0.174 0.075 / 0.213 0.008 / 0.078 0.010 / 0.105 0.014 / 0.130 0.016 / 0.143 0.045 / 0.095 0.039 / 0.120 0.037 / 0.150 0.032 / 0.176
off after 1 Apr 0.110 / 0.127 0.110 / 0.144 0.124 / 0.178 0.141 / 0.212 0.040 / 0.090 0.031 / 0.122 0.026 / 0.146 0.016 / 0.151 0.074 / 0.107 0.070 / 0.132 0.074 / 0.161 0.077 / 0.180
off after 1 Mar 0.122 / 0.125 0.124 / 0.139 0.140 / 0.171 0.160 / 0.203 0.049 / 0.085 0.038 / 0.112 0.032 / 0.137 0.024 / 0.141 0.085 / 0.104 0.080 / 0.124 0.085 / 0.153 0.091 / 0.170
off after 15 Feb 0.124 / 0.127 0.127 / 0.142 0.145 / 0.175 0.166 / 0.208 0.059 / 0.088 0.047 / 0.113 0.044 / 0.133 0.038 / 0.134 0.091 / 0.106 0.086 / 0.126 0.093 / 0.153 0.100 / 0.169
per-level eta 0.094 / 0.115 0.087 / 0.136 0.091 / 0.172 0.091 / 0.207 0.017 / 0.073 0.017 / 0.101 0.017 / 0.126 0.014 / 0.143 0.055 / 0.092 0.049 / 0.117 0.049 / 0.147 0.043 / 0.173
reference + burn-in 0.092 / 0.116 0.085 / 0.137 0.090 / 0.173 0.091 / 0.209 0.011 / 0.078 0.010 / 0.105 0.012 / 0.128 0.014 / 0.144 0.050 / 0.096 0.045 / 0.120 0.044 / 0.149 0.040 / 0.174
seasonal window, carry 0.087 / 0.115 0.080 / 0.136 0.085 / 0.172 0.086 / 0.210 0.008 / 0.082 0.013 / 0.114 0.025 / 0.143 0.038 / 0.159 0.044 / 0.098 0.036 / 0.124 0.032 / 0.156 0.025 / 0.183
seasonal window, reset 0.087 / 0.115 0.080 / 0.136 0.085 / 0.172 0.086 / 0.210 0.045 / 0.076 0.033 / 0.103 0.024 / 0.126 0.020 / 0.148 0.066 / 0.094 0.056 / 0.118 0.054 / 0.147 0.053 / 0.177