Question: does shrinking the offset from 1 March, or switching calibration off from 1 March, change WIS and coverage compared with the same tracker without it?

Setup:

  • Disease: flu. Base forecaster: the clean windowed_seasonal replay (all 53 locations, the 55 hub rounds). Horizons h0–h3.
  • Learning truth: exact (re-run at every round on the data published by then; E00), one week of extra delay. Cold start, no burn-in season. States only.
  • Three constant-rate trackers: rate 0.018 and rate 0.032 per 100k (E16), and sqrt 0.018 (E16). Each crossed with five settings:
    • no decay.
    • late_decay with factor 0.85, 0.7 or 0.5: from 1 March to 30 June the stored offset is multiplied by the factor at each round before it is played. Learning continues, and because offsets carry across seasons, the shrink carries into the next season.
    • off_after = "03-01": from 1 March to 30 June the base is played, those rounds produce no updates, and the offset carries unchanged into the next season.
  • The 1 March date was chosen after E16, on these same seasons. (fast_decay and season_policy are other ways to shrink offsets; they are not run here.)

WIS reduction % = 100 × (base − calibrated) / base, so positive = calibrated WIS lower than base. L1 coverage bias: lower is better.

source(here::here("scripts/calibration/calibration_ws_replay.R"))
wi <- ws_inputs("flu")
base_cfg <- ws_configs(wi$rate_scales, burn_in = FALSE)[["E11 constant 0.1"]]
SQRT_BEST <- 0.018
RATES <- list(
  `rate 0.018` = list(lr = 0.018),
  `rate 0.032` = list(lr = 0.032),
  sqrt = list(lr = SQRT_BEST, transform = "sqrt")
)
names(RATES)[3] <- paste("sqrt", SQRT_BEST)
DECAYS <- list(
  `no decay` = list(),
  `decay 0.85` = list(late_decay = list(from = "03-01", factor = 0.85)),
  `decay 0.7` = list(late_decay = list(from = "03-01", factor = 0.7)),
  `decay 0.5` = list(late_decay = list(from = "03-01", factor = 0.5)),
  `off after 1 Mar` = list(off_after = "03-01")
)
grid <- tidyr::expand_grid(rate = names(RATES), decay = names(DECAYS))
# Cached under cache/calibration/experiments/runs; the no-decay runs are E16's.
cals <- purrr::pmap(grid, function(rate, decay) {
  cfg <- c(utils::modifyList(base_cfg, RATES[[rate]]), DECAYS[[decay]])
  do.call(cal_run_cached, c(list(wi$fc, wi$truth), cfg, list(learn = "exact", vintages = wi$vintages, workers = params$workers)))
})
fc <- purrr::map2(cals, seq_len(nrow(grid)), function(cal, i) {
  cal$forecasts %>% mutate(rate = grid$rate[i], decay = grid$decay[i])
}) %>%
  bind_rows() %>%
  mutate(season = season_of(reference_date), rate = factor(rate, levels = names(RATES)), decay = factor(decay, levels = names(DECAYS))) %>%
  filter(season %in% LIVE, location != "US")
DECAY_COLS <- setNames(c("black", "#56B4E9", "#0072B2", "#D55E00", "#009E73"), names(DECAYS))
mv <- cal_month_view(setNames(cals, paste(grid$rate, grid$decay, sep = ", ")), seasons = LIVE)
# The plots show sqrt 0.018 with each decay setting; the tables cover all three trackers.
mv_plot_cfgs <- paste(names(RATES)[3], names(DECAYS), sep = ", ")
mv_cols <- setNames(DECAY_COLS, mv_plot_cfgs)

Month view: WIS, coverage, median error

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
rate 0.018, no decay +0.7 +0.4 +0.3 +0.2 +1.9 +1.0 +0.9 +1.0 +1.2 +0.6 +0.5 +0.5
rate 0.018, decay 0.85 +0.4 +0.2 +0.1 +0.1 +1.1 +0.5 +0.5 +0.5 +0.7 +0.3 +0.3 +0.3
rate 0.018, decay 0.7 +0.4 +0.2 +0.1 +0.1 +0.8 +0.4 +0.4 +0.4 +0.5 +0.3 +0.2 +0.2
rate 0.018, decay 0.5 +0.3 +0.2 +0.1 +0.1 +0.7 +0.4 +0.4 +0.5 +0.5 +0.3 +0.3 +0.2
rate 0.018, off after 1 Mar +0.3 +0.2 +0.1 +0.1 +1.5 +0.9 +1.0 +1.2 +0.7 +0.5 +0.5 +0.5
rate 0.032, no decay +1.0 +0.5 +0.3 +0.2 +1.9 +1.0 +0.8 +0.8 +1.3 +0.7 +0.5 +0.4
rate 0.032, decay 0.85 +0.7 +0.4 +0.2 +0.2 +1.6 +0.8 +0.7 +0.8 +1.1 +0.5 +0.4 +0.4
rate 0.032, decay 0.7 +0.6 +0.3 +0.2 +0.2 +1.3 +0.7 +0.6 +0.7 +0.9 +0.5 +0.4 +0.4
rate 0.032, decay 0.5 +0.6 +0.3 +0.3 +0.2 +1.2 +0.7 +0.7 +0.8 +0.8 +0.5 +0.4 +0.4
rate 0.032, off after 1 Mar +0.5 +0.3 +0.3 +0.2 +2.2 +1.4 +1.6 +2.0 +1.2 +0.7 +0.8 +0.9
sqrt 0.018, no decay +1.7 +0.6 +0.3 +0.1 +3.0 +1.0 +1.1 +1.5 +2.2 +0.8 +0.6 +0.6
sqrt 0.018, decay 0.85 +1.7 +0.7 +0.4 +0.2 +2.1 +0.6 +0.6 +0.8 +1.8 +0.6 +0.5 +0.4
sqrt 0.018, decay 0.7 +1.7 +0.8 +0.5 +0.3 +1.8 +0.5 +0.6 +0.7 +1.7 +0.7 +0.5 +0.5
sqrt 0.018, decay 0.5 +1.7 +0.8 +0.6 +0.4 +1.7 +0.6 +0.7 +0.9 +1.7 +0.7 +0.6 +0.6
sqrt 0.018, off after 1 Mar +1.6 +0.8 +0.7 +0.5 +3.4 +1.6 +1.9 +2.6 +2.3 +1.2 +1.2 +1.3 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.127 / 0.130 0.129 / 0.144 0.148 / 0.179 0.169 / 0.211 0.090 / 0.097 0.081 / 0.112 0.084 / 0.130 0.086 / 0.146 0.108 / 0.112 0.105 / 0.127 0.115 / 0.153 0.127 / 0.177
rate 0.018, no decay 0.110 / 0.118 0.113 / 0.136 0.127 / 0.169 0.143 / 0.200 0.032 / 0.088 0.030 / 0.106 0.028 / 0.128 0.024 / 0.147 0.071 / 0.102 0.071 / 0.120 0.076 / 0.147 0.082 / 0.172
rate 0.018, decay 0.85 0.124 / 0.128 0.126 / 0.143 0.144 / 0.177 0.164 / 0.210 0.077 / 0.086 0.071 / 0.107 0.074 / 0.125 0.075 / 0.141 0.100 / 0.106 0.098 / 0.124 0.108 / 0.149 0.118 / 0.173
rate 0.018, decay 0.7 0.126 / 0.129 0.128 / 0.144 0.147 / 0.179 0.168 / 0.212 0.084 / 0.092 0.076 / 0.110 0.079 / 0.129 0.081 / 0.145 0.104 / 0.109 0.102 / 0.126 0.112 / 0.152 0.124 / 0.176
rate 0.018, decay 0.5 0.126 / 0.129 0.129 / 0.144 0.148 / 0.178 0.168 / 0.211 0.086 / 0.094 0.078 / 0.109 0.081 / 0.129 0.083 / 0.145 0.106 / 0.111 0.103 / 0.126 0.114 / 0.152 0.125 / 0.176
rate 0.018, off after 1 Mar 0.126 / 0.129 0.129 / 0.144 0.147 / 0.178 0.169 / 0.211 0.079 / 0.095 0.073 / 0.110 0.076 / 0.132 0.079 / 0.141 0.102 / 0.111 0.100 / 0.125 0.111 / 0.154 0.123 / 0.174
rate 0.032, no decay 0.098 / 0.112 0.101 / 0.131 0.111 / 0.164 0.123 / 0.196 0.013 / 0.104 0.010 / 0.123 0.010 / 0.147 0.014 / 0.161 0.053 / 0.108 0.053 / 0.127 0.056 / 0.155 0.061 / 0.177
rate 0.032, decay 0.85 0.121 / 0.127 0.123 / 0.142 0.140 / 0.176 0.160 / 0.209 0.069 / 0.082 0.065 / 0.104 0.067 / 0.123 0.068 / 0.138 0.095 / 0.103 0.094 / 0.121 0.103 / 0.148 0.113 / 0.171
rate 0.032, decay 0.7 0.125 / 0.129 0.128 / 0.144 0.146 / 0.178 0.167 / 0.212 0.079 / 0.088 0.073 / 0.109 0.075 / 0.128 0.078 / 0.143 0.102 / 0.107 0.100 / 0.125 0.110 / 0.151 0.122 / 0.175
rate 0.032, decay 0.5 0.126 / 0.129 0.128 / 0.144 0.147 / 0.178 0.168 / 0.212 0.083 / 0.091 0.076 / 0.109 0.078 / 0.128 0.081 / 0.144 0.104 / 0.109 0.101 / 0.125 0.112 / 0.152 0.124 / 0.175
rate 0.032, off after 1 Mar 0.126 / 0.129 0.129 / 0.143 0.147 / 0.178 0.168 / 0.211 0.071 / 0.097 0.067 / 0.114 0.070 / 0.137 0.073 / 0.140 0.098 / 0.112 0.097 / 0.128 0.108 / 0.156 0.120 / 0.173
sqrt 0.018, no decay 0.099 / 0.115 0.104 / 0.136 0.120 / 0.170 0.137 / 0.204 0.021 / 0.075 0.023 / 0.098 0.024 / 0.119 0.022 / 0.138 0.059 / 0.094 0.062 / 0.116 0.071 / 0.143 0.078 / 0.169
sqrt 0.018, decay 0.85 0.117 / 0.125 0.120 / 0.141 0.139 / 0.176 0.159 / 0.210 0.066 / 0.084 0.064 / 0.108 0.067 / 0.127 0.070 / 0.140 0.091 / 0.103 0.091 / 0.123 0.102 / 0.149 0.114 / 0.173
sqrt 0.018, decay 0.7 0.121 / 0.127 0.124 / 0.142 0.143 / 0.176 0.164 / 0.211 0.074 / 0.088 0.069 / 0.110 0.073 / 0.129 0.076 / 0.143 0.097 / 0.106 0.096 / 0.125 0.107 / 0.151 0.119 / 0.174
sqrt 0.018, decay 0.5 0.123 / 0.127 0.125 / 0.142 0.144 / 0.176 0.165 / 0.210 0.078 / 0.090 0.071 / 0.109 0.075 / 0.128 0.078 / 0.142 0.100 / 0.107 0.098 / 0.124 0.109 / 0.151 0.121 / 0.174
sqrt 0.018, off after 1 Mar 0.124 / 0.126 0.127 / 0.141 0.145 / 0.176 0.167 / 0.209 0.065 / 0.084 0.062 / 0.104 0.066 / 0.127 0.067 / 0.136 0.094 / 0.104 0.094 / 0.121 0.105 / 0.150 0.116 / 0.170

Mean offset at the median over time

Calibrated minus base median in admissions per 100k, averaged over states at each round, h2.

pop <- wi$rate_scales %>% select(location, scale)
fc %>%
  filter(level == 0.5, horizon == 2) %>%
  left_join(pop, by = "location") %>%
  group_by(rate, decay, season, reference_date) %>%
  summarize(off = mean((value_cal - value_base) / scale), .groups = "drop") %>%
  ggplot(aes(reference_date, off, colour = decay)) +
  geom_hline(yintercept = 0, colour = "grey60") +
  geom_line() +
  geom_point(size = 0.6) +
  facet_grid(rate ~ season, scales = "free") +
  scale_colour_manual(values = DECAY_COLS, name = NULL) +
  scale_x_date(date_breaks = "1 month", date_labels = "%b") +
  labs(x = "reference date", y = "mean offset at the median, per 100k") +
  theme(legend.position = "bottom")