# Shared by the experiment notebooks: hub submissions, finalized truth for
# scoring, NHSN vintages for learning, and population for rates. The disease
# comes from `params$disease` (default flu).
suppressPackageStartupMessages(source(here::here("R/load_all.R")))
disease <- params$disease %||% "flu"
archive <- get_nhsn_data_archive(disease)
truth <- nhsn_read_truth(disease, archive)
vintages <- nhsn_read_vintages(disease, archive)
hub_target <- c(flu = HUB_FLU_TARGET, covid = HUB_COVID_TARGET)[[disease]]
hub_fc <- suppressMessages(hub_read_forecasts(hub_dir = here::here(params$hub_dir), target = hub_target))
population <- get_population_data() %>%
  distinct(state_code, .keep_all = TRUE) %>%
  filter(state_code %in% unique(hub_fc$location)) %>%
  select(location = state_code, population)
rate_scales <- population %>% transmute(location, from = as.Date("2000-01-01"), scale = population / 1e5)
LIVE <- c("2024-2025", "2025-2026")
run <- function(...) cal_run_cached(hub_fc, truth, ..., vintages = vintages, workers = params$workers)

Question: does the scale the tracker works on (count, sqrt, rate) change WIS and coverage?

Setup:

  • Disease: flu. Base forecaster: the submitted hub ensemble (spoiled submissions dropped).
  • Reference: REF-paper. One offset per level; adaptive learning rate of 0.03 × the 90th percentile of the last 20 rounds’ absolute errors; offsets carried across seasons; one week of extra delay (settle_days = 14).
  • Burn-in: 2023-24, used only to warm up the learning rate. Offsets start at zero at the first 2024-25 round (cold start).
  • Learning truth: exact (re-run at every round on the data published by then, as prod does; E00). Scores against finalized truth.
  • Seasons scored: 2024-25 and 2025-26. Horizons h−1 to h3. States only (US excluded).
config change from REF-paper
count none (additive offsets in admissions)
sqrt offsets on the square root of admissions, mapped back; learning-rate floor 0.001
rate offsets in admissions per 100k; learning-rate floor 0.001

The learning-rate floor (0.1 by default) is in the tracker’s working units, so the sqrt and rate configs use 0.001.

cals <- list(
  count = run("paper"),
  sqrt = run("paper", transform = "sqrt", lr_args = list(mult = 0.03, floor = 1e-3)),
  rate = run("paper", scales = rate_scales, lr_args = list(mult = 0.03, floor = 1e-3))
)
mv <- cal_month_view(cals, seasons = LIVE, horizons = NULL)

Month view: WIS, coverage, median error

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h-1 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h-1 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h-1 both h0 both h1 both h2 both h3
count +4.6 +1.6 +0.5 -1.5 -4.5 -2.0 -1.7 -4.9 -5.6 -6.5 +1.9 +0.6 -1.5 -3.1 -5.3
sqrt +8.1 +2.1 +1.1 -0.2 -1.4 -1.1 +2.6 -2.0 -3.1 -2.5 +4.3 +2.3 -0.0 -1.3 -1.8
rate +4.6 +1.6 +0.5 -1.5 -4.5 -2.0 -1.7 -4.9 -5.6 -6.5 +1.9 +0.6 -1.5 -3.1 -5.3 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h-1 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h-1 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h-1 both h0 both h1 both h2 both h3
base 0.183 / 0.191 0.131 / 0.146 0.135 / 0.152 0.146 / 0.169 0.159 / 0.190 0.037 / 0.084 0.078 / 0.089 0.055 / 0.098 0.050 / 0.108 0.046 / 0.117 0.110 / 0.134 0.104 / 0.116 0.094 / 0.123 0.097 / 0.137 0.101 / 0.151
count 0.100 / 0.160 0.057 / 0.122 0.050 / 0.135 0.042 / 0.172 0.046 / 0.223 0.013 / 0.076 0.008 / 0.085 0.018 / 0.106 0.028 / 0.118 0.040 / 0.120 0.044 / 0.115 0.030 / 0.102 0.032 / 0.120 0.033 / 0.143 0.031 / 0.168
sqrt 0.123 / 0.149 0.091 / 0.118 0.089 / 0.122 0.089 / 0.143 0.082 / 0.181 0.027 / 0.072 0.034 / 0.089 0.038 / 0.113 0.034 / 0.118 0.030 / 0.122 0.049 / 0.108 0.032 / 0.102 0.026 / 0.117 0.028 / 0.130 0.026 / 0.150
rate 0.100 / 0.160 0.057 / 0.122 0.050 / 0.135 0.042 / 0.172 0.046 / 0.223 0.013 / 0.076 0.008 / 0.085 0.018 / 0.106 0.028 / 0.118 0.040 / 0.120 0.044 / 0.115 0.030 / 0.102 0.032 / 0.120 0.033 / 0.143 0.031 / 0.168