Question: how do results change when the tracker learns from NHSN data as published, rather than from finalized values?

Setup:

  • Disease: flu. Base forecaster: the submitted hub ensemble (spoiled submissions dropped).
  • References: REF-paper (count scale, one offset per level) and REF-op (sqrt scale, leak 0.1, slow term with a burn-in-quantile warm start); see the index.
  • Burn-in: 2023-24, used only to warm up the learning rate. Offsets start at zero at the first 2024-25 round (REF-op’s slow term starts from its warm start).
  • Seasons scored: 2024-25 and 2025-26, burn-in rounds excluded. Horizons h−1 to h3. States only (US excluded).
  • Scores are always against finalized truth.

Configs are ref/settle_days/learn:

axis values
ref paper, op
settle_days 14 (one week of extra delay, the reference) or 7 (none)
learn final: latest vintage. exact: re-run from the first round at every round on the data published by then, as prod does. vintage: one run where each outcome keeps the value it had at the round that revealed it (a stateful tracker that never revisits a step)

A round with reference date d sees data published by d - 3. Vintages start 2024-11-19; earlier as-of dates use that first vintage. “WIS reduction %” is 100 × (base WIS − calibrated WIS) / base WIS, so positive = calibrated WIS lower than base. “L1 coverage bias” is lower-is-better.

suppressPackageStartupMessages(source(here::here("R/load_all.R")))
archive <- get_nhsn_data_archive("flu")
truth <- nhsn_read_truth("flu", archive)
vintages <- nhsn_read_vintages("flu", archive)
hub_fc <- hub_read_forecasts(hub_dir = here::here(params$hub_dir))
grid <- tidyr::expand_grid(ref = c("paper", "op"), settle_days = c(7L, 14L), learn = c("final", "vintage", "exact"))
cals <- purrr::pmap(grid, function(ref, settle_days, learn) {
  cal_run(hub_fc, truth, ref, settle_days = settle_days, learn = learn, vintages = vintages, workers = params$workers)
})
names(cals) <- with(grid, paste(ref, settle_days, learn, sep = "/"))
LIVE <- c("2024-2025", "2025-2026")
first_version <- min(vintages$version)
n_clamped <- sum(unique(hub_fc$reference_date) - 3L < first_version &
  hub_label_seasons(unique(hub_fc$reference_date))$season %in% LIVE)
mv <- cal_month_view(cals, seasons = LIVE, horizons = NULL)
# Runs drawn in the month view; the tables keep all twelve.
mv_plot_cfgs <- c("op/14/final", "op/14/exact", "op/7/vintage", "op/7/exact", "paper/14/final", "paper/14/exact")

0 live rounds are issued before the first vintage (2024-11-19); only burn-in rounds use that vintage in place of their own. Configs are ref/settle_days/learn.

Month view: WIS, coverage, median error

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h-1 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h-1 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h-1 both h0 both h1 both h2 both h3
paper/7/final +8.4 +2.5 +0.7 -1.0 -3.8 -0.8 -0.1 -3.4 -4.8 -5.8 +4.5 +1.7 -0.8 -2.5 -4.6
paper/7/vintage -0.5 +0.9 +0.7 -0.2 -2.3 -2.3 -0.1 -2.7 -3.0 -3.6 -1.3 +0.6 -0.5 -1.3 -2.8
paper/7/exact +5.7 +2.1 +0.8 -0.8 -3.5 -0.9 -0.3 -3.4 -4.5 -5.6 +2.9 +1.3 -0.7 -2.2 -4.3
paper/14/final +6.3 +1.8 +0.3 -1.8 -5.0 -2.0 -1.5 -4.8 -5.7 -6.6 +2.9 +0.7 -1.6 -3.3 -5.6
paper/14/vintage +4.6 +1.7 +0.7 -1.3 -4.3 -0.5 -0.4 -3.7 -4.7 -6.0 +2.5 +1.0 -0.9 -2.6 -4.9
paper/14/exact +4.6 +1.6 +0.5 -1.5 -4.5 -2.0 -1.7 -4.9 -5.6 -6.5 +1.9 +0.6 -1.5 -3.1 -5.3
op/7/final +12.9 +4.0 +3.1 +3.1 +4.4 +1.3 +4.6 -0.2 -1.1 -1.1 +8.1 +4.2 +1.9 +1.5 +2.4
op/7/vintage -0.1 +2.3 +2.3 +2.5 +4.2 +0.4 +2.5 -0.0 -0.7 -0.7 +0.1 +2.3 +1.4 +1.3 +2.4
op/7/exact +8.3 +3.4 +2.9 +3.0 +4.4 +1.0 +4.4 -0.1 -1.0 -1.1 +5.3 +3.7 +1.8 +1.4 +2.4
op/14/final +11.1 +3.7 +2.6 +2.5 +3.9 +0.9 +4.4 +0.0 -0.6 -0.8 +6.9 +3.9 +1.7 +1.3 +2.2
op/14/vintage +6.5 +3.2 +2.5 +2.4 +4.0 +0.8 +3.9 +0.1 -0.6 -0.7 +4.1 +3.4 +1.6 +1.3 +2.2
op/14/exact +7.6 +3.3 +2.5 +2.5 +3.9 +0.8 +4.2 +0.1 -0.5 -0.8 +4.8 +3.6 +1.6 +1.3 +2.2 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h-1 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h-1 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h-1 both h0 both h1 both h2 both h3
base 0.183 / 0.191 0.131 / 0.146 0.135 / 0.152 0.146 / 0.169 0.159 / 0.190 0.037 / 0.084 0.078 / 0.089 0.055 / 0.098 0.050 / 0.108 0.046 / 0.117 0.110 / 0.134 0.104 / 0.116 0.094 / 0.123 0.097 / 0.137 0.101 / 0.151
paper/7/final 0.077 / 0.166 0.048 / 0.122 0.043 / 0.133 0.047 / 0.171 0.044 / 0.211 0.019 / 0.073 0.015 / 0.090 0.012 / 0.109 0.013 / 0.115 0.021 / 0.123 0.029 / 0.116 0.019 / 0.105 0.025 / 0.120 0.027 / 0.141 0.028 / 0.164
paper/7/vintage 0.204 / 0.209 0.107 / 0.135 0.100 / 0.132 0.094 / 0.157 0.070 / 0.200 0.083 / 0.097 0.075 / 0.085 0.051 / 0.110 0.038 / 0.111 0.043 / 0.125 0.143 / 0.149 0.091 / 0.108 0.074 / 0.120 0.064 / 0.132 0.055 / 0.160
paper/7/exact 0.105 / 0.155 0.064 / 0.116 0.054 / 0.126 0.055 / 0.168 0.048 / 0.208 0.014 / 0.070 0.010 / 0.087 0.015 / 0.110 0.015 / 0.115 0.023 / 0.124 0.047 / 0.110 0.029 / 0.100 0.032 / 0.118 0.032 / 0.139 0.032 / 0.163
paper/14/final 0.076 / 0.172 0.042 / 0.128 0.037 / 0.141 0.034 / 0.175 0.046 / 0.227 0.016 / 0.079 0.009 / 0.085 0.015 / 0.109 0.025 / 0.119 0.039 / 0.120 0.030 / 0.122 0.021 / 0.105 0.024 / 0.124 0.027 / 0.145 0.029 / 0.170
paper/14/vintage 0.117 / 0.159 0.069 / 0.119 0.057 / 0.133 0.047 / 0.169 0.045 / 0.220 0.029 / 0.077 0.027 / 0.080 0.028 / 0.107 0.029 / 0.116 0.047 / 0.122 0.073 / 0.115 0.047 / 0.098 0.041 / 0.119 0.036 / 0.141 0.037 / 0.168
paper/14/exact 0.100 / 0.160 0.057 / 0.122 0.050 / 0.135 0.042 / 0.172 0.046 / 0.223 0.013 / 0.076 0.008 / 0.085 0.018 / 0.106 0.028 / 0.118 0.040 / 0.120 0.044 / 0.115 0.030 / 0.102 0.032 / 0.120 0.033 / 0.143 0.031 / 0.168
op/7/final 0.122 / 0.147 0.083 / 0.114 0.084 / 0.120 0.089 / 0.140 0.080 / 0.161 0.012 / 0.063 0.017 / 0.066 0.007 / 0.094 0.015 / 0.111 0.027 / 0.114 0.066 / 0.102 0.050 / 0.089 0.041 / 0.106 0.043 / 0.125 0.041 / 0.136
op/7/vintage 0.193 / 0.202 0.105 / 0.130 0.103 / 0.133 0.105 / 0.148 0.093 / 0.162 0.048 / 0.078 0.052 / 0.075 0.025 / 0.093 0.026 / 0.111 0.028 / 0.113 0.120 / 0.136 0.078 / 0.101 0.062 / 0.112 0.059 / 0.128 0.052 / 0.136
op/7/exact 0.138 / 0.156 0.089 / 0.118 0.088 / 0.123 0.094 / 0.142 0.083 / 0.160 0.013 / 0.063 0.021 / 0.065 0.008 / 0.093 0.016 / 0.111 0.026 / 0.113 0.075 / 0.107 0.054 / 0.090 0.045 / 0.107 0.045 / 0.125 0.042 / 0.135
op/14/final 0.122 / 0.150 0.084 / 0.116 0.085 / 0.124 0.089 / 0.142 0.077 / 0.165 0.013 / 0.067 0.016 / 0.069 0.007 / 0.098 0.017 / 0.111 0.027 / 0.112 0.066 / 0.106 0.050 / 0.091 0.041 / 0.110 0.044 / 0.125 0.039 / 0.137
op/14/vintage 0.147 / 0.164 0.093 / 0.121 0.091 / 0.127 0.094 / 0.144 0.081 / 0.164 0.024 / 0.072 0.029 / 0.069 0.012 / 0.096 0.021 / 0.110 0.026 / 0.113 0.086 / 0.115 0.060 / 0.093 0.049 / 0.110 0.049 / 0.126 0.043 / 0.137
op/14/exact 0.135 / 0.155 0.090 / 0.119 0.089 / 0.126 0.092 / 0.143 0.080 / 0.164 0.014 / 0.066 0.018 / 0.067 0.007 / 0.098 0.017 / 0.110 0.026 / 0.112 0.074 / 0.107 0.053 / 0.091 0.044 / 0.111 0.045 / 0.125 0.041 / 0.136

Vintage vs exact: difference in the calibrated median

Median over locations of |vintage − exact| in the calibrated median, relative to the base median, by month, one panel per ref/settle_days:

drift <- purrr::map(unique(paste(grid$ref, grid$settle_days, sep = "/")), function(cfg) {
  pick <- function(mode) {
    cals[[paste(cfg, mode, sep = "/")]]$forecasts %>%
      filter(level == 0.5, !is_burn_in, !is.na(value_base)) %>%
      select(location, horizon, reference_date, value_base, value_cal)
  }
  inner_join(pick("vintage"), pick("exact"), by = c("location", "horizon", "reference_date", "value_base"), suffix = c("_cheap", "_exact")) %>%
    mutate(config = cfg)
}) %>%
  bind_rows() %>%
  mutate(month = lubridate::floor_date(reference_date, "month")) %>%
  group_by(config, horizon, month) %>%
  summarize(rel = median(abs(value_cal_cheap - value_cal_exact) / pmax(value_base, 1)), .groups = "drop")
ggplot(drift, aes(month, 100 * rel, colour = factor(horizon), group = horizon)) +
  geom_line() +
  geom_point(size = 0.8) +
  facet_wrap(~config) +
  labs(x = NULL, y = "median |vintage − exact| / base median (%)", colour = "horizon")