# Shared by the experiment notebooks: hub submissions, finalized truth for
# scoring, NHSN vintages for learning, and population for rates. The disease
# comes from `params$disease` (default flu).
suppressPackageStartupMessages(source(here::here("R/load_all.R")))
disease <- params$disease %||% "flu"
archive <- get_nhsn_data_archive(disease)
truth <- nhsn_read_truth(disease, archive)
vintages <- nhsn_read_vintages(disease, archive)
hub_target <- c(flu = HUB_FLU_TARGET, covid = HUB_COVID_TARGET)[[disease]]
hub_fc <- suppressMessages(hub_read_forecasts(hub_dir = here::here(params$hub_dir), target = hub_target))
population <- get_population_data() %>%
  distinct(state_code, .keep_all = TRUE) %>%
  filter(state_code %in% unique(hub_fc$location)) %>%
  select(location = state_code, population)
rate_scales <- population %>% transmute(location, from = as.Date("2000-01-01"), scale = population / 1e5)
LIVE <- c("2024-2025", "2025-2026")
run <- function(...) cal_run_cached(hub_fc, truth, ..., vintages = vintages, workers = params$workers)

Question: how do the two finalists, REF-op and sqrt constant 0.018, do on flu and covid, on the submitted ensemble and on the clean replay?

Part 1 runs them on both diseases and both bases, h0–h3, on identical rows. Part 1 also checks whether the finalists rank the same way on both bases. Part 2 is the covid ensemble at h−1–h3.

Part 1: both diseases, both bases

Setup:

  • Bases: the submitted CMU-TimeSeries ensemble (hub) and the clean windowed_seasonal replay, on the (round, location, horizon, level) rows both have, h0–h3 (the ensemble’s h−1 is a different model).
  • Learning truth: exact (re-run every round on the store’s NHSN vintages), one week of extra delay. Scores against the store’s finalized NHSN, states only, the two live seasons (August–July).
  • Configs: REF-op warm (flu prod) and cold (covid prod); sqrt constant 0.018 per 100k (E16), cold and warm (slow_init = "burn_in_quantile"). Warm runs see a 2023-24 burn-in season: for flu the rows both bases have (hub submissions and the HHS-stitched backfill, as E07); for covid only the replay’s backfill, since covid submissions start 2024-11-23. So the covid ensemble has no warm runs. Cold runs see the live seasons only.
source(here::here("scripts/calibration/calibration_ws_replay.R"))
FIN <- purrr::map(c(flu = "flu", covid = "covid"), \(d) fin_runs(d, workers = params$workers))

Per disease, a month view for each base. The ensemble and replay rows are the same (round, location, horizon, level) cells, so the two bases’ tables compare like for like.

for (d in names(FIN)) {
  cat("\n\n", cal_h(2), " ", d, "\n\n", sep = "")
  for (b in c("ensemble", "replay")) {
    mv <- cal_month_view(FIN[[d]]$cals[startsWith(names(FIN[[d]]$cals), paste0(b, ", "))], seasons = LIVE)
    cal_month_section(paste(d, b), level = 3L)
  }
}

flu

flu ensemble

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
ensemble, REF-op warm +3.7 +2.9 +2.8 +4.3 +4.3 +0.1 -0.7 -0.9 +3.9 +1.9 +1.5 +2.3
ensemble, REF-op cold +2.3 +1.0 -0.3 -1.1 +3.9 +0.2 -0.4 -0.0 +2.8 +0.7 -0.3 -0.7
ensemble, sqrt 0.018 +1.6 +0.8 +0.3 +0.0 +4.0 +0.1 -0.3 -0.1 +2.4 +0.6 +0.1 -0.0
ensemble, sqrt 0.018 warm +4.4 +4.0 +4.9 +7.9 +4.4 -0.2 -0.9 -1.7 +4.4 +2.5 +2.7 +4.3
ensemble, sqrt 0.018 warm + leak +4.2 +3.9 +5.0 +8.1 +4.5 +0.7 -0.0 -0.8 +4.3 +2.8 +3.0 +4.8 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.131 / 0.146 0.135 / 0.152 0.146 / 0.169 0.159 / 0.190 0.078 / 0.089 0.055 / 0.098 0.050 / 0.108 0.046 / 0.117 0.104 / 0.116 0.094 / 0.123 0.097 / 0.137 0.101 / 0.151
ensemble, REF-op warm 0.084 / 0.116 0.081 / 0.121 0.085 / 0.138 0.072 / 0.163 0.012 / 0.067 0.008 / 0.100 0.018 / 0.111 0.030 / 0.116 0.047 / 0.090 0.036 / 0.110 0.039 / 0.124 0.035 / 0.138
ensemble, REF-op cold 0.104 / 0.126 0.107 / 0.134 0.116 / 0.156 0.116 / 0.182 0.037 / 0.072 0.022 / 0.093 0.021 / 0.106 0.022 / 0.103 0.070 / 0.097 0.064 / 0.112 0.067 / 0.129 0.067 / 0.140
ensemble, sqrt 0.018 0.108 / 0.128 0.115 / 0.138 0.122 / 0.155 0.130 / 0.176 0.023 / 0.075 0.022 / 0.101 0.018 / 0.109 0.018 / 0.119 0.051 / 0.100 0.049 / 0.118 0.054 / 0.130 0.057 / 0.146
ensemble, sqrt 0.018 warm 0.084 / 0.115 0.082 / 0.122 0.084 / 0.139 0.075 / 0.159 0.024 / 0.084 0.041 / 0.116 0.043 / 0.128 0.055 / 0.136 0.030 / 0.099 0.020 / 0.119 0.025 / 0.133 0.023 / 0.147
ensemble, sqrt 0.018 warm + leak 0.094 / 0.122 0.093 / 0.127 0.096 / 0.144 0.089 / 0.162 0.025 / 0.065 0.013 / 0.091 0.024 / 0.109 0.032 / 0.121 0.059 / 0.092 0.046 / 0.108 0.047 / 0.126 0.043 / 0.140

flu replay

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
replay, REF-op warm +3.2 +1.9 +1.4 +1.7 +2.2 -0.1 -0.0 +0.3 +2.8 +1.1 +0.8 +1.2
replay, REF-op cold +2.2 +0.9 -0.0 -0.8 +1.8 +0.1 +0.3 +0.9 +2.0 +0.6 +0.1 -0.2
replay, sqrt 0.018 +1.7 +0.6 +0.3 +0.1 +3.0 +1.0 +1.1 +1.5 +2.2 +0.8 +0.6 +0.6
replay, sqrt 0.018 warm +3.4 +2.0 +2.0 +3.5 +3.3 +0.9 +0.8 +1.3 +3.4 +1.6 +1.5 +2.7
replay, sqrt 0.018 warm + leak +3.5 +2.3 +2.4 +4.2 +2.8 +0.8 +0.9 +1.3 +3.3 +1.7 +1.8 +3.1 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.127 / 0.130 0.129 / 0.144 0.148 / 0.179 0.169 / 0.211 0.090 / 0.097 0.081 / 0.112 0.084 / 0.130 0.086 / 0.146 0.108 / 0.112 0.105 / 0.127 0.115 / 0.153 0.127 / 0.177
replay, REF-op warm 0.087 / 0.116 0.081 / 0.134 0.082 / 0.175 0.066 / 0.208 0.037 / 0.078 0.033 / 0.111 0.030 / 0.128 0.027 / 0.145 0.061 / 0.096 0.051 / 0.121 0.047 / 0.150 0.037 / 0.174
replay, REF-op cold 0.106 / 0.121 0.107 / 0.140 0.117 / 0.175 0.124 / 0.213 0.057 / 0.078 0.050 / 0.100 0.049 / 0.121 0.047 / 0.127 0.081 / 0.098 0.077 / 0.119 0.082 / 0.146 0.085 / 0.167
replay, sqrt 0.018 0.099 / 0.115 0.104 / 0.136 0.120 / 0.170 0.137 / 0.204 0.021 / 0.075 0.023 / 0.098 0.024 / 0.119 0.022 / 0.138 0.059 / 0.094 0.062 / 0.116 0.071 / 0.143 0.078 / 0.169
replay, sqrt 0.018 warm 0.077 / 0.115 0.073 / 0.131 0.072 / 0.172 0.058 / 0.216 0.018 / 0.083 0.026 / 0.120 0.031 / 0.152 0.049 / 0.174 0.041 / 0.098 0.035 / 0.125 0.033 / 0.162 0.037 / 0.194
replay, sqrt 0.018 warm + leak 0.094 / 0.118 0.089 / 0.130 0.091 / 0.170 0.077 / 0.204 0.049 / 0.083 0.041 / 0.114 0.036 / 0.139 0.041 / 0.160 0.071 / 0.099 0.058 / 0.121 0.052 / 0.153 0.042 / 0.181

covid

covid ensemble

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
ensemble, REF-op cold +1.0 +1.6 +2.0 +1.4 +0.7 -2.0 -2.6 -2.9 +0.8 -0.1 -0.3 -0.8
ensemble, sqrt 0.018 +2.0 +1.2 +0.7 +0.2 -2.0 -6.9 -8.2 -8.3 +0.1 -2.7 -3.7 -4.2 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.089 / 0.117 0.078 / 0.132 0.075 / 0.143 0.072 / 0.153 0.049 / 0.091 0.031 / 0.120 0.029 / 0.142 0.051 / 0.159 0.063 / 0.102 0.042 / 0.125 0.031 / 0.143 0.024 / 0.157
ensemble, REF-op cold 0.082 / 0.109 0.079 / 0.130 0.080 / 0.144 0.083 / 0.157 0.033 / 0.087 0.025 / 0.124 0.023 / 0.148 0.032 / 0.165 0.050 / 0.096 0.040 / 0.126 0.035 / 0.147 0.031 / 0.161
ensemble, sqrt 0.018 0.076 / 0.104 0.079 / 0.131 0.081 / 0.147 0.084 / 0.161 0.017 / 0.081 0.025 / 0.120 0.035 / 0.146 0.051 / 0.164 0.027 / 0.091 0.024 / 0.125 0.023 / 0.147 0.021 / 0.163

covid replay

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
replay, REF-op warm -0.5 -2.3 -1.2 +1.4 +0.2 -1.2 +0.3 +4.1 -0.1 -1.8 -0.4 +2.8
replay, REF-op cold +1.9 +0.3 +0.0 +0.1 +0.6 -2.4 -3.2 -3.1 +1.3 -1.1 -1.7 -1.6
replay, sqrt 0.018 +2.2 -0.2 -0.8 -0.8 -0.9 -3.3 -2.1 +0.9 +0.7 -1.8 -1.5 +0.1
replay, sqrt 0.018 warm -1.3 -5.1 -4.6 -2.8 -0.5 -1.7 +0.8 +5.0 -0.9 -3.4 -1.8 +1.3
replay, sqrt 0.018 warm + leak -3.2 -6.8 -6.0 -4.0 -1.1 -2.6 -0.7 +2.2 -2.2 -4.7 -3.2 -0.8 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
base 0.080 / 0.081 0.030 / 0.082 0.013 / 0.109 0.048 / 0.131 0.049 / 0.103 0.032 / 0.119 0.050 / 0.147 0.087 / 0.177 0.057 / 0.094 0.023 / 0.103 0.034 / 0.131 0.070 / 0.157
replay, REF-op warm 0.104 / 0.104 0.110 / 0.120 0.116 / 0.135 0.125 / 0.155 0.052 / 0.107 0.067 / 0.133 0.078 / 0.156 0.090 / 0.176 0.073 / 0.106 0.084 / 0.128 0.093 / 0.147 0.104 / 0.167
replay, REF-op cold 0.068 / 0.071 0.035 / 0.089 0.014 / 0.115 0.023 / 0.137 0.033 / 0.095 0.026 / 0.122 0.031 / 0.147 0.052 / 0.175 0.042 / 0.085 0.022 / 0.108 0.017 / 0.133 0.039 / 0.159
replay, sqrt 0.018 0.060 / 0.065 0.029 / 0.089 0.008 / 0.114 0.037 / 0.135 0.025 / 0.092 0.025 / 0.125 0.032 / 0.147 0.049 / 0.172 0.025 / 0.080 0.015 / 0.110 0.021 / 0.133 0.043 / 0.156
replay, sqrt 0.018 warm 0.091 / 0.094 0.106 / 0.116 0.117 / 0.135 0.136 / 0.163 0.022 / 0.092 0.025 / 0.122 0.046 / 0.150 0.076 / 0.181 0.038 / 0.093 0.056 / 0.120 0.075 / 0.144 0.100 / 0.173
replay, sqrt 0.018 warm + leak 0.111 / 0.110 0.120 / 0.128 0.129 / 0.142 0.145 / 0.167 0.068 / 0.117 0.093 / 0.149 0.114 / 0.175 0.146 / 0.208 0.085 / 0.114 0.103 / 0.140 0.120 / 0.161 0.145 / 0.191

WIS by season with bootstrap intervals

Every cell carries a 90% interval from a round bootstrap: the season’s rounds are resampled with replacement 1000 times and the WIS reduction recomputed on each draw (cal_wis_boot()), the same draws for every config. The paired-gap tables give the finalist’s gap to prod’s config on each base with its paired interval. The round is the resampling unit, which ignores dependence between neighbouring rounds, so the intervals are if anything too narrow.

FIN_PAIRS <- list(
  c("sqrt 0.018 warm + leak", "REF-op warm"),
  c("sqrt 0.018 warm + leak", "sqrt 0.018 warm"),
  c("sqrt 0.018", "REF-op cold")
)
for (d in names(FIN)) {
  cat("\n\n", cal_h(3), " ", d, "\n\n", sep = "")
  bt <- cal_wis_boot(FIN[[d]]$cals, seasons = LIVE)
  cal_wis_boot_table(bt) %>%
    knitr::kable(caption = paste0(d, ": ", cal_wis_boot_caption(bt))) %>%
    print()
  gaps <- purrr::map(c("ensemble", "replay"), function(b) {
    purrr::map(FIN_PAIRS, function(pr) {
      a <- paste(b, pr[1], sep = ", "); bb <- paste(b, pr[2], sep = ", ")
      if (!all(c(a, bb) %in% names(FIN[[d]]$cals))) return(NULL)
      cal_wis_boot_diff(bt, a, bb) %>% mutate(base = b, pair = paste(pr[1], "−", pr[2]), .before = 1)
    }) %>% bind_rows()
  }) %>% bind_rows()
  if (nrow(gaps) > 0) {
    gaps %>%
      transmute(base, pair, col = paste(season, h), cell = sprintf("%+.1f [%+.1f, %+.1f]", diff, lo, hi)) %>%
      tidyr::pivot_wider(names_from = col, values_from = cell) %>%
      knitr::kable(caption = paste0(d, ": gap in WIS reduction % (first − second) with its 90% paired bootstrap interval")) %>%
      print()
  }
}

flu

flu: WIS reduction % (positive = calibrated WIS lower than base), states only, with a 90% interval from 1000 round bootstrap draws (rounds resampled within each season; both pools the seasons’ draws).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
ensemble, REF-op warm +3.7 [+2.0, +5.1] +2.9 [+0.7, +4.5] +2.8 [-0.3, +4.7] +4.3 [-0.2, +6.8] +4.3 [+0.3, +7.5] +0.1 [-5.0, +4.0] -0.7 [-7.0, +3.6] -0.9 [-7.8, +3.7] +3.9 [+2.3, +5.3] +1.9 [-0.5, +3.6] +1.5 [-1.4, +3.6] +2.3 [-1.4, +5.0]
ensemble, REF-op cold +2.3 [+0.8, +3.7] +1.0 [-0.4, +2.4] -0.3 [-1.9, +0.8] -1.1 [-3.1, +0.1] +3.9 [+0.9, +6.5] +0.2 [-3.4, +3.0] -0.4 [-4.4, +2.5] -0.0 [-3.2, +2.3] +2.8 [+1.4, +4.1] +0.7 [-0.8, +2.1] -0.3 [-1.9, +1.0] -0.7 [-2.2, +0.5]
ensemble, sqrt 0.018 +1.6 [+0.9, +2.6] +0.8 [+0.3, +1.5] +0.3 [-0.1, +0.8] +0.0 [-0.4, +0.4] +4.0 [-1.2, +8.2] +0.1 [-4.4, +3.6] -0.3 [-4.6, +2.8] -0.1 [-3.7, +2.8] +2.4 [+0.8, +4.0] +0.6 [-1.0, +2.0] +0.1 [-1.4, +1.4] -0.0 [-1.3, +1.2]
ensemble, sqrt 0.018 warm +4.4 [+2.7, +5.8] +4.0 [+1.7, +5.6] +4.9 [+1.6, +7.1] +7.9 [+2.7, +10.7] +4.4 [-2.3, +9.4] -0.2 [-7.6, +5.2] -0.9 [-9.6, +4.7] -1.7 [-12.3, +5.6] +4.4 [+2.3, +6.3] +2.5 [-0.6, +4.8] +2.7 [-1.0, +5.5] +4.3 [-1.0, +7.8]
ensemble, sqrt 0.018 warm + leak +4.2 [+2.4, +5.6] +3.9 [+1.5, +5.5] +5.0 [+1.6, +7.1] +8.1 [+3.1, +10.8] +4.5 [+0.8, +7.4] +0.7 [-4.1, +4.3] -0.0 [-5.9, +3.9] -0.8 [-9.2, +5.1] +4.3 [+2.7, +5.6] +2.8 [+0.4, +4.5] +3.0 [+0.0, +5.4] +4.8 [+0.1, +7.8]
replay, REF-op warm +3.2 [+1.6, +4.6] +1.9 [+0.1, +3.6] +1.4 [-0.9, +3.2] +1.7 [-1.6, +3.8] +2.2 [-0.9, +5.6] -0.1 [-3.5, +2.9] -0.0 [-3.5, +3.0] +0.3 [-3.7, +3.2] +2.8 [+1.3, +4.3] +1.1 [-0.7, +2.7] +0.8 [-1.2, +2.5] +1.2 [-1.3, +3.0]
replay, REF-op cold +2.2 [+0.5, +3.6] +0.9 [-1.1, +2.5] -0.0 [-2.5, +1.6] -0.8 [-3.7, +0.9] +1.8 [-0.8, +4.1] +0.1 [-2.7, +2.4] +0.3 [-2.5, +2.5] +0.9 [-1.6, +2.8] +2.0 [+0.6, +3.3] +0.6 [-1.0, +2.0] +0.1 [-1.7, +1.4] -0.2 [-2.0, +1.1]
replay, sqrt 0.018 +1.7 [+0.6, +2.7] +0.6 [-0.3, +1.4] +0.3 [-0.6, +0.9] +0.1 [-0.7, +0.6] +3.0 [-0.9, +6.6] +1.0 [-2.1, +3.5] +1.1 [-1.9, +3.3] +1.5 [-1.4, +3.6] +2.2 [+0.5, +3.7] +0.8 [-0.6, +1.9] +0.6 [-0.7, +1.6] +0.6 [-0.5, +1.5]
replay, sqrt 0.018 warm +3.4 [+1.9, +4.8] +2.0 [+0.6, +3.5] +2.0 [+0.2, +3.6] +3.5 [+1.0, +5.3] +3.3 [-1.0, +7.6] +0.9 [-2.7, +4.3] +0.8 [-3.3, +4.4] +1.3 [-4.6, +5.3] +3.4 [+1.4, +5.2] +1.6 [-0.2, +3.2] +1.5 [-0.5, +3.3] +2.7 [+0.2, +4.7]
replay, sqrt 0.018 warm + leak +3.5 [+2.2, +4.8] +2.3 [+1.1, +3.8] +2.4 [+1.0, +4.0] +4.2 [+2.1, +5.8] +2.8 [+0.3, +6.2] +0.8 [-1.6, +3.7] +0.9 [-2.0, +4.0] +1.3 [-2.9, +4.6] +3.3 [+2.0, +4.7] +1.7 [+0.4, +3.1] +1.8 [+0.4, +3.4] +3.1 [+1.1, +4.8]
flu: gap in WIS reduction % (first − second) with its 90% paired bootstrap interval
base pair 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
ensemble sqrt 0.018 warm + leak − REF-op warm +0.5 [-0.1, +0.9] +1.0 [+0.2, +1.6] +2.1 [+1.2, +2.8] +3.8 [+2.7, +4.6] +0.2 [-0.2, +0.7] +0.7 [-0.1, +1.3] +0.7 [-0.2, +1.4] +0.2 [-2.2, +1.7] +0.4 [+0.0, +0.7] +0.9 [+0.3, +1.3] +1.6 [+0.9, +2.1] +2.4 [+1.4, +3.2]
ensemble sqrt 0.018 warm + leak − sqrt 0.018 warm -0.2 [-0.5, -0.1] -0.1 [-0.3, -0.0] +0.0 [-0.1, +0.1] +0.2 [+0.1, +0.4] +0.1 [-2.1, +3.5] +1.0 [-0.9, +3.7] +0.9 [-0.8, +3.5] +0.9 [-0.6, +3.1] -0.1 [-0.9, +0.7] +0.3 [-0.4, +1.1] +0.4 [-0.3, +1.2] +0.5 [-0.1, +1.2]
ensemble sqrt 0.018 − REF-op cold -0.7 [-1.5, +0.3] -0.2 [-1.1, +0.8] +0.6 [-0.2, +1.9] +1.1 [+0.1, +2.7] +0.1 [-2.4, +1.9] -0.1 [-1.5, +0.9] +0.1 [-0.6, +0.7] -0.1 [-0.9, +0.6] -0.4 [-1.4, +0.5] -0.2 [-0.9, +0.6] +0.4 [-0.1, +1.1] +0.7 [+0.0, +1.5]
replay sqrt 0.018 warm + leak − REF-op warm +0.3 [-0.3, +1.2] +0.5 [-0.4, +1.9] +1.0 [-0.1, +2.8] +2.5 [+1.4, +4.1] +0.6 [-0.3, +1.9] +0.9 [-0.3, +2.8] +0.9 [-0.1, +2.5] +1.0 [+0.3, +1.8] +0.4 [-0.1, +1.2] +0.6 [-0.1, +1.7] +0.9 [+0.1, +2.1] +1.9 [+1.2, +2.8]
replay sqrt 0.018 warm + leak − sqrt 0.018 warm +0.2 [-0.2, +0.7] +0.4 [+0.0, +0.8] +0.5 [+0.1, +1.1] +0.7 [+0.3, +1.3] -0.5 [-2.2, +1.9] -0.1 [-1.4, +1.8] +0.0 [-1.1, +2.0] -0.0 [-1.1, +1.9] -0.1 [-0.9, +0.8] +0.2 [-0.4, +0.9] +0.3 [-0.3, +1.0] +0.4 [-0.2, +1.2]
replay sqrt 0.018 − REF-op cold -0.5 [-1.0, +0.1] -0.3 [-1.2, +0.8] +0.3 [-0.8, +1.9] +0.9 [-0.3, +3.0] +1.2 [-0.4, +2.5] +0.9 [+0.3, +1.4] +0.8 [+0.4, +1.1] +0.6 [-0.2, +1.1] +0.2 [-0.6, +0.9] +0.2 [-0.5, +0.9] +0.5 [-0.2, +1.4] +0.8 [-0.1, +1.9]

covid

covid: WIS reduction % (positive = calibrated WIS lower than base), states only, with a 90% interval from 1000 round bootstrap draws (rounds resampled within each season; both pools the seasons’ draws).
config 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
ensemble, REF-op cold +1.0 [-0.5, +2.7] +1.6 [+0.1, +3.2] +2.0 [+0.3, +3.5] +1.4 [-0.2, +2.9] +0.7 [-1.5, +3.0] -2.0 [-4.5, +0.4] -2.6 [-5.3, +0.4] -2.9 [-5.8, +0.4] +0.8 [-0.5, +2.3] -0.1 [-1.6, +1.4] -0.3 [-2.1, +1.4] -0.8 [-2.8, +1.0]
ensemble, sqrt 0.018 +2.0 [+0.8, +3.6] +1.2 [+0.1, +2.4] +0.7 [-0.2, +1.6] +0.2 [-0.6, +0.9] -2.0 [-6.9, +3.5] -6.9 [-11.3, -2.5] -8.2 [-11.7, -4.5] -8.3 [-11.2, -5.3] +0.1 [-2.5, +2.8] -2.7 [-5.2, -0.5] -3.7 [-6.0, -1.7] -4.2 [-6.2, -2.4]
replay, REF-op warm -0.5 [-1.4, +0.5] -2.3 [-5.4, +0.9] -1.2 [-7.0, +4.6] +1.4 [-6.6, +8.8] +0.2 [-1.0, +1.5] -1.2 [-3.8, +1.0] +0.3 [-4.7, +4.7] +4.1 [-3.2, +10.4] -0.1 [-0.9, +0.7] -1.8 [-3.8, +0.2] -0.4 [-4.3, +3.0] +2.8 [-2.6, +7.7]
replay, REF-op cold +1.9 [+0.9, +2.9] +0.3 [-0.6, +1.2] +0.0 [-1.1, +1.2] +0.1 [-1.6, +1.9] +0.6 [-1.6, +2.9] -2.4 [-4.4, -0.3] -3.2 [-4.8, -1.4] -3.1 [-5.0, -0.8] +1.3 [+0.0, +2.5] -1.1 [-2.2, +0.1] -1.7 [-2.7, -0.5] -1.6 [-3.0, -0.1]
replay, sqrt 0.018 +2.2 [+1.2, +3.3] -0.2 [-0.7, +0.4] -0.8 [-1.7, -0.0] -0.8 [-2.2, +0.3] -0.9 [-5.1, +3.7] -3.3 [-5.5, -1.0] -2.1 [-3.1, -0.8] +0.9 [-1.3, +3.0] +0.7 [-1.6, +3.0] -1.8 [-3.0, -0.6] -1.5 [-2.2, -0.7] +0.1 [-1.3, +1.3]
replay, sqrt 0.018 warm -1.3 [-2.8, +0.5] -5.1 [-10.1, +0.3] -4.6 [-13.6, +3.9] -2.8 [-15.5, +8.9] -0.5 [-3.9, +3.3] -1.7 [-3.6, +0.1] +0.8 [-6.1, +7.0] +5.0 [-7.0, +15.6] -0.9 [-2.8, +1.3] -3.4 [-6.1, -0.7] -1.8 [-7.6, +3.3] +1.3 [-7.4, +9.1]
replay, sqrt 0.018 warm + leak -3.2 [-4.6, -1.8] -6.8 [-12.0, -1.5] -6.0 [-15.4, +2.7] -4.0 [-17.1, +7.9] -1.1 [-2.8, +0.2] -2.6 [-9.1, +3.1] -0.7 [-11.7, +8.9] +2.2 [-13.1, +15.4] -2.2 [-3.3, -1.1] -4.7 [-8.8, -0.8] -3.2 [-10.8, +3.3] -0.8 [-11.2, +8.3]
covid: gap in WIS reduction % (first − second) with its 90% paired bootstrap interval
base pair 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h0 both h1 both h2 both h3
ensemble sqrt 0.018 − REF-op cold +1.0 [+0.2, +1.8] -0.5 [-1.3, +0.4] -1.3 [-2.2, -0.3] -1.2 [-2.2, -0.2] -2.7 [-5.6, +0.6] -4.9 [-7.2, -2.5] -5.6 [-7.1, -3.9] -5.5 [-7.0, -3.9] -0.7 [-2.2, +0.8] -2.6 [-3.9, -1.3] -3.4 [-4.4, -2.4] -3.4 [-4.4, -2.5]
replay sqrt 0.018 warm + leak − REF-op warm -2.7 [-3.6, -1.8] -4.4 [-6.8, -2.0] -4.8 [-8.5, -1.3] -5.4 [-10.5, -0.8] -1.3 [-3.1, +0.2] -1.4 [-5.7, +2.5] -1.1 [-7.4, +4.9] -1.9 [-10.4, +6.2] -2.0 [-3.0, -1.1] -2.9 [-5.4, -0.5] -2.8 [-6.6, +0.5] -3.6 [-8.8, +1.1]
replay sqrt 0.018 warm + leak − sqrt 0.018 warm -2.0 [-3.0, -1.2] -1.7 [-2.8, -0.9] -1.3 [-2.2, -0.6] -1.2 [-2.0, -0.6] -0.6 [-5.4, +3.8] -0.9 [-6.3, +3.8] -1.6 [-6.1, +2.4] -2.9 [-6.9, +0.7] -1.3 [-3.8, +1.0] -1.3 [-4.1, +1.2] -1.4 [-3.8, +0.7] -2.1 [-4.2, -0.2]
replay sqrt 0.018 − REF-op cold +0.3 [-0.0, +0.7] -0.4 [-1.0, +0.1] -0.9 [-1.7, -0.2] -0.9 [-1.8, -0.2] -1.5 [-3.7, +1.0] -1.0 [-1.9, +0.1] +1.2 [-0.0, +2.4] +4.0 [+2.2, +5.8] -0.6 [-1.8, +0.7] -0.7 [-1.3, -0.2] +0.2 [-0.5, +0.9] +1.7 [+0.6, +2.8]

Do the finalists rank the same way on both bases?

Per disease, the candidates run on both bases (covid has no warm runs on the ensemble), ordered by mean WIS reduction % (positive = calibrated WIS lower than base) over h0–h3, separately on each base, states only. rank 1 is the largest WIS reduction; cov rank 1 is the lowest month-avg L1 coverage bias. Spearman ρ compares the two bases over all (candidate, horizon) cells of a season.

for (d in names(FIN)) {
  mv_all <- cal_month_view(FIN[[d]]$cals, seasons = LIVE)
  wis <- bind_rows(mv_all$month, mv_all$month %>% mutate(season = "both")) %>%
    group_by(config, season, horizon) %>%
    summarize(wis = 100 * (sum(wis_base) - sum(wis_cal)) / sum(wis_base), .groups = "drop")
  cells <- mv_all$cov %>%
    transmute(config, season = as.character(season), horizon, cov = mavg_cal) %>%
    inner_join(wis, by = c("config", "season", "horizon")) %>%
    tidyr::separate(config, c("base", "cand"), sep = ", ", extra = "merge")
  shared <- cells %>% distinct(base, cand) %>% count(cand) %>% filter(n == 2) %>% pull(cand)
  cells <- cells %>% filter(cand %in% shared)
  cat("\n\n", cal_h(3), " ", d, "\n\n", sep = "")
  cells %>%
    group_by(base, season, cand) %>%
    summarize(wis = mean(wis), cov = mean(cov), .groups = "drop") %>%
    group_by(base, season) %>%
    mutate(rank = rank(-wis), cov_rank = rank(cov)) %>%
    ungroup() %>%
    tidyr::pivot_wider(names_from = base, values_from = c(wis, rank, cov, cov_rank)) %>%
    arrange(season, rank_ensemble) %>%
    transmute(season, cand,
      `WIS % ensemble` = round(wis_ensemble, 2), `rank ensemble` = rank_ensemble,
      `WIS % replay` = round(wis_replay, 2), `rank replay` = rank_replay,
      `cov ensemble` = round(cov_ensemble, 3), `cov rank ensemble` = cov_rank_ensemble,
      `cov replay` = round(cov_replay, 3), `cov rank replay` = cov_rank_replay
    ) %>%
    knitr::kable(caption = "Mean over h0–h3, states only. Coverage is month-avg L1 coverage bias (lower is better)") %>%
    print()
  cells %>%
    select(season, cand, horizon, base, wis, cov) %>%
    tidyr::pivot_wider(names_from = base, values_from = c(wis, cov)) %>%
    group_by(season) %>%
    summarize(
      `ρ WIS cells` = cor(wis_ensemble, wis_replay, method = "spearman"),
      `ρ coverage cells` = cor(cov_ensemble, cov_replay, method = "spearman")
    ) %>%
    knitr::kable(digits = 2, caption = "Spearman correlation of ensemble vs replay over (candidate, horizon) cells") %>%
    print()
}

flu

Mean over h0–h3, states only. Coverage is month-avg L1 coverage bias (lower is better)
season cand WIS % ensemble rank ensemble WIS % replay rank replay cov ensemble cov rank ensemble cov replay cov rank replay
2024-2025 sqrt 0.018 warm 5.31 1 2.71 2 0.134 1 0.158 4
2024-2025 sqrt 0.018 warm + leak 5.28 2 3.13 1 0.139 3 0.155 1
2024-2025 REF-op warm 3.44 3 2.07 3 0.134 2 0.158 3
2024-2025 sqrt 0.018 0.71 4 0.66 4 0.149 4 0.156 2
2024-2025 REF-op cold 0.49 5 0.55 5 0.149 5 0.162 5
2025-2026 sqrt 0.018 warm + leak 1.12 1 1.44 3 0.097 2 0.124 4
2025-2026 sqrt 0.018 0.96 2 1.64 1 0.101 4 0.108 2
2025-2026 REF-op cold 0.94 3 0.78 4 0.094 1 0.107 1
2025-2026 REF-op warm 0.69 4 0.60 5 0.098 3 0.115 3
2025-2026 sqrt 0.018 warm 0.37 5 1.60 2 0.116 5 0.132 5
both sqrt 0.018 warm + leak 3.71 1 2.46 1 0.116 2 0.139 4
both sqrt 0.018 warm 3.46 2 2.27 2 0.124 5 0.145 5
both REF-op warm 2.40 3 1.48 3 0.115 1 0.135 3
both sqrt 0.018 0.76 4 1.05 4 0.124 4 0.130 1
both REF-op cold 0.64 5 0.63 5 0.120 3 0.133 2
Spearman correlation of ensemble vs replay over (candidate, horizon) cells
season ρ WIS cells ρ coverage cells
2024-2025 0.88 0.89
2025-2026 0.48 0.92
both 0.95 0.94

covid

Mean over h0–h3, states only. Coverage is month-avg L1 coverage bias (lower is better)
season cand WIS % ensemble rank ensemble WIS % replay rank replay cov ensemble cov rank ensemble cov replay cov rank replay
2024-2025 REF-op cold 1.48 1 0.58 1 0.135 1 0.103 2
2024-2025 sqrt 0.018 1.00 2 0.09 2 0.136 2 0.101 1
2025-2026 REF-op cold -1.68 1 -2.01 2 0.131 2 0.135 2
2025-2026 sqrt 0.018 -6.35 2 -1.34 1 0.128 1 0.134 1
both REF-op cold -0.09 1 -0.76 2 0.133 2 0.121 2
both sqrt 0.018 -2.63 2 -0.63 1 0.131 1 0.120 1
Spearman correlation of ensemble vs replay over (candidate, horizon) cells
season ρ WIS cells ρ coverage cells
2024-2025 0.60 0.95
2025-2026 0.02 0.98
both 0.48 0.93

Part 2: covid ensemble, h−1–h3

  • Disease: covid. Base forecaster: the submitted CMU-TimeSeries ensemble.
  • Learning truth: NHSN covid vintages, re-run at every round as prod does (E00). Scores against finalized covid NHSN.
  • No burn-in season (covid submissions start 2024-11-23); both trackers start from zero offsets. REF-op runs without its warm start, as in covid prod.
  • Rounds 2024-11-30 to 2026-09-19. The 2024-11-23 round is excluded (made from a bad NHSN release corrected the next day).
  • Covid is submitted year-round; per-season views use August–July seasons.
  • Horizons h−1 to h3. Scores: states only (US excluded), the two live seasons.
config tracker
REF-op cold covid prod: sqrt scale, adaptive learning rate with floor, slow term 0.003, leak 0.1; no warm start
sqrt 0.018 sqrt of the per-100k rate, constant learning rate 0.018 (E16), single term

WIS reduction % = 100 × (base − calibrated) / base, so positive = calibrated WIS lower than base. L1 coverage bias: lower is better.

no_burn_in <- list(burn_in_seasons = character(0), slow_init = NULL)
cals <- list(
  `REF-op cold` = do.call(run, c(list("op"), no_burn_in)),
  `sqrt 0.018` = do.call(run, c(list("paper", transform = "sqrt", lr = 0.018, scales = rate_scales), no_burn_in))
)
season_of <- function(d) {
  y <- as.integer(format(d, "%Y")) - (as.integer(format(d, "%m")) < 8)
  paste0(y, "-", y + 1)
}
cals <- purrr::map(cals, function(cal) {
  cal$forecasts <- cal$forecasts %>% mutate(season = season_of(reference_date))
  cal
})
LIVE <- c("2024-2025", "2025-2026")
mv <- cal_month_view(cals, seasons = LIVE, horizons = NULL)
cal_month_section("Month view", level = 2L)

Month view

Scores by calendar month of the reference date, one figure per season, states only (US excluded). Rows are the three metrics, columns the horizons:

  • WIS reduction %: 100 × (WIS_base − WIS_cal) / WIS_base within the month, so positive = calibrated WIS lower than base. Grey bars are the month’s share of the season’s base WIS, so a large swing in a month with a small bar costs little overall.
  • L1 coverage bias reduction %: 100 × (bias_base − bias_cal) / bias_base, so positive = calibrated coverage bias lower than base. The bias is computed within the month: the mean over the 23 levels of |share of outcomes at or below the quantile − level|. A season-pooled bias lets under-prediction on the ramp cancel over-prediction after the peak; the monthly one does not. When the base’s bias is already small, a small absolute change shows as a large percentage; the month-average column of the coverage table below gives the absolute values.
  • Median AE reduction %: the same as WIS reduction % for the absolute error of the 0.5 quantile, so positive = calibrated median closer to the outcome.

A month holds one to five rounds of about 52 states, and outcomes in the same round move together, so a single month’s coverage bias is noisy and sits above zero even for a calibrated forecaster. That’s why the plot shows it relative to the base.

mv_plot_month <- mv$month
if (!is.null(get0("mv_plot_cfgs"))) {
  mv_plot_month <- mv_plot_month %>% filter(config %in% mv_plot_cfgs) %>% mutate(config = droplevels(config))
}
for (s in sort(unique(mv_plot_month$season))) {
  cat("\n\n", cal_h(2), " ", s, "\n\n", sep = "")
  print(cal_month_plot(mv_plot_month, s, colours = get0("mv_cols"), linetype = get0("mv_linetype"), colour_by = get0("mv_colour_by") %||% "config"))
}

2024-2025

2025-2026

cat("\n\n", cal_h(2), " By horizon\n\n", sep = "")

By horizon

cat("WIS reduction % (positive = calibrated WIS lower than base), states only; `both` pools the seasons.\n\n")

WIS reduction % (positive = calibrated WIS lower than base), states only; both pools the seasons.

print(knitr::kable(cal_month_wis_table(mv)))
config 2024-2025 h-1 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h-1 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h-1 both h0 both h1 both h2 both h3
REF-op cold +7.5 +1.0 +1.6 +2.0 +1.4 +7.9 +0.7 -2.0 -2.6 -2.9 +7.7 +0.8 -0.1 -0.3 -0.8
sqrt 0.018 +10.7 +2.0 +1.2 +0.7 +0.2 +8.6 -2.0 -6.9 -8.2 -8.3 +9.7 +0.1 -2.7 -3.7 -4.2 
print(knitr::kable(cal_month_cov_table(mv), caption = CAL_MONTH_COV_CAPTION))
L1 coverage bias (lower is better), states only, as season-pooled / month-avg. Season-pooled scores all of a season’s rounds together. Month-avg is the mean over calendar months of the bias computed within each month, every month weighted equally (‘both’: every season-month weighted equally).
config 2024-2025 h-1 2024-2025 h0 2024-2025 h1 2024-2025 h2 2024-2025 h3 2025-2026 h-1 2025-2026 h0 2025-2026 h1 2025-2026 h2 2025-2026 h3 both h-1 both h0 both h1 both h2 both h3
base 0.159 / 0.161 0.089 / 0.117 0.078 / 0.132 0.075 / 0.143 0.072 / 0.153 0.085 / 0.139 0.049 / 0.091 0.031 / 0.120 0.029 / 0.142 0.051 / 0.159 0.103 / 0.148 0.063 / 0.102 0.042 / 0.125 0.031 / 0.143 0.024 / 0.157
REF-op cold 0.133 / 0.139 0.082 / 0.109 0.079 / 0.130 0.080 / 0.144 0.083 / 0.157 0.051 / 0.110 0.033 / 0.087 0.025 / 0.124 0.023 / 0.148 0.032 / 0.165 0.070 / 0.123 0.050 / 0.096 0.040 / 0.126 0.035 / 0.147 0.031 / 0.162
sqrt 0.018 0.106 / 0.115 0.076 / 0.104 0.079 / 0.131 0.081 / 0.147 0.084 / 0.161 0.028 / 0.093 0.017 / 0.081 0.025 / 0.120 0.035 / 0.146 0.051 / 0.165 0.031 / 0.103 0.027 / 0.091 0.024 / 0.125 0.023 / 0.147 0.021 / 0.163