The Big Macro Nowcasting Ranking

The objective of this page is to maintain and populate the ranking below, with newly introduced statistical models in an effort to provide a continuously updated comprehensive comparison of methodologies for nowcasting and forecasting macroeconomic activity. The model comparison is based on a set of (pseudo) real-time vintages, formed using a rich standardized set of variables at mixed-frequencies.

The performance evaluation currently covers US real GDP growth rate (QoQ%), comparing 20 distinct methodologies (including 14 Machine Learning algorithms, as well as standard econometric techniques and workhorse benchmarks), which are combined with three data transformations (D1,D2,D3) for taking into account the mixed-frequency dimension. This results into a ranking containing a total of 85 specifications.

Dataset

The dataset consists of 87 quarterly, and 171 monthly (including 32 financial market) indicators. The indicators correspond to the series found in the FRED-MD and -QD datasets and were downloaded at their original (i.e. highest-available) sampling frequency, whenever possible. Publication release delays were inferred from the metadata in the FRED database (https://fred.stlouisfed.org) and then applied to each series to mimic the ragged-edge structure the forecaster would face in reality. The out-of-sample evaluation exercise uses end-of-month vintages at a monthly periodicity, assuming that economic activity is monitored in real-time by updating the projections at the end of every month.

A list of the indicators that compose the mixed-frequency dataset, can be accessed here. ‘SeriesID’ refers to the FRED mnemonic, while the release delay (‘RDelay’) measures the approximate number of days it takes for the respective indicator to be released after the closing of the reference month or quarter. The monthly mixed-frequency vintages used in the POOS model evaluation can be downloaded here. The full set consists of 376 end-of-month vintages spanning the period 19900131-20210430, including monthly and quarterly unbalanced panels whose ragged edge has been imposed by applying the inferred publication delays.

List of forecasting models

List of methods for treating mixed-frequencies

The Ranking: Real-time model evaluation

The table reports RMSE’s relative to the AR(1) benchmark for the n-quarters ahead prediction, with n=0 reflecting the nowcast. The models have been ranked wrt the last column which corresponds to the average relative RMSE over all 5 horizons. The real-time POOS evaluation is based on 219 monthly vintages over the period Jan-2003 to Mar-2021. D1 denotes single-frequency information set; D2 Unrestricted (U-MIDAS) lag polynomials; and D3 Legendre-aggregated mixed-frequency information set. The table ranks specifications coming from the different combinations of ML models with all the transformations D1, D2 and D3 plus their factor-only counterparts. Models with an acronym ending in ‘F’ contain only factors on the RHS.

The graph provides a visualization of the horse race. Furthermore, it adds a second crucial metric for measuring comparative performance, the MAE. The axes show the relative error measures (RMSE and MAE) averaged over all 5 horizons. Candidate models that are closest to the origin (-bottom left) are the best performers.