Weather Forecasts From Previous Model Runs
Analysing Day-To-Day Forecast Performance
tl;dr: You now have the capability to access forecasts generated by up to 7 days old weather model updates, allowing for the analysis of day-to-day variations in forecast stability or use past forecasts to optimise machine learning models for day-ahead predictions.
The post explains:
The diminishing accuracy of forecast models as the prediction horizon extends further into the future
Examination of whether recent events like the floods in Dubai were accurately forecasted in advance
An exploration of how weather forecasts from preceding days are generated.
A Loosing Battle: Weather Model vs. Chaos Theory
Ever wondered what the weather forecast looked like for today a couple of days ago? Sometimes, forecast models hit the mark precisely, while other times, they seem indecisive, with predictions shifting daily. This uncertainty can be particularly frustrating, especially when planning activities for the upcoming weekend and the forecast remains ambiguous about rain.
Predicting large, well-established high or low pressure systems is generally more straightforward for models. These systems adhere to well-understood atmospheric physics, resulting in more accurate short-term forecasts (2-3 days). However, challenges arise when dealing with developing or transitioning systems. These include newly formed or rapidly changing low-pressure systems (storms), which pose difficulties due to their evolving nature. Consequently, the accuracy of weather forecasts for the next few days varies.
The new Open-Meteo "Previous Runs API" now allows access to forecasts from past days for direct comparisons with current forecasts. This feature is available for any location worldwide and any open-data weather model.
Let's explore how this API functions and examine some data:
The Previous Runs API offers lead-time offsets ranging from 1 to 7 days. For instance, a 1-day offset displays data forecasted 24 hours earlier. By analyzing the progression of forecasts over preceding days, we gain insights into how predictions evolve leading up to actual weather events.
Examining the most recent forecast data (day 0), alongside lead-time offsets of one and two days, reveals how precise the forecast was. For instance, the most recent data indicates a temperature of 7.2°C, whereas forecasts from one and two days earlier predicted 9.0°C and 9.1°C, respectively. Concluding, the forecast was too warm and got corrected to 7.2°C with most recent updates. However, overall, the forecast appears relatively stable.
Adding more lead-time offsets of three, four and five days old forecasts, shows larger fluctuations in the next chart. “Previous day 5“ (Lila) with 10.4°C the largest deviation.
Weather models rely on initial conditions, such as temperature, pressure, and humidity, to simulate future atmospheric states. However, due to the chaotic nature of the atmosphere, small errors in these initial conditions can amplify over time, leading to divergent forecasts. As the forecast extends further into the future, its accuracy get lower.
Weather models are updated every couple of hours with recent observations from satellites, radars, measurement stations, buoys, soundings, or aircraft. These updates aim to refine the forecast for the upcoming days, improving accuracy.
The chart below illustrates forecast skill from ECMWF IFS, AIFS, GraphCast, and Pangu-Weather models. Initially, accuracy is high, but it rapidly declines beyond the fourth day.
For the sake of completes, let’s add forecasts from 6 and 7 days ago graph. While the lead-time offset of day 6 days (teal) shows some larger deviations, the 7 day old forecast (red) is clearly unsteady, highlighting the inherent challenge of long-term forecasting.
Did Weather Models Predict Floods In Dubai?
Nevertheless, the accuracy of forecasts can vary depending on different large-scale weather systems and differs from one location to another. Forecasts in dry climates like the Saharan desert may exhibit more stability, but the recent floods in Dubai and the UAE tell a diffrent story.
By examining temperature forecasts for Dubai with various lead times of up to 7 days, it becomes evident that an exceptional event occurred.
Using the data to address the question: Did weather models anticipate significant precipitation in Dubai?
The following chart depicts precipitation forecasts from the near-realtime archived forecast (teal) and 6 days in advance (dark blue). While the distribution of precipitation varies, the forecasted amount closely matches. Remarkably, this alignment for a forecast made 6 days in advance is phenomenal!
How Is This Data Generated?
Weather models update every 1, 3, or 6 hours with the lastest observations. Each update generates a forecast spanning up to 16 days, and while subsequent updates may differ from previous ones, weather models generally provide fairly accurate predictions for the upcoming days. However, due to the inherent unpredictability of weather forecasting, each update alters the forecast to some extent.
Evaluating the performance of weather forecasts across different time horizons (1-7 days into the future) requires assessing every update of each weather model. This task demands a vast amount of data and processing power, making it difficult to analyse data on a global scale within a short timeframe. It would take days to complete such an analysis. Therefore, using this method to answer simple questions about the accuracy of past weather forecasts for specific cities is not trivial.
Open-Meteo introduces an innovative solution by storing continuous time-series data for each forecast horizon day, referred to as "Previous day N". This approach simplifies access to historical forecast data, enabling easier analysis and comparison.
Previous Day 0 corresponds to the most recent forecast update, with past data closely aligned with measurements through weather assimilation to initialize a weather model. In a model updating every 3 hours, one time-step is taken from initialization, closely reflecting measurements, while the subsequent two forecast time-steps are reasonably close. The graph below shows the specific time-steps derived from each update. In regions like North America and Europe, where models update hourly, past data aligns closely with measurements.
Continuous Time-Series of 3-Hourly Updating Weather Model For Previous Day 1, with a forecast lead-time offset of 24 hours, it doesn't entail retrieving a single weather model run from 24 hours ago. Instead, the initial 24 hours of each weather model run are disregarded. In models updating every 3 hours, time steps 24, 25, and 26 are utilized to form a continuous time-series.
Continuous Time-Series of Previous Day 1 Previous Day N: This methodology extends to subsequent days, such that for Previous Day 2, time steps 48, 49, and 50 are used. Previous day 3 = 72, 73 and 74. Previous Days 3 onwards are available for local weather models forecasting only up to 3 days into the future.
While it may appear complex at first glance, delving deeper into this data is remarkably straightforward. To compute the day-to-day variance in weather forecasts, one can easily subtract the data from Previous Day 1 from that of Day 0. By averaging this difference over the course of a month, one can obtain the average forecast error for forecasts one day ahead.
The data is generated for all open-data weather models, such as ECMWF IFS, DWD ICON, NCEP GFS, ECCC GEM, MeteoFrance Arome, and more. All weather models that provide forecasts are included within the Previous Models Runs API.
This approach has a couple of limitation though:
Only variations from 1 day to another are retained. Arguably, it would be interesting to keep also a lead-time offset of 12 or 36 hours instead of full days, the difference in variation is relatively minor. Although technically possible, it's essential to consider that each additional "previous day" increases the volume of data that must be processed and stored.
Only 3 months of data are available for a limited number of surface weather variables is kept. The data generation system was established in January 2024 and continuous storing previous model data. Despite the limited time frame, the data has already reached 4.2 TB. The objective is to preserve data spanning multiple years.
“du -sh“ says more than a thousand words. 4.2 TB for 3 months of previous weather model run data Fluctuations in data stability may arise when combining updates from multiple weather models. As depicted in the graphs above for Previous Day 7, there can be abrupt shifts in temperature, with changes of up to 15°C observed. For Previous Days 0-4, data is mostly steady.
While not flawless, this method enabled straightforward analysis of extensive weather data, which previously posed challenges and demanded innovative solutions to handle hundreds of gigabytes or even terabytes of GRIB files. With the Previous Runs API you could analyse different weather models in a spreadsheet and figure out which model was model accurate.
Furthermore, machine learning models can improve this process. By using data from previous model runs across multiple weather models, machine learning models can be trained to generate enhanced forecasts, specifically targeting improved accuracy for one or two-day-ahead forecasts. GraphCast and AIFS have demonstrated that machine learning-optimized weather models can achieve high accuracy for longer forecast horizons using this approach, learning from past weather forecast model simulations and projecting into the future. With the new Previous Runs API, this capability can now be extended to a regional level, while reducing training time and offering higher resolution.
Open-Meteo takes pride in offering this novel data access approach to researchers and the general public alike, lowering barriers to entry for using open-data weather forecasts to their fullest potential and enriching the open-source open-data community.
If you have great ideas for using this data, please share them in the comments below!
Thank you for this improvement! Very interesting to see the analysis of the Dubai floods.
Hi, Eric. To download previous models runs data, you will have to run downloaders for individual national weather providers yourself. Here is a guide: https://github.com/open-meteo/open-meteo/blob/main/docs/downloading-datasets.md. Unfortunately, due to the amount of data, the previous runs database is not published directly.