THE MISS
How big were the errors?
Scores like root mean square error measure the average size of the errors, with large misses counting more. A perfect score is 0.
Pick a scoreHOW FORECASTS ARE MARKED
Score the forecast,
not the hype.
Every forecast gets marked against what happened.
Pick a score
VERIFYagainst reality
Since 2025 the European Centre for Medium-Range Weather Forecasts has run AI forecasts in operations beside its physics-based model. Whether the AI is better depends on the score, the weather variable and what the forecast is checked against.
At a glance
- Measure the miss
- Score the spread
- Check the pattern
- Different scores, different answers
≈ 1,000× less
Energy use per forecast, AI vs physics
ECMWF says its AI ensemble can produce forecasts over 10 times faster than the physics-based system while cutting energy use by approximately 1,000 times.
ENERGY · ECMWF
up to 25%
Largest improvement over the physics ensemble
On ECMWF's scorecard, improvements reached up to 25%, with higher skill for upper-air variables. Temperature high in the atmosphere, at 100 hPa, got worse.
SKILL · ECMWF SCORECARD
TWO NUMBERS WORTH KNOWING
Faster, cheaper,
and often sharper.
One number on cost,
one on skill.
The AI model still starts from the same initial conditions as the physics model, built from around 60 million quality-controlled observations from satellites, planes, boats, buoys and ground stations.
WHICH SCORE?
Pick a score,
see what it asks.
Each verification score answers a different question about a forecast. The bar shows how well the score suits forecasts that give a range of outcomes.
Definitions from the Verification FAQ of the WMO Joint Working Group on Forecast Verification Research, and ECMWF's AIFS ENS evaluation.
THE SHORT ANSWER
Better on some scores,
worse on others.
Forecasters compare models on scorecards.
Each cell is one variable, one score, one lead time.
THE RANGE
Was the spread honest?
Ensembles run many forecasts at once. The continuous ranked probability score marks the whole range of outcomes against what was observed. Lower is better.
See the numbers
THE SCORECARD
Blue and red cells
ECMWF's AI ensemble beat its physics-based ensemble for 2-metre temperature at all lead times, but the physics model was more skilful for 10-metre wind speed.
Why AI looks smooth
FOUR QUESTIONS TO ASK
Before you believe
a better forecast.
Headlines say an AI model beats the old one. These questions come straight from how forecasters verify their models.
Which score?
An error score, a pattern score and a probability score can rank the same two models differently.
Checked against what?
Against a model analysis or against real observations from stations and weather balloons? ECMWF found the answer changed early lead times.
Which weather, how far ahead?
A model can win on 2-metre temperature and lose on 10-metre wind. Look at the variable and the lead time.
One forecast or a range?
Weather is chaotic, so forecasters prefer many runs from slightly different starts. Ask how the spread was scored.
WHY EARLY AI FORECASTS LOOKED SMOOTH
Trained on error,
blurred on detail.
ECMWF explains that first-generation AI weather models tended to produce overly smooth fields, lacking small-scale features seen in reality, because they were trained to minimise mean squared error.
Smoothing narrows the spread between ensemble members and can make probabilities unreliable. Training on a proper score, the CRPS, encourages realistic variability, and that is the approach behind ECMWF's operational AI ensemble.
Questions
Questions people ask
Are AI weather models better than physics-based ones?
On many measures, not all. ECMWF says its AI models outperform physics-based models for many measures, with gains of up to 20%, but its physics ensemble remains more skilful for some, such as 10-metre wind speed.
Do AI models still need weather observations?
Yes. ECMWF's AI forecasts start from the same initial conditions as its physics model, combining a short-term forecast with around 60 million quality-controlled observations.
Why run an ensemble of forecasts?
The Met Office explains that weather is chaotic: small differences in conditions now can grow. Running many simulations from slightly different starts is better than a single run.
Why is the AI model coarser?
ECMWF's AI ensemble currently runs at 31 km, against 9 km for the physics-based ensemble, which it says remains indispensable for high-resolution fields.
Has forecasting got more accurate?
The Met Office says its four-day forecast is now as accurate as its one-day forecast was 30 years ago.
Sources
Where the facts come from
ABOUT THIS NAME
Interested in leasing aimodelscore.com?
aimodelscore.com is part of the DoodleSale portfolio. It isn't in an auction right now, but you can ask about leasing it.