New research tests AI’s ability to forecast severe weather
Researchers at the National Severe Storms Laboratory and several universities in the U.S. and U.K. recently tested artificial intelligence’s ability to generate a severe weather forecast. After some prompting and prodding, the AI eventually correctly identified the risk of severe weather. However, there were enough errors in the text output to ensure human meteorologists still have a job, at least for now.
“The integration of generative AI such as GPT-4V into meteorology brings opportunity for improving forecast communication of weather hazards across languages and communities,” write the researchers. “We do not seek to replace the human forecaster; rather, we evaluate competence of the first public release of GPT-4V to augment the meteorologist’s toolbox.”
With charts supplied by Pivotal Weather, researchers first tested AI’s ability to analyze the atmosphere using data from the North American Model (NAM) and Global Forecasting System (GFS). The initial dataset was limited. “We give sufficient information to GPT-4V that a human would capture the general flow pattern from the same resources.”
Right away, researchers noticed some ‘hallucinations’ in the text analysis generated by GPT-4V. For example, the AI response referred to precipitation that did not exist in the provided data, and the terminology used to describe the weather sometimes differed from the typical language used by human meteorologists.
Interestingly, the initial prompt also asked the large language model what data it needed to improve its weather analysis. In response, GPT-4V requested additional upper-air soundings, surface dew points, sea-level pressure and humidity charts, and upper-level vorticity. To me, this suggests the large language model has some understanding of atmospheric thermodynamics.
READ MORE: AI AND THE FUTURE OF BROADCAST METEOROLOGY
For complex tasks such as this, OpenAI recommends splitting the process into small, manageable prompts and responses. This method allows humans to assess the quality of the responses and adjust additional prompts to obtain valuable responses.
After the initial analysis, researchers asked the AI to identify states with a possibility of severe weather. While the response contained vague rationale for some of the forecasts, “GPT-4V nonetheless provides five specific regions with elevated risk” that ultimately resemble the SPC forecast.”
Researchers also tested AI’s ability to produce hazard summaries in English and Spanish. Here, the AI could have performed better, especially in the Spanish translation.
“Issuing a bilingual hazard outlook is a complex task that requires competence in image recognition, understanding in time and space, communication to a lay audience, and translation abilities.”
But was the AI forecast accurate? Ultimately, a tornado developed in one of the risk areas identified by the artificial intelligence.
While some people are naturally and rightfully skeptical of artificial intelligence, I am not. In terms of the broadcasting and weather forecasting business, which I’m most familiar with, I can see the possibilities and how AI could help us do our jobs with more speed and precision. This study proves we’re not there…yet.
“In operations tasked with protecting lives and property, there is little room for error in issuing timely, correct warnings for hazards, and little time for thorough vetting of language-model output,” write the researchers. “Rigorous testing must be completed before humans can be removed further from the operational loop.”
As we say on TV, “Stay tuned.”
Story thumbnail image created by Microsoft Designer powered by DALL-E 3 artificial intelligence.
Tim Heller is an AMS Certified Broadcast Meteorologist, Talent Coach, and Weather Content Consultant. He helps local TV stations and broadcast meteorologists communicate more effectively and efficiently on-air, online, and on social media.