Recap: Hallucinations with Unshakeable Confidence
In Part 1, we observed a critical failure when we had a generative AI analyze a weather chart: it fabricated a nonexistent low-pressure system and forecast "widespread severe weather" despite the area being under a high-pressure system. Even more alarming, the AI failed to catch its own errors during a self-verification phase, doubling down with a second lie by declaring everything "perfectly consistent."
So why does this happen? Can it be fixed with better prompting, or is this a structural limitation rooted in current AI architecture? In this article, we dig into the causes from four angles.
An important note on scope: what "AI" and "limitations" mean in this article
The "limitations" discussed here refer specifically to the approach of feeding human-readable weather charts (PDFs or images) to a general-purpose multimodal LLM like ChatGPT or Claude. This is an entirely different thing from dedicated AI numerical weather prediction models (GraphCast, Pangu, AIFS, etc.), which have recently been reported as reaching practical deployment.
The latter never "read" a weather chart image at all. They learn directly from numerical data (reanalysis tensors such as ERA5)—the raw numbers before they're ever rendered for human eyes—and generate the next time step's numerical fields from the current ones. In some domains, they outperform human (operational numerical weather prediction) methods, including improving typhoon track forecasts by roughly 20% over conventional approaches.
In other words, "LLMs fail when asked to read weather chart images" and "AI numerical weather prediction has reached practical use" are not contradictory. They operate at different layers of the weather AI stack. Unless otherwise noted, everything that follows is limited to the former case (general-purpose LLMs reading rendered weather charts). This distinction becomes critically important when we reach Limitation 3.
Mapping the "Reach" of Each Limitation Up Front
The four limitations discussed in this article do not all carry the same weight or the same universality. Let's first sort out how broadly each limitation applies—which categories of AI it actually affects.
| Limitation | Nature | Reach (how broadly it applies) |
|---|---|---|
| Limitation 1: Map projection incompatibility | Vision-model specific. Stems from reading images at all. | Applies only when using an LLM to read images. An "input design" problem that can be avoided by passing numbers directly. |
| Limitation 2: Difficulty tracing contour lines | Vision-model specific. Same as above. | Applies only when using an LLM to read images. If you already have the underlying numbers, there's no line to trace. |
| Limitation 3: Lack of physical models and climatological common sense | Category-dependent. | Applies only when a general-purpose LLM reads images. Does not apply to dedicated AI numerical weather prediction models (discussed below). |
| Limitation 4: Overconfidence | Universal across model types. | Applies to all generative AI. The most fundamental limitation and the hardest to avoid. |
To preview the key takeaway: Limitations 1 and 2 stem from the choice to feed rendered charts to an LLM—they're avoidable by changing the design. Limitation 3 depends on the AI category and doesn't apply to dedicated numerical weather prediction models. And only Limitation 4 is truly universal, applying regardless of model type—making it the most troublesome of all.
Keep this difference in severity in mind as you read through each section below.
Limitation 1: Map Projection Incompatibility — Unable to Recognize a "Map" as a Map
Upper-air weather charts from the Japan Meteorological Agency are drawn in polar stereographic projection (reference: 60°N, 140°E). This is an azimuthal projection centered on the North Pole, where latitude/longitude grids appear as curves.
A human forecaster can dynamically correct for this projection's distortion in their head: "Given the curvature of this latitude line, this must be the Sea of Japan, not the interior of the continent."
AI image recognition models, however, process images as 2D pixel arrays and have no internal engine for dynamic coordinate transformation. Even if you instruct the model via prompt to "analyze this as a polar stereographic projection," no GIS (Geographic Information System) is conjured up inside it.
As a result, the AI interprets "relative position on screen" as flat, literal space, leading to spatial recognition errors such as:
- "Warping" a low-pressure system located over the continent to somewhere near Kyushu
- Confusing 40°N with 50°N
- Misjudging whether a contour line actually passes over Japan
The phenomenon we observed in Part 1—where the AI warped a continental low to near Kyushu and generated a severe weather forecast—is a textbook example of this limitation.
Note that this is a limitation inherent to vision models reading images. Flip it around: if you pass coordinate and lat/lon data as text from the start, this problem fundamentally cannot occur. Keep in mind that Limitation 1 is also an "input design" problem.
Limitation 2: Poor Resolution of Dense Visual Information — Terrible at "Following a Line"
Specialized weather charts have countless unlabeled curves layered on top of each other—height contours, isotherms, equivalent potential temperature lines, wind barbs, frontal symbols, and more.
A human forecaster can lock their gaze onto a single line and trace it from end to end. AI is extremely bad at this.
Current vision models are:
- Strong at recognizing discrete objects and text (e.g., labels like "H," "L," "1018")
- Very poor at tracking an unlabeled curve while distinguishing it from other lines in a dense environment
As a result, when analyzing frontal zones or troughs where lines cluster together, AI often gets by with a rough texture-recognition shortcut: "lots of lines here, must be a frontal zone." This is the root cause of errors in reading equivalent potential temperature gradients or locating vorticity maxima.
Like Limitation 1, this is also a problem that stems from choosing to "read an image." If you can work directly with the gridpoint numerical values that the contour lines were derived from in the first place, there's no need to visually trace any lines at all.
Limitation 3: Lack of Physical Models and "Climatological Common Sense" — But Only When a General-Purpose LLM Reads Images
Human meteorologists read weather charts with an understanding of fluid dynamics and thermodynamics. That's why they naturally notice things like:
- "The surface is under a high-pressure system (1018 hPa), but there's a strong updraft at 700 hPa? Something's wrong."
- "A midwinter-level cold air mass and a 996 hPa low over Kyushu in April? Climatologically impossible."
This kind of physical and climatological unease—a form of metacognition—comes naturally to human forecasters.
General-purpose LLMs reading rendered weather charts have none of this. A general-purpose LLM holds no 3D atmospheric grid internally and has no fluid dynamics computation engine. It merely "talks about" weather phenomena as probabilistic sequences of text.
This leads to behaviors like:
- Casually presenting physically contradictory interpretations across different chart levels
- Failing to flag seasonally extreme values as anomalous and outputting them as-is
- Being unable to verify after the fact whether its own output is consistent with the laws of physics
The description we observed in Part 1—"a cold front passing near the center of a high-pressure system"—is a logical contradiction any human would catch instantly, but a general-purpose LLM writes it out to the end without noticing the inconsistency.
An important caveat: this is not a limitation of "AI in general"
What I want to emphasize here is that Limitation 3 only holds in the specific case of feeding human-rendered weather charts to a general-purpose LLM not trained for meteorology. Extending this to the general claim that "AI doesn't understand physics" would be factually wrong.
In fact, dedicated AI numerical weather prediction models (GraphCast, Pangu, AIFS) overturn all three of the above points (based on a lecture by Dr. Tsuyoshi Sekiyama of the Meteorological Research Institute).
- Far from "lacking" a 3D grid—that grid is their very core. These models divide the Earth into a 3D grid and maintain the state of the atmosphere within it. Some implementations, like stretched-grid models that resolve the area around Japan at 5 km mesh, go even further. "No internal 3D grid" describes general-purpose LLMs; dedicated models literally hold and evolve a 3D state field through time.
- Physical parameter estimation is, in some respects, better than humans. Dedicated AI numerical weather prediction models learn physical behavior from reanalysis data, and for certain parameter estimates they are reported to outperform human (conventional numerical weather prediction) methods.
- Physics can be injected into the loss function (physics-informed). Physical consistency can be built directly into the training objective, constraining outputs so they don't stray from the laws of physics. Rather than hoping physics will emerge after the fact, it can be woven into the design from the start.
- They're not black boxes either. Techniques exist to inspect their internal states, allowing us to trace, to some degree, what the model is basing its forecast on. The image of an "opaque box whose internals are unknowable" doesn't necessarily hold.
Summary of Limitation 3: The criticisms—"no physical model / no 3D grid / can't detect contradictions / black box"—apply only when a general-purpose LLM reads a rendered weather chart. They do not apply to dedicated AI numerical weather prediction models, which do learn physics. Since this article is about the former, Limitation 3 is a real wall here—but it must not be recast as "a fundamental physical limitation of AI in general." Conflating the two categories leads to the false generalization that "AI is useless for weather."
Limitation 4: Overconfidence
Technically, Limitations 1–3 might look like the main culprits—but the most dangerous in terms of real-world harm, and the one that applies universally regardless of model type, is this one. Limitations 1 and 2 can be avoided through input design; Limitation 3 is category-dependent. Limitation 4 offers no such escape.
Large language models are trained and tuned to generate responses that sound plausible, fluent, and authoritative. Even when internal confidence is only 10%, the output gets converted into sentences like:
- "This will be…"
- "It is expected that…"
- "We were able to verify that…"
—language that projects 100% certainty.
This makes it impossible for users to tell "how much the AI actually read correctly versus how much it was guessing." The output we observed in Part 1—"we were able to verify that everything is perfectly consistent"—was the result of a shaky internal process being overwritten by a tone of absolute confidence.
What makes hallucinations most dangerous is not the factual error itself, but the fact that it is presented with a confident tone.
A Practical Question: What About Color-Coded Surface Charts?
This raises a natural question:
ASAS and FSAS charts include text labels and are color-coded. If you explicitly instruct the model to analyze them as polar stereographic projections, couldn't surface charts at least reach a practical level of accuracy?
This is half right—and half still limited.
Where improvement is realistic:
- Extraction accuracy for elements with strong contrast and accompanying text—like "red line (warm front)," "blue line (cold front)," and "central pressure values"—will definitely improve.
- Recognition errors in rough geographic placement—like "the low is north of Hokkaido"—will decrease substantially.
Limitations that remain:
- Since no dynamic GIS engine is generated inside the AI, precise coordinate transformation of latitude and longitude remains impossible.
- Quantitatively reading pressure gradients from the number and spacing of isobars remains a weak point.
In short, sticking to color-coded surface charts with rich text labels raises the odds of avoiding catastrophic spatial warping. That's a valid risk-mitigation strategy.
Conversely, it seems wise to simply accept that visual analysis of dense, monochrome upper-air charts (FXJP854, FXFE502, etc.) is currently not feasible.
The Division-of-Labor Mindset — What to Delegate to AI, What to Keep for Humans
Given these limitations, a natural design philosophy emerges:
Stop expecting AI to be a "perfect analyst," and instead position it as an "assistant that extracts data and presents theoretical checklists."
Concretely:
| Task | Assigned to | Reason |
|---|---|---|
| Extracting text information (central pressure, typhoon parameters) | AI | OCR-like processing is a strength |
| Rough geographic layout recognition from color surface charts | AI | High contrast makes this feasible |
| Articulating "what to check next" | AI | Text generation is its home turf |
| Contour line tracing on upper-air charts | Human | Vision model limitation (Limitation 2) |
| Final judgment on physical consistency across levels | Human | Difficult for LLMs reading rendered charts (Limitation 3) |
| Detecting climatological anomalies | Human | Requires metacognition |
In Part 3, we'll translate this philosophy into concrete prompt design and service implementation.
Addendum: Can Structured Prompts Eliminate Hallucinations?
Based on experience trying various "structured prompts for accurate chart analysis," the honest answer is: overcoming Limitations 1–4 through prompt engineering alone is currently impossible.
That said, it is possible to nudge the model toward hallucinating less. Three key levers:
- Mandate "no guessing" and allow "unreadable" as a valid output — explicitly permit the response "Unreadable" for regions the model can't parse.
- Force the AI to metacognize — have it declare upfront: "This region of the chart has dense lines; my confidence here is low."
- Make physical and climatological contradiction-detection the central task — frame the job as "checking data for inconsistencies," not "composing a forecast scenario."
Prompt designs that incorporate these elements—and how to handle the risks that remain even then (including user-facing disclaimers)—will be covered in detail in Part 3.
Summary
The limitations that current generative AI carries when used to read rendered weather charts are structural—they go beyond what prompt engineering can fix. That said, how broadly each limitation applies varies.
- Limitation 1: Cannot dynamically correct for projection distortion; misidentifies geographic locations. (Vision-model specific. Avoidable by passing numbers directly.)
- Limitation 2: Cannot trace dense contour lines; falls back on texture recognition. (Vision-model specific. Same as above.)
- Limitation 3: Lacks physical models and climatological common sense; cannot detect contradictions in its own output. (Applies only when a general-purpose LLM reads images. Dedicated AI numerical weather prediction models learn physics and are not subject to this.)
- Limitation 4: Always outputs with confident tone regardless of internal certainty. (Universal across model types. The most fundamental limitation.)
It would be a mistake to optimistically assume "the next generation of models will automatically solve all this"—but it would equally be an overreach to declare "this is fundamentally impossible and the structure won't change for years." In fact, the field of dedicated AI numerical weather prediction has a track record of advancing faster than expected: ECMWF's AIFS reached operational deployment sooner than most experts predicted, for instance. AI weather technology has a precedent of outpacing forecasts. Multimodal LLM chart-reading capability may not be an exception.
So this article's position is: "Currently difficult. But don't underestimate the pace of progress—revisit this periodically." That said, the practical conclusion stands firm: for now, a design that doesn't let AI operate autonomously is essential, and the responsibility of catching the final lie must never leave human hands.
Next time, we'll look at how to actually integrate AI into a real service given these limitations—specifically, the "co-pilot" design for a weather app like Tenkiz Port.
Continue → Part 3: The Path to Practical Use — Designing AI as a "Co-Pilot"