AI hallucination nearly triggers US military operation | TechCrunch

A U.S. military operation directed against a Chinese ship was canceled just before execution after commanders realized the primary intelligence had been fabricated by an artificial intelligence chatbot.

The incident began when a special operations analyst utilized the AI system to aggregate open-source information with classified intelligence reports. The software incorrectly analyzed the vessel's cargo manifest, falsely claiming it transported components intended for a nuclear weapons program. This erroneous data was then formatted into an official-looking intelligence summary by the same automated tool and distributed up the military chain of command, prompting the deployment of combat aircraft.

The near-miss underscores growing anxieties about the speed at which artificial intelligence is being integrated into national defense systems. Although defense officials are eager to adopt these models to accelerate tactical decision-making and gain an advantage over global rivals, critics warn that the rapid transmission of unchecked AI hallucinations poses extreme risks. Military experts emphasize that while these automated utilities can still offer value, organizations must prioritize implementing strict human-in-the-loop safeguards and comprehensive training programs to prevent flawed machine-generated conclusions from triggering unintended international conflicts.