Ask a language model something it doesn't actually know, and instead of shrugging and saying "no idea," it will often just make something up, deliver it with total confidence, and move on like nothing happened. That's a hallucination, and it's basically the AI equivalent of a party guest who has never once said "I'm not sure" out loud in their entire life.
Here's the part worth understanding before you laugh at the highlights below. These models aren't looking things up in some giant fact checked encyclopedia every time you ask a question. They're predicting which word probably comes next based on patterns learned from mountains of text, which turns out to be a very different skill from actually knowing something.
Most of the time that prediction lines up with reality closely enough to be useful. Sometimes it drifts off into total fiction, and the model has no built-in alarm bell to flag the difference, because from its own perspective a confident wrong answer and a confident right one feel exactly the same to produce. It's less lying and more a machine that genuinely cannot tell the difference between remembering and improvising.
Take Air Canada, which learned this the expensive way. A customer named Jake Moffatt asked the airline's chatbot about bereavement fares after his grandmother passed away, and the bot cheerfully told him he could book his flight now and apply for the discount within 90 days. That was not the real policy.
When Air Canada refused to honor what its own chatbot had promised, the case went to a tribunal, where the airline actually tried arguing the chatbot was basically its own separate legal entity. The tribunal was not moved by that logic, and Air Canada ended up paying up. Lawyers have gotten burned too, and arguably worse. In one now infamous case, an attorney used ChatGPT to help research a legal brief, and it confidently produced several supporting court cases.
Fully invented, quotes and all, down to the internal citations. Nobody bothered to check before filing, the fabricated cases got submitted to an actual judge, and things went about as well as you'd expect from there. CNBC laid out the painful unraveling in detail, right down to the part where the lawyer asked the chatbot if the cases were real and simply believed it when it said yes.
Google has had its share of viral moments too. When AI Overviews launched inside search results, it told at least one person that adding a splash of non-toxic glue would help cheese stick better to pizza, pulling the tip from an old joke post rather than an actual recipe. It also suggested eating a rock a day was healthy, sourced from a satirical article it apparently took completely at face value. Google eventually walked back a lot of this.
