Major study finds AI assistants give flawed answers nearly half the time

A comprehensive international investigation has found that popular artificial intelligence assistants frequently deliver inaccurate or poorly sourced information, casting doubt on their reliability for journalistic and factual queries. The study, coordinated by Norway's NRK and involving 21 other public service media organizations, evaluated AI models from leading companies including OpenAI's ChatGPT, Google's Gemini, Microsoft's Copilot, and Perplexity. The results indicate significant challenges in how these large language models handle factual news content.

Widespread Errors in Citation and Facts

According to the detailed report published by NRK Beta, nearly half (45%) of all AI-generated answers contained at least one major error. The most common problem involved sourcing, with 31% of responses featuring incorrect or completely missing citations for the information provided. Furthermore, 20% of the answers contained outright false factual claims, including incorrect dates, names, and descriptions of events. In several instances, the AI systems fabricated news article links, creating plausible-looking URLs that led to non-existent pages.

Methodology and Industry Response

The investigation was modeled on an earlier experiment by the BBC. Journalists from the participating organizations posed a standardized set of 30 questions to each of the four AI platforms, covering topics from technology to geopolitics. The responses were then rigorously assessed based on five key journalistic criteria: factual accuracy, quality of sourcing, clear separation of fact from opinion, neutrality, and contextual relevance. The findings were concerning enough that NRK has decided to permanently block AI companies from scraping its website content, a temporary permission granted for this test. "The answers we received are worrying and have not made us feel more confident about loosening control," stated NRK's Technology Director, Pal Nedregotten.

Implications for Information Integrity

The evaluation demonstrates that even the most widely used AI tools currently struggle to process and present news-related information responsibly. The persistent issues with verification and source attribution highlight a fundamental weakness in these systems. For media organizations and the public, the study serves as a critical reminder to approach AI-generated answers with caution and to verify important information through established and authoritative sources.