AI systems like ChatGPT can provide useful information, but can they be trusted to justify their outputs? Learn why AI lacks real reasoning and how it impacts our understanding of truth.
Introduction
Every month, more than 500 million people rely on AI systems like ChatGPT and Gemini to answer questions on a vast range of topics, from cooking to algebra. While AI is increasingly trusted as an authority on these subjects, there is a critical flaw in this assumption: the content provided by large language models (LLMs) like ChatGPT is not based on genuine reasoning. Despite their ability to generate human-like responses, LLMs lack the capacity to provide justifications for their outputs, which raises serious concerns about whether we can truly trust the information they deliver.
At the World Economic Forum in January, OpenAI CEO Sam Altman reassured the public that AI systems could eventually explain their reasoning. However, this promise is misleading because LLMs aren’t designed to reason. Instead, they are trained to recognize patterns in vast amounts of human text and predict what comes next based on these patterns. This article explores the limitations of AI-generated content, why we cannot fully trust its “justifications,” and the implications for knowledge in the digital age.
The Role of Justification in Knowledge
In everyday life, we don’t consider something knowledge unless it is justified. For example, when you believe something to be true, you typically feel confident because it is supported by evidence, reasoning, or trusted sources. Without these justifications, we can’t know for sure whether what we believe is accurate.
This same principle applies to AI-generated content. If an AI system, such as ChatGPT, provides an answer, we might take it at face value. But if that system can’t explain why its output is true or justified, how can we trust it? Without understanding the underlying reasoning, there’s a risk that we’re simply accepting an illusion of knowledge.
As philosophers Hicks, Humphries, and Slater pointed out in their essay “ChatGPT is Bullshit,” LLMs are built to produce text that seems plausible but without concern for whether it is actually true. This means that while AI outputs may often be factually correct, the process by which they are generated is divorced from any real understanding or justification.
The Mirage of AI Justification: Gettier Cases
To illustrate the problem, consider an example from the 8th-century Indian Buddhist philosopher Dharmottara. Imagine you’re searching for water on a hot day and, upon seeing what looks like a water source in the distance, you approach it. What you saw turns out to be a mirage, but as luck would have it, there is indeed water hidden beneath a nearby rock. Did you really know there was water there, or did you just get lucky?
This philosophical puzzle illustrates the nature of AI outputs. Even if an AI system provides a factually correct answer, the reasoning behind it is nonexistent—much like the travelers finding water by accident. This type of situation is what philosophers call a Gettier case, where true beliefs are arrived at through unreliable methods. When you ask an AI a question, its response might be accurate, but since the system isn’t engaging in real reasoning, the output is more like a mirage than genuine knowledge.
Altman’s claim that AI can justify its outputs fails to account for this reality. While an AI may provide an explanation that sounds reasonable, it is merely another predictive response, not a genuine justification.
The Risk of Deception: The “Quasi-Matrix” Effect
As AI-generated content becomes more sophisticated, we are moving closer to a scenario where many people will take its outputs as truth without realizing the absence of justification. Those who understand the limitations of LLMs might maintain a healthy skepticism, but for the general public, this could lead to a dangerous situation.
We risk living in what can be called a “quasi-matrix,” where people can no longer discern fact from fiction because they have been lulled into trusting AI outputs without questioning their origins. This deception is not malicious, but rather a byproduct of AI’s design, which prioritizes pattern recognition over logical reasoning.
The Importance of Transparency and Verification
Despite their limitations, LLMs are incredibly powerful tools that can assist in various tasks, from writing to coding. However, users need to understand their limitations. AI should not be viewed as a replacement for expertise, but rather as a tool to augment human capabilities. For example, programmers might use an AI system to draft code, but they will still need to review and adjust the code based on their own knowledge.
The public, however, often turns to AI for advice in areas where they lack expertise—whether it’s teenagers researching algebra or seniors seeking health advice. In such cases, relying on an AI without understanding how it generates its answers could lead to misinformation or poor decision-making. Therefore, it is crucial that AI outputs are accompanied by transparent explanations or references, allowing users to verify information independently.
Conclusion
AI systems like ChatGPT are astonishingly good at mimicking human language and providing answers to complex questions. However, the critical flaw lies in their inability to reason and justify their outputs. Without this capability, AI cannot be considered a reliable source of knowledge, as its responses are more akin to lucky guesses than informed conclusions.
Users must approach AI-generated content with caution, understanding that these systems are not providing true justifications. As AI continues to evolve, there needs to be greater emphasis on transparency, verification, and the recognition that human oversight remains essential. Just as we wouldn’t trust AI to cook pasta with gasoline, we shouldn’t blindly trust its advice on more complex matters without careful scrutiny.
In a world increasingly influenced by AI, knowing when—and when not—to trust it will be crucial to navigating the future of knowledge and information.
