Has humanity outsourced truth itself to a black box nobody understands?

[Ella81]

For a rapidly growing number of people, an AI chatbot has become the default first stop for figuring out what is actually true about almost any topic, replacing the search engine, which itself had already replaced the encyclopedia and the reference librarian before it. Each step in that particular chain moved us further from directly consulting a primary source and closer to trusting a system whose internal reasoning process is, in an important sense, genuinely opaque even to the people who actually built it.

That opacity is worth sitting with directly rather than rushing past it. Nobody, including the engineers at the company that trained a given model, can fully trace out why it produced this specific answer rather than some other plausible one, in the same detailed, mechanistic way you could trace a calculator's output back to its underlying arithmetic operations. We have interpretability research chipping away steadily at this problem, but at the level of everyday practical use, most people are trusting an output whose provenance nobody can fully explain or verify from first principles.

This would matter less if these systems were reliably, consistently accurate, but they demonstrably are not. They confabulate confidently sounding, well structured wrong answers with exactly the same fluent tone and stylistic confidence as their correct ones, and unless a person already independently knows enough about the topic in question to catch the error themselves, there is frequently no readily available way to tell the difference from the outside.

The optimistic counterargument is that human epistemic authority was never actually as clean, direct, or trustworthy as this framing nostalgically implies in the first place. People have always trusted institutions, experts, and reference works they could not personally fully verify from scratch, doctors, encyclopedias, teachers, and outsourcing trust to an AI system is simply the newest entry in a very long, familiar historical pattern rather than some genuinely unprecedented rupture.

What feels genuinely new here is the sheer combined scale and fluency of the outsourcing happening all at once. Never before has a single source of information been consulted by this many people this many times a day while remaining this fundamentally unauditable by any of them individually, and that specific combination of properties, massive scale plus genuine opacity, is worth taking seriously as something meaningfully new under the sun.

[alwaysRock40]

I keep thinking about how this specifically affects people who do not already have deep domain expertise to fall back on and catch errors themselves. Experts can generally catch a hallucinated wrong answer in their own specific field fairly easily, but that same protection simply does not extend to literally everything else those same experts happen to ask about outside their own narrow area.