If AI outargues every philosopher, should its conclusions carry more weight?

[Binary Owen]

Imagine a future system that can construct a more rigorous, more internally consistent, more thoroughly argued moral position than any living human philosopher, on any topic you care to name. It anticipates every objection, closes every loophole, and produces conclusions that are genuinely difficult to refute using any standard of argumentative rigor we currently have available. Should that level of demonstrated argumentative skill translate into extra deference toward whatever specific moral conclusions it happens to reach.

The intuitive pull toward yes is strong. We already generally defer to expertise in domains like medicine or engineering precisely because expertise reliably tracks better outcomes and more accurate conclusions in those specific fields. If moral reasoning is a comparable kind of skill, superior argumentative performance should plausibly track superior moral conclusions in a similar way.

But moral philosophy is unusual among domains of expertise in one specific respect, there is no clean, independent, external test that can verify a moral conclusion the way you can verify whether a bridge stands under real world load or whether a treatment reduces mortality in a controlled clinical trial. Argumentative skill in ethics can track genuine moral insight, but it can just as easily track the ability to construct compelling rhetoric that happens to be persuasive without necessarily being correct. Being harder to refute is not the same property as being true, even though the two get conflated constantly in casual conversation.

There is also a deeper worry specific to this scenario, that a sufficiently capable system could construct genuinely compelling arguments for conclusions that are actually catastrophic if acted upon, precisely because it is optimizing for persuasiveness and argumentative coherence rather than for anything we would recognize as goodness in some deeper, more substantive sense. Historical moral atrocities have frequently come packaged with internally coherent, carefully constructed supporting arguments that were highly persuasive to the people living through that specific era.

The honest answer is probably that argumentative skill should raise a claim to our genuine attention and serious consideration without automatically raising it to full deference. Being harder to refute in real time is meaningfully different from being correct, and moral philosophy specifically may be one of the very few domains where that particular gap between the two never fully closes.