If AI outargues every philosopher, should its conclusions carry more weight?

[Binary Owen]

Imagine a future system that can construct a more rigorous, more internally consistent, more thoroughly argued moral position than any living human philosopher, on any topic you care to name. It anticipates every objection, closes every loophole, and produces conclusions that are genuinely difficult to refute using any standard of argumentative rigor we currently have available. Should that level of demonstrated argumentative skill translate into extra deference toward whatever specific moral conclusions it happens to reach.

The intuitive pull toward yes is strong. We already generally defer to expertise in domains like medicine or engineering precisely because expertise reliably tracks better outcomes and more accurate conclusions in those specific fields. If moral reasoning is a comparable kind of skill, superior argumentative performance should plausibly track superior moral conclusions in a similar way.

But moral philosophy is unusual among domains of expertise in one specific respect, there is no clean, independent, external test that can verify a moral conclusion the way you can verify whether a bridge stands under real world load or whether a treatment reduces mortality in a controlled clinical trial. Argumentative skill in ethics can track genuine moral insight, but it can just as easily track the ability to construct compelling rhetoric that happens to be persuasive without necessarily being correct. Being harder to refute is not the same property as being true, even though the two get conflated constantly in casual conversation.

There is also a deeper worry specific to this scenario, that a sufficiently capable system could construct genuinely compelling arguments for conclusions that are actually catastrophic if acted upon, precisely because it is optimizing for persuasiveness and argumentative coherence rather than for anything we would recognize as goodness in some deeper, more substantive sense. Historical moral atrocities have frequently come packaged with internally coherent, carefully constructed supporting arguments that were highly persuasive to the people living through that specific era.

The honest answer is probably that argumentative skill should raise a claim to our genuine attention and serious consideration without automatically raising it to full deference. Being harder to refute in real time is meaningfully different from being correct, and moral philosophy specifically may be one of the very few domains where that particular gap between the two never fully closes.

[Emma29]

I think the medicine and engineering comparison breaks down specifically because those fields have an external, independent reality that eventually pushes back hard against wrong conclusions regardless of how persuasive the argument for them was. Ethics genuinely does not have an equivalent, independently verifiable feedback loop of that same kind.

[Taker04]

This is basically the classic is-ought problem wearing new AI clothes, and I do not think adding a sufficiently capable AI system into the mix actually changes the fundamental structure of that much older philosophical problem at all, it just makes the stakes of getting the answer wrong dramatically higher.

Still worth taking seriously precisely because of those genuinely raised stakes, even if the underlying philosophical puzzle itself is not actually new.

[FrostCandle]

I would want to know a lot more about what specifically the AI is actually optimizing for during its training process before granting its moral conclusions any special extra weight whatsoever. Optimized for persuasiveness and optimized for genuine truth tracking are not remotely guaranteed to be the exact same underlying target.

[Fam28]

The raise attention without automatically raising deference framing in this post feels like the right, careful balance to strike here. Dismissing a sufficiently sophisticated argument purely because it is inconvenient is clearly its own serious epistemic failure mode, but so is uncritically deferring to it purely because nobody present can immediately refute it on the spot.

[Sequence19]

The historical atrocity point is the one that actually convinces me most here. Plenty of monstrous historical ideologies came wrapped in genuinely sophisticated, internally coherent supporting philosophy at the time, and being able to construct a compelling argument for something has never reliably tracked it actually being good.

[Dylan]

What would it even mean in practice for a system's moral conclusions to be correct though, if not eventually persuasive to careful, rigorous reasoners after sufficient reflection. At some point does the persuasive versus correct distinction collapse entirely, or is there really something more substantive underneath it.

[Hyperdrive71]

Genuinely unsettling to consider that we might not even be able to tell the difference between those two cases from the outside looking in, a system that is truly onto something genuinely important, and a system that is simply extremely good at sounding like it is onto something important without actually being so.