If AGI reliably beats us at ethics, should we defer or is that a category error?

[BackRowBob]

Suppose an artificial general intelligence arrives that reliably outperforms human experts across essentially every measurable domain, medicine, engineering, strategy, and suppose it also consistently produces ethical judgments that hold up better under sustained scrutiny than what any human ethics panel manages to produce. Should humanity then defer to that system specifically on moral questions the same way it might reasonably defer to it on, say, protein folding or aircraft design.

The case for deferring treats ethics as fundamentally continuous with every other domain of knowledge, a field where genuine expertise exists, where some answers really are meaningfully better than others, and where a sufficiently capable reasoner should therefore be expected to outperform limited human judgment for entirely unsurprising reasons, the same reasons any superior reasoner outperforms a lesser one in any other domain.

The case against deferring holds that ethics is different in kind, not merely in degree, from empirical domains like protein folding. Empirical claims can be checked against an external, independent reality in a way that settles genuine disagreement over time. Moral claims, on at least some influential philosophical views, are not truth apt in that exact same sense, or if they are, we currently lack any external, independent way to check them the way we can check a bridge design or a drug's efficacy in a trial.

There is also a genuinely deeper worry specific to this exact scenario, that deferring to an external authority on ethics specifically undermines something important about moral agency itself, the idea that being a genuinely moral agent requires actually doing your own reasoning and taking real ownership of your own conclusions, rather than simply outsourcing that reasoning wholesale to whatever system happens to currently be smartest. Under this view, even a truly perfect external ethical oracle would not fully solve the problem, because part of what we actually want out of ethics is precisely the process of reasoning it through ourselves, not merely the correct answer delivered to us from outside.

A more moderate position treats AGI ethical output as an extremely valuable input worth taking seriously into human moral reasoning, without fully replacing that reasoning altogether. That preserves human moral agency as something that still matters in its own right, while not simply throwing away a genuinely valuable resource just because it happens to originate from a machine rather than a person.

[TinyGriffin]

The moral agency argument is the one that actually gets me here. Even if an AGI reliably produces better answers than we do, something does feel like it would be lost if we simply outsourced the entire process of moral reasoning wholesale rather than doing at least some of that work ourselves.

[Florence19]

I think the empirical versus non-empirical distinction being drawn here is doing a huge amount of the actual philosophical work in this whole argument, and it happens to be one of the most contested, unresolved questions in all of metaethics. This entire debate essentially reduces down to whichever side of that older debate you already happen to land on.