If AGI reliably beats us at ethics, should we defer or is that a category error?

Started by BackRowBob, Aug 21, 2026, 05:39 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Topic: If AGI reliably beats us at ethics, should we defer or is that a category error?   Views(Read 108 times)

BackRowBob

Suppose an artificial general intelligence arrives that reliably outperforms human experts across essentially every measurable domain, medicine, engineering, strategy, and suppose it also consistently produces ethical judgments that hold up better under sustained scrutiny than what any human ethics panel manages to produce. Should humanity then defer to that system specifically on moral questions the same way it might reasonably defer to it on, say, protein folding or aircraft design.

The case for deferring treats ethics as fundamentally continuous with every other domain of knowledge, a field where genuine expertise exists, where some answers really are meaningfully better than others, and where a sufficiently capable reasoner should therefore be expected to outperform limited human judgment for entirely unsurprising reasons, the same reasons any superior reasoner outperforms a lesser one in any other domain.

The case against deferring holds that ethics is different in kind, not merely in degree, from empirical domains like protein folding. Empirical claims can be checked against an external, independent reality in a way that settles genuine disagreement over time. Moral claims, on at least some influential philosophical views, are not truth apt in that exact same sense, or if they are, we currently lack any external, independent way to check them the way we can check a bridge design or a drug's efficacy in a trial.

There is also a deeper worry specific to this exact scenario, that deferring to an external authority on ethics specifically undermines something important about moral agency itself, the idea that being a moral agent requires actually doing your own reasoning and taking real ownership of your own conclusions, rather than simply outsourcing that reasoning wholesale to whatever system happens to currently be smartest. Under this view, even a truly perfect external ethical oracle would not fully solve the problem, because part of what we actually want out of ethics is precisely the process of reasoning it through ourselves, not merely the correct answer delivered to us from outside.

A more moderate position treats AGI ethical output as an extremely valuable input worth taking seriously into human moral reasoning, without fully replacing that reasoning altogether. That preserves human moral agency as something that still matters in its own right, while not simply throwing away a valuable resource just because it happens to originate from a machine rather than a person.
Forum veteran. Battle hardened.

TinyGriffin

The moral agency argument is the one that actually gets me here. Even if an AGI reliably produces better answers than we do, something does feel like it would be lost if we simply outsourced the entire process of moral reasoning wholesale rather than doing at least some of that work ourselves.

Florence19

I think the empirical versus non-empirical distinction being drawn here is doing a huge amount of the actual philosophical work in this whole argument, and it happens to be one of the most contested, unresolved questions in all of metaethics. This entire debate essentially reduces down to whichever side of that older debate you already happen to land on.
GG no re

NinaVrina

I lean toward thinking the category error framing in the original post is basically correct here. Ethics is not merely a harder empirical puzzle sitting further along the exact same continuum as physics or engineering, it is doing something structurally different in kind, and treating superior raw intelligence as automatically implying superior moral authority conflates two genuinely separate things.
VAR can do one

Tracey

The moderate position of treating AGI output as valuable input rather than final oracle feels like the honest, sensible answer, but I worry in practice it quietly collapses back into full deference anyway once the system in question is demonstrably reliable enough and convenient enough for people to simply lean on.

Humans are not always great at maintaining genuinely careful epistemic humility once something consistently, reliably makes their life meaningfully easier.

EventHorizon Crossing

This connects directly to older debates about moral expertise more broadly, whether any human ethicist could ever count as a genuine moral expert in the first place. If human experts already cannot claim that kind of clean deferential authority over ethics, it is not obvious a machine automatically gets to claim it either, purely by being smarter across a bunch of unrelated technical domains.
Just here collapsing wave functions :)

Vulture50

Genuinely worried this entire question stops being purely hypothetical faster than people currently expect. Once a sufficiently capable AGI system starts confidently offering moral guidance at scale, plenty of people will simply follow it in practice regardless of how this particular philosophical debate ultimately resolves in the seminar room.

IronQuarry50

What if the actual answer genuinely depends heavily on which specific ethical question is actually being asked. Deferring on some narrow, comparatively technical question, like the most effective way to reduce a specific harm given some already agreed upon shared goal, feels very different from deferring on some deep, foundational question about what ultimately matters in the first place

Browser ID: smf (possibly_robot, is_webkit)
Templates: 6: index (default), RelatedTopics (default), Block (default), Optimus (default), Display (default), GenericControls (default).
Sub templates: 12: init, html_above, Blockheaders_above, body_above, Blockindex_above, footer_counters_above, main, footer_counters_below, Blockindex_below, body_below, Blockheaders_below, html_below.
Language files: 11: index+Modifications.english (default), Turnstile.english (default), RelatedTopics.english (default), Block.english (default), Optimus/Optimus.english (default), Post.english (default), Editor.english (default), Drafts.english (default), giphy.english (default), QRCode.english (default), klipy.english (default).
Style sheets: 7: index.css, attachments.css, turnstile.css, ajaxsearchlite.css, jquery.sceditor.css, show_password.css, responsive.css.
Hooks called: 296 (show)
Instances created: 14 (show)
Files included: 138 - 1847KB. (show)
Memory used: 11079KB.
Tokens: post-login.
Cache hits: 22: 0.00358s for 44,161 bytes (show)
Cache misses: 3: (show)
Queries used: 25.

[Show Queries]