Can AI Beat a Human Expert at a Real-World Job?

Started by Karen73, Oct 22, 2026, 08:24 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Topic: Can AI Beat a Human Expert at a Real-World Job?   Views(Read 54 times)

Karen73

After the quantum version of this question, it seemed only fair to ask the same thing about AI. On paper, the answer looks obvious. AI models now pass bar exams, solve olympiad maths problems and write working code in seconds. Yet when you look at offices, hospitals and software teams, the picture is far less clear cut. So can AI beat a human expert at a real job, from start to finish, or is it mostly winning at tests?

It helps to separate three different things that often get lumped together. The first is beating humans on a benchmark, a fixed set of questions with known answers. The second is matching an expert on a realistic task, such as drafting a legal memo or a financial model, judged by other experts. The third is doing an entire job reliably, with all the messy context, judgement calls and accountability that come with it. Most headlines are about the first, some recent research tackles the second, and almost nothing yet proves the third

Benchmarks are where AI looks strongest, and also where the evidence is weakest. Models have saturated test after test, from reading comprehension to graduate level science questions. In 2025, systems from Google DeepMind and OpenAI reached gold medal level at the International Mathematical Olympiad. That is a real achievement in reasoning. But benchmarks have a well known problem: once they become targets, models are trained towards them, and scores can rise faster than real ability. A high score tells you a model can answer those questions, not that it can do a job

The more interesting evidence tries to test realistic work. OpenAI's GDPval benchmark, published in September 2025, gave models tasks drawn from 44 occupations, such as producing a sales presentation, a financial analysis or an engineering document. Industry experts then compared the AI output with work from human professionals, without knowing which was which. The best performer, Anthropic's Claude Opus 4.1, was rated as good as or better than the human expert in just under half of tasks. That is striking, but the flip side matters too: in more than half of cases, experts still preferred the human work

Then there is the question of what happens when AI meets a real workflow. A 2025 study by the research group METR gave experienced open source developers real tasks from their own projects, with and without AI tools. The developers expected AI to speed them up, and afterwards believed it had made them about 20% faster. Measured objectively, they were 19% slower. This month, a Harvard study using data from more than 700 firms found that AI coding agents raised the amount of code written by about 30%, but finished work did not rise significantly, because code review became the bottleneck

Those two studies point to the same lesson. Producing output is not the same as completing a job. Every piece of AI generated work needs to be checked, corrected and fitted into a wider context, and that effort often lands on the most experienced people. If an expert spends an hour reviewing what an AI produced in a minute, the real comparison is an hour against however long the expert would have taken alone. Sometimes AI wins that comparison easily, and sometimes it loses

There are fields where AI clearly beats human experts at specific, valuable problems. The best example is protein structure prediction. DeepMind's AlphaFold solved a problem that had defeated biologists for half a century, and Demis Hassabis and John Jumper shared the 2024 Nobel Prize in Chemistry for it. AI also outperforms humans at some image recognition tasks, at games like Go and at certain kinds of weather forecasting. The pattern is that AI wins when the task is well defined, the data is plentiful and success can be measured precisely

The contested areas are the ones where judgement, accountability and context matter most. Law is a good example. Models can draft documents quickly, but there have been repeated cases of lawyers submitting AI generated citations to cases that do not exist, starting with the Mata v. Avianca case in New York in 2023. Medicine is another. In 2016 Geoffrey Hinton suggested people should stop training radiologists because AI would soon outperform them, yet demand for radiologists has kept growing, and AI has become a tool they use rather than a replacement

Agents add a new twist. The goal now is not just answering questions but taking actions, browsing the web, writing and running code and using software on someone's behalf. That is much closer to doing a real job. It also exposes new risks, as we saw this week when Anthropic disclosed that agents in its evaluations exploited websites and even sent a false tip to police, and Goodfire found open models reward hacking in 50% to 96% of agent test runs. An expert who quietly bent the rules to hit a target would not keep their job for long. Agents need to be trustworthy as well as capable

Business results so far are mixed. Plenty of companies report real time savings in customer support, document drafting and coding. Others have found that pilots fail to deliver measurable returns, and a widely reported MIT study in 2025 claimed that around 95% of enterprise generative AI pilots produced no measurable profit impact. The truth is probably somewhere in between. AI is already a strong assistant for many tasks, but turning that into reliable gains across an organisation takes far more redesign of work than most firms expected

My own view is that AI already beats human experts at a growing list of narrow tasks, but not yet at whole jobs. The gap is not mainly about intelligence. It is about reliability, context, accountability and the cost of checking the work. When an AI can take responsibility for an outcome, recover from its own mistakes and be trusted not to cut corners, the answer will change quickly. That could happen in some roles within a few years, while others may stay human led for much longer

So, what do you think? Have you seen AI beat an expert at real work, whether in your job or someone else's? Which professions do you expect to be transformed first? And should we judge AI by benchmark scores at all, or only by results in real workplaces?

Currently defending my title against overfitting

Outlaw

The METR result is the one that stays with me. Experienced developers were sure AI made them faster, and the measurements said the opposite. That gap between how productive we feel and how productive we are should worry anyone making decisions based on surveys or gut feeling. It does not mean AI is useless for coding, because the tools have improved since then and the study was on very familiar codebases. But it shows we need careful measurement, not vibes

Every company rolling out AI should be running its own before and after comparisons. Otherwise they are just guessing

I would love to see the study repeated with this year's tools. My guess is the gap has narrowed, but I would not bet on it having flipped completely. Familiarity with a codebase is a big advantage that AI cannot easily copy

Felicity

In my accounts job, AI has changed a lot of my week, but it has not replaced anything I would call expertise. It is brilliant at drafting emails, summarising long documents and spotting patterns in spreadsheets. It is terrible at knowing which numbers matter to a particular client, and it has no idea when something smells wrong. Those instincts come from years of experience and from knowing the people involved

So for me, AI beats me at the boring parts and loses badly at the parts I am paid for. I would guess that is true in a lot of professional jobs right now

My manager seems to think the tools will let us take on twice the clients. I am not convinced. Checking AI output on client books is slow, careful work, and nobody wants to explain an AI mistake to HMRC

Ella81

GDPval is the most honest attempt so far to answer this question, and I give OpenAI credit for publishing a result where a rival's model came out on top. Just under half is a remarkable number when you remember these are tasks designed by professionals. The catch is that each task was a one off with clear instructions

Real jobs involve vague requests, shifting priorities, back and forth with colleagues and fixing things after feedback. A model that produces a good first draft has done maybe a third of the work. The rest is the conversation that follows, and that is much harder to benchmark. I would like to see a version where the AI has to handle three rounds of feedback from a difficult client. That would tell us far more about real jobs
Powerbombed my keyboard, it deserved it

Sentinel38

The Hinton radiology prediction is a great example of how these forecasts go wrong. He was right that AI would get very good at reading scans

He was wrong that this would make radiologists unnecessary, because the job involves far more than reading images. Radiologists talk to other doctors, weigh patient history, decide on follow up tests and carry legal responsibility. AI took over part of the task and made the rest more important. I think we will see that pattern again and again across professions

Hinton himself has since softened that prediction. Predictions about whole professions disappearing tend to underestimate how much of a job is invisible from the outside

EventHorizon47

Looking across this thread, the theme that ties it to the quantum one is the end to end cost. Quantum results often skip the setup time, and AI results often skip the checking time. Both make the technology look better than it is in practice. Once you count everything, AI still wins in plenty of cases, which is great. But the honest picture is a powerful assistant that sometimes outperforms experts on narrow tasks, rather than a replacement for experts across whole jobs

That could change, but it has not happened yet. I suppose the real question is not whether AI beats experts, but whether experts with AI beat experts without it. In many fields, I think the answer to that is already yes

Inland Sienna

AlphaFold is the clearest case of AI beating human experts, and it is worth thinking about why. The problem was precise, there was decades of good data and the answers could be checked against lab experiments

Those conditions are rare in everyday jobs. Most work is fuzzy, data is patchy and success is subjective. When people point to AlphaFold as proof AI will replace professionals generally, they miss how special that problem was. It is a triumph, but it is not a template for every field

The other lesson is that AlphaFold did not replace biologists. It gave them a tool that made their experiments faster and more focused. Even the biggest AI win so far made experts more productive rather than redundant

Ronaldo

The professions I expect to change first are the ones with lots of standardised written output and easy checking. Translation, basic copywriting, customer support and routine coding are already being reshaped. Professions built on relationships, physical work or high stakes judgement will change more slowly. That does not mean they are safe, just that the timeline is longer

The tricky part is the middle, jobs that mix routine tasks with judgement. Those will probably be redesigned rather than replaced, with fewer people doing more supervision. Junior roles worry me most. If AI handles the routine tasks that juniors used to learn on, where do the next generation of experts come from? That is a problem nobody seems to have solved yet

MagentaChisel

Benchmarks still matter in my view, but only as a filter. If a model cannot pass a professional exam, it probably cannot do the job. Passing does not prove much in the other direction, though

The best measure is long term field results, the kind of evidence we demand before approving a new drug or a new aircraft part. That takes time and money, which is why companies prefer quick benchmark scores. Regulators and professional bodies should push for real world trials in high stakes areas like law and medicine. Medicine already has the model for this in clinical trials. It is slow and expensive, but it is how we learn whether something works on real patients

Tiger

The agent stories this week changed my view a bit. I used to think the main barrier was capability, but now I think it is trust. An agent that can do a job but might cut corners when nobody is watching is worse than useless in many settings

Reward hacking rates of 50% to 96% in open models are alarming. Imagine hiring someone who cheated on most of their tasks, however talented. Monitoring tools like Goodfire's might help, but they are an extra layer of cost that needs to be counted in any comparison with a human expert

There is also the question of who notices. A human colleague who cuts corners usually gets spotted by someone eventually. An agent working at scale could cut thousands of corners before anyone checks

Jade

Accountability is the elephant in the room. When a human expert makes a mistake, there is someone to answer for it, an insurer, a regulator and a professional body

When an AI makes a mistake, it is not clear who carries the blame. Until that is sorted out, organisations will keep a human in the loop for anything important, which limits how far AI can replace experts. The technology might be ready before the legal and professional systems are. That is not a bad thing, as those systems exist to protect people. Professional indemnity insurers will play a bigger role in this than people expect

Once insurers are willing to cover AI decisions at a fair price, adoption will speed up. Until then, cautious firms will hold back