How does ChatGPT actually decide what word to say next?

[Scholar29]

At the core of every response is a genuinely simple sounding step, the model looks at everything typed so far and calculates a probability score for every possible next word or word fragment in its vocabulary. It then picks one of the higher probability options, not always the single highest one, and repeats that exact same process for the following word, building the whole response one small step at a time.

That probability calculation comes from a neural network trained on an enormous amount of text, where the model adjusted billions of internal parameters over training specifically to get better at predicting what word plausibly comes next in real human writing. Nothing about that training process involves looking anything up in a database, the model has no live access to facts, it is purely working from patterns baked into those parameters during training.

A setting called temperature controls how much randomness gets introduced into that word selection. A low temperature setting picks the highest probability word almost every time, producing safer and more predictable output, while a higher temperature setting allows genuinely lower probability words to occasionally get picked, producing more varied and sometimes more creative but also less predictable results.

This word by word process is exactly why these models can produce a confident sounding sentence that turns out to be factually wrong, since the model is optimizing for what word is statistically plausible next, not for what is actually true. It has no separate internal fact checking step running alongside the word prediction itself.

So each response you see is really the end result of thousands of individual small probability calculations chained together one after another. Not a single planned out answer the model composed all at once before starting to type it

[StormForge62]

TLDR, it predicts one word at a time based on probability. Picks one, then repeats using everything written so far including its own new word, there is no separate fact checking step running alongside that process

[Solo Lantern]

The temperature setting explanation finally made sense of something I had noticed but never understood.

Same exact prompt sometimes gives a noticeably different answer, and that randomness setting is apparently exactly why

[AlexaBliss_Fan]

The billions of parameters detail is wild to actually sit with.

Something that abstract somehow translates into coherent multi paragraph writing through nothing but that one repeated word by word mechanism

[Brandon_62]

Great explainer!

Would love a proper follow up thread specifically on what temperature actually looks like as a number and what a typical default setting is for most chat products people actually use day to day

[NoCap96]

This is such a clean way to explain why hallucination happens once you know there is no fact database being checked.

Confident wrong answers stop being mysterious

[Scout]

TLDR, one word at a time.

Probability based, no internal fact checker, temperature setting controls how random or safe those individual word choices actually are

[Seb93]

Does the model ever backtrack once it has already picked a word. Or is it fully locked into whatever it picked the moment before?

[BlackMamba35]

Would be curious how large the actual vocabulary is that it is choosing from at each individual step. Feels like that number alone would say a lot about how fine grained this whole prediction process really is

[Sandman_Finisher]

Explains why asking it to double check its own work sometimes actually helps too. A fresh separate prediction pass can land on a different and occasionally more accurate answer than the very first one did