One idea from a 2017 research paper is quietly running every chatbot you have ever used

[Cheeky Blake]

Before 2017, AI language models processed text the way a person reads a difficult sentence one word at a time, in order, never able to properly hold the whole thing in mind at once. A paper with the almost cheeky title Attention Is All You Need introduced a different architecture called the transformer, which does something that sounds almost too simple to matter, it lets every word in a piece of text look at every other word simultaneously, deciding for itself which other words actually matter for understanding it

That mechanism, called self attention, is why a transformer can tell that in the sentence the trophy would not fit in the suitcase because it was too big, the word it refers to the trophy rather than the suitcase, a distinction that depends on relationships across the whole sentence rather than just the words sitting right next to each other. Processing every word in parallel rather than one at a time also happens to make transformers dramatically faster to train on modern computer hardware, which turned out to matter just as much as the improved understanding itself

This single architecture is the quiet, unglamorous engine sitting underneath nearly every major AI system people actually interact with today, ChatGPT, Claude, Gemini, and the open weight models built by labs around the world. Different companies have built increasingly elaborate systems on top of it, but the core insight from that 2017 paper, let the model figure out for itself what to pay attention to, remains essentially unchanged at the foundation of the entire generative AI boom

What is a Transformer Model? | IBMwww.ibm.com

[Hollow Ronan]

The trophy and suitcase example is such a perfect illustration, that is exactly the kind of contextual understanding that used to trip up older AI systems constantly