How the Transformer Works — The Engine Inside ChatGPT, Claude, and Gemini

Glossary

Token — the smallest unit of text a language model processes. Can be a word, part of a word, a punctuation mark, or a character.

Embedding — a list of numbers (a vector) that represents a token's meaning. Similar meanings are close together in this numerical space.

Attention — the mechanism by which each token assigns importance weights to every other token in the sequence, so the model knows what context to use.

Context Window — the maximum number of tokens a model can "see" at once during a single interaction.

Transformer — the neural network architecture introduced in 2017 that uses self-attention instead of sequential processing.

Parameter — a learned numerical weight inside the model. GPT-4 has an estimated one trillion parameters.

Pre-training — the phase where a model learns language patterns by predicting missing words across billions of text documents.

Fine-tuning — a second training phase where the model is trained on curated examples to behave helpfully, accurately, and safely.

In 2017, eight researchers at Google Brain published a paper titled Attention Is All You Need. It was twelve pages long. It introduced no new hardware, no exotic dataset, no proprietary trick. It proposed a new architecture for processing sequences — a way of letting every word in a sentence look at every other word simultaneously, rather than reading left to right.

That architecture is called the Transformer.

Today, ChatGPT, Claude, Gemini, Llama, Mistral, Krutrim, and every serious language model on the planet is built on it. Understanding the Transformer does not require a maths degree. It requires one good example and three core ideas.

The problem Transformers solved

Before 2017, the dominant approach to language AI was called a Recurrent Neural Network (RNN), or its more capable cousin, the Long Short-Term Memory network (LSTM).

These models read text like a person following a ticker tape — one word at a time, left to right. Each word was processed sequentially. A hidden state was passed forward, carrying a compressed memory of everything seen so far.

The problem: by the time the model reached word 40, it had half-forgotten word 3.

Take this sentence:

Priya deposited ₹50,000 at the bank near the river.

An RNN reading this word by word reaches "bank" and has already started forgetting "deposited" and "₹50,000." It has a faint statistical memory of financial context — but the further back those words are, the weaker the signal.

If the sentence were a paragraph, the context would be nearly gone.

The Transformer solved this by refusing to read sequentially at all.

Q: Why did RNNs struggle with long sentences?

A: Because they processed words one at a time and "forgot" early context by the time they reached later words — like reading through a slowly fading keyhole.

Concept 1 — Tokens: breaking language into atoms

Before the model sees any text, every sentence is split into tokens.

Tokens are not always individual words. The tokenizer (a preprocessing step) breaks text into the smallest meaningful units that the model's vocabulary covers.

For English text, a word like deposited might be one token. A word like ₹50,000 might be split into ₹, 50, ,, 000 — four tokens. Indian language scripts often need more tokens per word than English because the vocabulary coverage is different.

Each token is then converted into an embedding — a list of numbers, typically 768 to 12,288 values long, depending on the model. These numbers encode meaning: the embedding for deposited will be mathematically close to the embeddings for transferred, credited, and paid because they appear in similar contexts across billions of training documents.

This is how the model "knows" words. Not because someone wrote a dictionary. Because patterns emerged statistically from seeing how words relate to each other billions of times.

Concept 2 — Attention: who is talking to whom?

Here is where Transformers became genuinely different from everything before them.

Consider our sentence again:

Priya deposited ₹50,000 at the bank near the river.

When the Transformer processes the token "bank," it asks: which other tokens matter most for understanding what "bank" means in this sentence?

The attention mechanism computes a score — a weight between 0 and 1 — from "bank" to every other token in the sentence simultaneously.

The model does not need anyone to tell it that "deposited" and "₹50,000" are more important than "river." It learned this from training on millions of sentences where "bank" near "deposited" always meant a financial institution, while "bank" near "river" always meant a riverbank.

The result: the model concludes with high confidence that "bank" here is a financial institution, not a geographical one.

This calculation happens not just for "bank" — it happens for every single token in the sentence, simultaneously. Every token looks at every other token and assigns weights.

That is self-attention.

Why "attention heads" matter

The original Transformer paper used not one attention mechanism but eight in parallel — called attention heads.

Each head learns to attend to different kinds of relationships:

• One head might focus on grammatical agreement (subject-verb)
• Another might focus on coreference (which pronoun refers to which noun)
• Another might focus on semantic similarity (deposited ↔ ₹50,000)
• Another might focus on negation (did not deposit → very different meaning)

The outputs of all eight heads are concatenated and combined. Modern frontier models use 96 or more attention heads per layer, stacked across 96 or more layers. This is how subtle nuance — sarcasm, irony, cultural reference — gets represented numerically.

Q: If GPT-4 has 96 attention layers with 96 heads each, roughly how many separate attention calculations happen for a single sentence?

A: 96 × 96 = 9,216 separate attention calculations, each looking at the full token sequence simultaneously. For a sequence of 50 tokens, that's 9,216 × 50 × 50 = ~23 million individual weight computations — for one forward pass.

Concept 3 — Context Window: how much the model can see

The context window is everything the model can read at once in a single interaction.

Think of it as the model's working memory. Anything inside the context window is "visible" and can influence the response. Anything outside it is invisible — the model has no access to earlier conversation turns that have been scrolled out of the window.

Context windows have grown dramatically:

To put this in Indian terms: a Gemini 2.5 Pro context window is large enough to hold the complete works of Rabindranath Tagore — all poems, novels, plays, and essays — and still have room for a technical manual.

A GPT-4 context window holds roughly the entire Harry Potter and the Goblet of Fire in one sitting.

What happens during training

The Transformer learns by seeing an enormous amount of text and playing a prediction game:

1. Take a sentence: "Ratan Tata founded the Tata ___"
2. Hide the last word
3. Ask the model to predict it
4. Measure how wrong the prediction was
5. Adjust all parameters slightly in the direction of a better answer
6. Repeat — billions of times, across trillions of words

This is called pre-training. The model never sees the same exact text twice (ideally). It learns grammar, facts, reasoning patterns, and the statistical structure of language entirely through this prediction game.

After pre-training, models undergo fine-tuning — a second phase where human raters score thousands of responses for quality, helpfulness, and safety. This shapes the model's personality and teaching style.

The combination is why modern AI can write a cover letter, explain a theorem, debug code, and suggest a recipe — it has compressed billions of human-written examples into its parameters.

Why Transformers run efficiently on GPUs

This is not an accident. The attention calculation is fundamentally matrix multiplication — computing the dot product between every pair of token vectors to produce attention weights.

Modern GPUs are designed to perform matrix multiplications in parallel at enormous speed. The Transformer architecture maps perfectly onto what GPUs do best: thousands of identical calculations happening simultaneously rather than sequentially.

This is why training a model like GPT-4 requires thousands of A100 GPUs running for months — but also why inference (generating a response) can happen in seconds even on consumer hardware.

RNNs could not be parallelised in the same way — they had to process tokens one by one, even on GPUs. Transformers removed that bottleneck.

What the Transformer does NOT do

It helps to be clear about what language models are not:

They do not understand. They predict. A model that produces a brilliant explanation of quantum physics is doing sophisticated pattern completion — it has seen billions of physics explanations and learned which words follow which words in this domain. That can produce genuinely useful outputs. But it is not the same as understanding.

They do not have memory across sessions (unless explicitly given one via the context window or tools). Each conversation starts fresh unless the system provides prior context.

They can hallucinate. If a model is asked about something rare or internally inconsistent, the prediction machinery can generate fluent-sounding text that is factually wrong. This is a fundamental property of the architecture, not a bug being actively fixed.

They do not read the internet in real time (unless a search tool is attached). A base model's knowledge is frozen at its training cutoff date.

Q: A customer asks an AI chatbot: "What is the current gold price in Mumbai?" The chatbot confidently gives a price. Should you trust it?

A: No, unless the chatbot has a live search tool. Base language models don't access real-time data — the price they give is from training data that may be months or years old.

From the lab to your pocket — the complete pipeline

Here is the full journey from the 2017 paper to the assistant on your phone:

Step 1 — Architecture (2017): The Transformer paper shows that self-attention alone can handle sequence tasks better than RNNs.

Step 2 — Scaling (2018–2020): GPT-1, BERT, GPT-2, GPT-3 show that making the model bigger and training on more data keeps improving performance with no ceiling in sight.

Step 3 — Alignment (2022): InstructGPT and RLHF (Reinforcement Learning from Human Feedback) teach the model to be helpful rather than just predict text statistically.

Step 4 — ChatGPT (November 2022): The first model combining scale + alignment deployed as a consumer product. 100 million users in 60 days.

Step 5 — Indian deployment (2023–2025): Krutrim (Ola), Sarvam AI (fine-tuned for 10 Indian languages), government NDEAR AI initiatives all built on the same Transformer backbone.

What this means for SAP professionals

Every AI feature landing in SAP's portfolio — Joule (the SAP AI copilot), AI-assisted FP&A in S/4HANA, anomaly detection in SAP Analytics Cloud, generative AI in SAP Build — is a Transformer model or a system that calls one.

When Joule suggests a journal entry correction, it is running attention over the transaction context you provided. When SAP's anomaly detection flags an unusual posting pattern, it is comparing embeddings of current transactions against learned normal behaviour.

Understanding Transformers does not mean you need to train one. It means you can reason about what these systems can and cannot do, where their confidence is reliable, and where a human expert still needs to check the work.

The Transformer is the invisible engine. Now you know how the engine works.

Next in the series: AI in Your Daily Life — India. Six ways AI already runs your day without you realising it.