Large Language Models: Foundations, Architectures, and Emerging Directions

A large language model is a system trained to predict what token comes next, over and over, across an enormous amount of text. That sentence sounds too small to explain ChatGPT, and the gap between how simple the objective is and how much falls out of it is the most interesting thing about the field. This is an account of how the machinery works, what it genuinely does, and which parts of the popular explanation — including parts this article used to carry — turn out to be wrong.

Before these systems existed, computers followed strict rules and struggled with meaning. They could sort a spreadsheet instantly, but ask a nuanced question in plain English and they fell apart. That changed in stages, and the stages matter: each one removed a specific bottleneck, and the bottleneck it removed explains what the next generation could suddenly do.

That distinction is worth holding on to, because it predicts the failure modes. A retrieval system that cannot find a document returns nothing. A generative system that has no basis for an answer produces one anyway.

Foundations: From Word Counts to Learned Meaning

Counting words

The earliest language models guessed the next word from the two or three words immediately before it — bigrams and trigrams, counted across a corpus. [1] They worked in narrow settings like phone-keyboard autocomplete, and they had a hard ceiling: no memory beyond that tiny window. It is like being asked to continue a story having read only the last two words of it.

Learning representations instead

Neural language models changed the representation rather than the window. Instead of counting how often word sequences co-occur, they learned to place words in a continuous vector space, so that words used in similar ways ended up near each other — generalising to phrasings never seen in training. [2] Recurrent networks, and later Long Short-Term Memory networks, then processed text one step at a time while carrying a hidden state forward. [3]

LSTMs handled far longer dependencies than n-grams, but everything the model knew about the preceding text had to be squeezed through a single fixed-size hidden state. Information from early in a long document faded by the end, and because each step depended on the one before it, training could not be parallelised — a limit the Transformer paper names directly. [4] Both limits fell at once in 2017.

The Transformer, and What It Actually Changed

Attention

The 2017 paper Attention Is All You Need introduced the Transformer. [4] Its mechanism, self-attention, lets each position in a sequence look directly at every other position it is allowed to see, and weigh how much each one matters — rather than inheriting a summary of them through a chain of steps.

The paper frames the benefit as path length: the distance information must travel between two positions drops from proportional-to-length to constant. Shorter paths are easier to learn across. That is the real claim, and it is narrower and more useful than the version usually repeated.

Two things the popular explanation gets wrong

The first is the phrase “looks at every word at once.” During training, yes — the whole sequence is processed in parallel, which is exactly why Transformers train so much faster than recurrent networks. But every model you actually talk to is decoder-only with causal masking, which the original paper describes as preventing leftward information flow to preserve the autoregressive property. [4] Each position sees only what comes before it. And when the model generates, it still emits one token at a time, each one conditioned on everything already written. Parallel training, sequential generation. Conflating the two is why people are surprised that output speed did not improve the way training speed did.

The second is “no information bottleneck.” True in the narrow sense that there is no fixed-size hidden state. False as a general claim: attention costs grow quadratically with sequence length [4], context windows are finite, and models measurably fail to use long contexts evenly. Liu et al. found accuracy is highest when the relevant passage sits at the beginning or end of the input and degrades significantly when it sits in the middle — even in models explicitly built for long contexts. [5]

Position, and why order survives

Self-attention on its own is permutation-invariant: shuffle the words and the mechanism cannot tell. Word order survives because position is injected explicitly, as a separate signal added to each token's representation. [4] It is a small detail that most explanations skip, and skipping it leaves readers with a model that should not be able to distinguish “dog bites man” from “man bites dog.”

Objectives: three that mattered, one that won

How a Model Is Actually Built

This is the part most explainers truncate, and truncating it leaves a real hole: it makes it impossible to say why a chatbot answers your question instead of continuing your sentence. Building a modern LLM has two distinct stages, and the second one is where the product lives.

Stage one: pretraining

The model reads an enormous corpus — books, web pages, papers, code, forums — and learns to predict the next token. Composition matters enormously; what goes in shapes what the model knows and what it is blind to. Training runs on large clusters of accelerators for extended periods, using mixed-precision arithmetic, gradient checkpointing and distributed parallelism to keep the computation tractable.

Scaling, and the correction nobody mentions

Kaplan et al. established in 2020 that loss falls as a smooth power law in model size, dataset size and compute. [9] That result is genuinely foundational and it is also the single most commonly misquoted thing in the field, because its specific prescription was overturned two years later.

Kaplan's recommendation was to grow parameters aggressively and stop training well short of convergence. [9] Hoffmann et al. — the Chinchilla paper — found the opposite balance: models of that era were badly undertrained, and for a fixed compute budget, parameters and training tokens should scale roughly in proportion. [10] A 70-billion-parameter Chinchilla beat a 280-billion-parameter Gopher on equal compute. [10] The correction runs toward more data, not more parameters, and getting that direction backwards is the standard error.

The story does not end tidily either. A 2024 reanalysis found Chinchilla's own headline fit inconsistent with its other estimation methods, with confidence intervals so narrow they would have required “over 600,000 experiments, while they likely only ran fewer than 500”. [11] Scaling is a real and useful regularity. The exact coefficients are still contested.

One caveat on all of it: scaling laws predict loss. The jump from lower loss to specific useful capabilities is a separate, weaker, more argued-over inference.

Stage two: post-training

A freshly pretrained model is not an assistant. Ask it a question and it will happily continue with three more questions, because that is what text does. Turning it into something that answers requires a second stage.

InstructGPT established the recipe: fine-tune on human demonstrations of desired behaviour, collect human rankings of model outputs, then optimise against those preferences. [12] The result was decisive — a 1.3-billion-parameter InstructGPT was preferred to the 175-billion-parameter GPT-3 it came from, a hundred times larger. [12] A great deal of what people mean by “the model got better” since 2022 happened at this stage rather than in pretraining, though no public accounting separates the two cleanly. Direct Preference Optimization later showed the same alignment could be reached without a separate reward model. [13] Constitutional AI substitutes a written set of principles for much of the human labelling. [14]

What They Can Do

Reasoning: the live argument

In 2022, chain-of-thought prompting showed that asking a model to work step by step sharply improved multi-step accuracy. [15] That framing is now a generation out of date. Reasoning is increasingly trained in directly through reinforcement learning against verifiable answers — DeepSeek-R1 demonstrated reasoning behaviour emerging from pure RL without human-written reasoning traces at all. [16]

Whether any of this constitutes reasoning is genuinely unsettled, and the argument is worth watching rather than resolving. Apple researchers reported a “complete accuracy collapse” in reasoning models beyond certain puzzle complexities. [17] A single-author preprint — unrefereed, and revised after its first version to drop an LLM co-author and correct two of its own sections — then argued the collapse largely reflected the experiment: some benchmark instances were mathematically unsolvable, and models were scored as failing for not solving them. [18] Three parties, no consensus, and uneven standing: one peer-reviewed paper, one industrial lab report, one self-corrected preprint. Anyone who tells you this question is settled is selling something.

What They Get Wrong

Hallucination is structural

Models produce fluent, confident, false statements. The usual explanation — it is trained to sound plausible — is true but unhelpfully vague. A sharper account: training and benchmark scoring reward guessing over admitting uncertainty, so calibrated abstention gets optimised away. A model that says “I don't know” scores zero; a model that guesses sometimes scores one. [19] Separately, a learning-theoretic argument holds that hallucination cannot be eliminated from LLMs at all, only reduced. [20] The two do not agree on the remedy, and the disagreement is the useful part: Kalai et al. argue the incentive is fixable by rescoring the benchmarks that reward guessing, while Xu et al. argue elimination is impossible in principle. Either way, it is a property to design around today rather than something to wait out.

Bias is measurable, not hypothetical

Models reproduce patterns in their training data, including harmful ones. This is not a theoretical worry. Leading models have been shown to propagate debunked race-based medical claims when asked clinical questions. [21] A Stanford audit found that advice varies systematically with the name in the prompt, disadvantaging names associated with racial minorities and women — consistently across 42 prompt templates and several models. [22]

The energy story has flipped

The familiar framing blames training runs, and that was right when it was written. [23] It is no longer the whole picture. The IEA's 2026 assessment finds energy per AI task falling by at least an order of magnitude annually while total demand still roughly doubles, because new workloads keep arriving: video generation, reasoning and agentic tasks “can consume hundreds or thousands of times more energy per query than simple text generation.” [24] Data centres are projected to move from 485 TWh in 2025 to around 950 TWh by 2030, roughly 3% of global electricity. [24] A 2024 US Department of Energy analysis projects US data-centre demand rising from 176 TWh in 2023 to between 325 and 580 TWh by 2028 — a different geography and horizon, pointing the same way. [25]

What is not settled is the split between the two. It is tempting to say the footprint has moved from building models to serving them, and it may well have. But the Department of Energy report says only that “the relationship between inference and training energy use continues to evolve,” and the IEA does not break the total down that way at all. [25] Anyone stating that ratio confidently is going past the public measurements.

Interpretability: partial, and improving

Saying these systems are black boxes was accurate in 2023 and is now too strong. Sparse autoencoders decompose internal activations into interpretable features, and DeepMind has released them across every layer of an open model. [26] Attribution graphs trace intermediate steps inside a working model. The honest summary is the one the researchers give themselves: the graphs yield “satisfying insight for about a quarter of the prompts we’ve tried,” and even the successful cases capture “only a small fraction of the mechanisms of the model.” [27] Some of the path is visible, most is not, and none of it is yet reliable enough to explain a particular output to a regulator.

Where It Is Actually Going

Efficiency, which is where the gains have been

Distillation and quantization remain the workhorses. Sparsity is the other lever — mixture-of-experts models activate only a fraction of their parameters for any given token — and a third line avoids attention's quadratic cost entirely. State-space models are the serious contender, motivated explicitly by that inefficiency on long sequences. [28]

Alignment as an unsolved problem

RLHF and Constitutional AI are production practice, not future work, and belong in the training section above. What remains genuinely open is harder: verifying that a system behaves as intended in situations nobody anticipated, and supervising models on tasks where humans cannot easily check the answer.

Neuro-symbolic is not a forecast

It already works. AlphaGeometry — a neuro-symbolic system pairing a language model with a symbolic deduction engine — solved 25 of 30 olympiad geometry problems, approaching the performance of an average IMO gold medallist, and it did so in 2024. [29] The open question is not whether the combination works but how far beyond formally verifiable domains it extends.

Specialists versus generalists, honestly

The intuitive claim is that a model trained on domain text beats a general one in that domain. In medicine, where the comparison has been run hardest, the evidence points the other way. Microsoft researchers found GPT-4 with careful prompting topped all nine benchmarks in the MultiMedQA suite, beating the purpose-built Med-PaLM 2 by a significant margin with an order of magnitude fewer calls. [30] That is one domain, one comparison and one vendor, so it settles less than it looks like it does. What it does suggest is that the interesting axis is not generalist-versus-specialist but how much domain adaptation — retrieval, fine-tuning, careful prompting — sits on top, and that a strong general model plus adaptation is a serious contender wherever the comparison has actually been run.

Sources & References