What is a foundation model?

Where the term comes from

"Foundation model" is a young term with a precise origin. It was coined in 2021 by the Center for Research on Foundation Models (CRFM) at the Stanford Institute for Human-Centered AI, in a 200-page report titled "On the Opportunities and Risks of Foundation Models", led by Rishi Bommasani and Percy Liang with more than 100 co-authors.

The report defines a foundation model as "any model that is trained on broad data (generally using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a wide range of downstream tasks." The authors chose the word "foundation" deliberately: these models are an unfinished base on which countless applications are constructed, much as a building's foundation is load-bearing but not itself the building.

The concept described something that already existed — models like BERT (2018) and GPT-3 (2020) — but gave the field a shared name for a paradigm shift: away from bespoke, task-specific models and towards a few large, general-purpose bases reused everywhere.

Pretraining and adaptation

A foundation model is built in two stages. First comes pretraining: the model learns general structure from an enormous, broad dataset — much of the public web, code, books or images — usually through self-supervision, where the training signal comes from the data itself (for example, predicting the next word) rather than from human labels. This stage is extraordinarily expensive, requiring large clusters of specialised chips.

Second comes adaptation. The same pretrained base is specialised to particular tasks through fine-tuning on smaller labelled datasets, instruction tuning, reinforcement learning from human feedback, prompting, or retrieval-augmented generation. Adaptation is comparatively cheap, which is the economic point of the whole approach.

This split — expensive to build once, cheap to reuse many times — is what makes foundation models such a powerful economic lever. A single organisation absorbs the enormous pretraining cost, and thousands of downstream developers reuse the result. It is closer to building a power plant than writing a program.

Why the term exists at all

The Stanford authors argued that existing labels were too narrow. "Large language model" captures only text and only scale; "self-supervised model" describes a training method, not the paradigm. They wanted a word for the model's function — a general, adaptable base — independent of size or modality.

The distinction matters commercially. A foundation model is a platform: whoever controls the base captures value from everything built on top. This reframes AI as an infrastructure business, where a handful of providers supply the underlying capability and a wide ecosystem of companies adapts it. It also reframes the policy debate, because decisions made by a few base-model providers propagate to everyone downstream.

It is worth stressing what the word does not claim. "Foundation" is not a quality guarantee. The Stanford report is explicit that these models are incompletely understood and carry serious risks; the name marks their structural role, not their trustworthiness.

Examples across modalities

The clearest examples are large language models: GPT-4, Claude and Gemini are foundation models for text, adapted into chat assistants, coding tools and search products. But the category is broader than text.

In images, models such as DALL-E, Midjourney and Stable Diffusion are foundation models: pretrained on huge image-text datasets, then adapted for generation, editing and style transfer. In audio there are speech and music models; in science, systems like AlphaFold act as foundation models for protein structure. Increasingly, multimodal foundation models handle text, images and audio within a single system.

What unites them is the recipe, not the data type: train broadly at scale, then adapt. Almost all of these systems are built on the transformer architecture, which is why the two concepts are so often discussed together.

Emergent capabilities

One of the most striking claims in the Stanford report is that "scale results in new emergent capabilities." Emergence describes abilities that are absent in smaller models but appear, sometimes abruptly, once a model crosses a certain size — for example, performing arithmetic, translating between language pairs never explicitly trained, or following multi-step instructions.

Emergence is a double-edged property. On one hand it is why simply scaling up models has been so productive, and why capability is hard to predict from a smaller prototype. On the other hand it makes foundation models fundamentally hard to anticipate: a larger model may acquire abilities — helpful or harmful — that its builders did not design and cannot fully foresee.

The research community continues to debate how sharp these jumps really are versus how much depends on how capability is measured, but the practical upshot is agreed: you cannot fully know what a foundation model can do until you build and test it at scale.

Homogenization and centralisation risk

The Stanford report's second structural warning is homogenization. Because so many applications are adapted from the same handful of base models, the whole ecosystem inherits their characteristics. As the authors put it, "the defects of the foundation model are inherited by all adapted models downstream."

This concentrates both capability and risk. A single flaw, bias or security weakness in a widely used base model propagates to every product built on it. It also concentrates market power: training frontier models costs enough that only a few well-funded organisations can do it, which raises questions about competition, accountability and access that regulators — including the EU AI Act and standards bodies such as the US NIST — are now actively addressing.

For anyone relying on AI, the lesson is to treat the choice of foundation model as strategic infrastructure. Its capabilities, licence, transparency and failure modes are inherited wholesale by whatever you build on top.

Frequently asked questions

Is a foundation model the same as an LLM?

Not exactly. Every large language model (LLM) is a foundation model for text, but the foundation-model category is broader: it also covers image models like Stable Diffusion, audio models and scientific models like AlphaFold. "LLM" specifies the modality (language) and scale; "foundation model" describes the paradigm of a broadly-trained, widely-adaptable base regardless of modality.

Who coined the term foundation model?

The term was coined in 2021 by Stanford's Center for Research on Foundation Models (CRFM), part of the Stanford Institute for Human-Centered AI, in the report "On the Opportunities and Risks of Foundation Models", led by Rishi Bommasani and Percy Liang with more than 100 co-authors.

What is the difference between pretraining and fine-tuning?

Pretraining is the expensive, one-time stage in which a model learns general structure from a huge, broad dataset, usually via self-supervision. Fine-tuning is a much cheaper later stage that adapts the pretrained base to a specific task using a smaller, often labelled dataset. The split is what makes foundation models economical to reuse.

What are emergent capabilities in foundation models?

Emergent capabilities are abilities that do not appear in smaller models but arise once a model is scaled up past a certain size — such as arithmetic or multi-step reasoning. The Stanford CRFM report highlighted that "scale results in new emergent capabilities", which makes large models both powerful and hard to predict.

Why are foundation models considered risky?

Because so many products are adapted from a small number of base models, any defect, bias or vulnerability in a base model is inherited by everything built on it — what Stanford calls homogenization. This concentrates capability, market power and risk in a few organisations, which is why regulators such as the EU AI Act and bodies like NIST scrutinise them.

Are foundation models always built on transformers?

Almost always, today. The overwhelming majority of current foundation models — across text, images and audio — are built on the transformer architecture introduced in 2017. The term "foundation model" refers to the training-and-adaptation paradigm, though, so it does not strictly require a transformer.

Related

Explainers: LLM, Transformer

Glossary: Pretraining, Fine-tuning, Large language model (LLM), Transformer (architecture), Neural network

Sources

• arXiv — On the Opportunities and Risks of Foundation Models (Bommasani et al., 2021)
• Stanford CRFM — Center for Research on Foundation Models
• IBM Think — What are foundation models?
• NIST — AI Risk Management Framework