AI & Machine Learning

Foundation Model

A foundation model is a large model trained on broad data, usually with self-supervised learning, that can be adapted to many downstream tasks through prompting or fine-tuning. Large language models and vision-language models are examples.

The term was popularised by Stanford researchers in 2021 to describe models such as BERT, GPT-3 and CLIP whose general capabilities transfer across tasks. Pre-training on large, diverse datasets of text, images, code or other data produces general representations, and adaptation then specialises the model through prompts, retrieval, fine-tuning or added task-specific layers.

Foundation models let organisations build applications without training models from scratch, including chat assistants, document understanding, code generation, image analysis and, increasingly, time-series forecasting and robotics. Open-weight models can be self-hosted, which matters where data must stay on premises.

Because one model underlies many applications, its flaws, biases and security weaknesses propagate downstream. The EU AI Act sets obligations for providers of general-purpose AI models, a closely related concept. Selection considers capability, licence, data handling, cost, latency and whether the model can run on the available hardware.

Key points

• A broadly pre-trained model adaptable to many downstream tasks
• Term popularised by Stanford researchers in 2021
• Adapted through prompting, retrieval or fine-tuning
• Flaws and biases propagate to every application built on it

Related terms

• Large Language Model (LLM)AI & Machine Learning
• Vision-Language Model (VLM)AI & Machine Learning
• Transfer LearningAI & Machine Learning
• Fine-TuningAI & Machine Learning
• Generative AIAI & Machine Learning

Terms that refer to Foundation Model

• Diffusion ModelAI & Machine Learning
• Unsupervised LearningAI & Machine Learning