AI Safety and Alignment: The Alignment Problem, Methods and Global Governance

Executive summary

AI safety and alignment are the fields concerned with ensuring that increasingly capable artificial-intelligence systems behave as their designers intend and do not cause harm. AI alignment is the narrower technical problem of getting a model to pursue the goals and values humans actually want, rather than a flawed proxy of them; AI safety is the broader discipline spanning that alignment work, adversarial testing, monitoring, and governance. The problem is hard because modern models are trained, not programmed: their objectives are learned from data and human feedback, and a system optimising a specification can satisfy the letter while violating the intent. The main technical toolkit today includes reinforcement learning from human feedback (RLHF), Anthropic's Constitutional AI, mechanistic interpretability, red-teaming and structured capability evaluations. Leading labs also publish frontier safety frameworks — Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework — that tie deployment to dangerous-capability thresholds in areas such as biosecurity, cyber and autonomous self-improvement. Governments have moved from the Bletchley Declaration (November 2023) to the Seoul Frontier AI Safety Commitments (May 2024) and a network of AI Safety Institutes, and the International AI Safety Report led by Yoshua Bengio (January 2025) now offers a scientific consensus baseline. Whether AI safety is 'solved' is contested: Stanford's AI Index 2026 records rising incidents and safety that collapses under jailbreak testing, while experts split between those focused on present-day harms and those warning of catastrophic or existential risk. This report is not investment or safety advice.

Key findings

What are AI safety and AI alignment?

AI safety is the field of research and practice concerned with preventing artificial-intelligence systems from causing harm, while AI alignment is the narrower problem of ensuring that an AI system actually pursues the goals and values its human designers intend. The two terms are often used together because alignment is widely seen as the central technical challenge within the broader safety agenda, which also spans robustness, monitoring, security and governance.

The distinction matters. Alignment is about intent: does the system want what we want, and does it reliably act on that? Safety is about outcomes: even a well-intentioned system can be unsafe if it is brittle, easily jailbroken, or deployed without oversight. NIST's AI Risk Management Framework (AI RMF 1.0), published on 26 January 2023, organises this broader effort around four functions — Govern, Map, Measure and Manage — and is the most widely referenced voluntary standard for operationalising AI risk.

AI safety spans time horizons too. 'Near-term' safety addresses harms already visible in deployed systems — bias, hallucination, misinformation, privacy and security failures. 'Frontier' or 'catastrophic' safety addresses risks from the most capable general-purpose models, including large-scale misuse (for example, uplift to biological or cyber weapons) and loss of human control over highly autonomous systems.

What is the alignment problem?

The alignment problem is the challenge of specifying and instilling human goals in an AI system so precisely that the system does what we mean rather than what we literally said. It arises because modern AI models are trained, not hand-coded: their objectives are inferred from data and feedback, so a model can optimise a measurable proxy that diverges from the designer's true intent — a failure researchers call specification gaming or reward hacking.

Two sub-problems are usually distinguished. 'Outer alignment' is the difficulty of writing down an objective that genuinely captures what humans want; almost any simple objective, pursued relentlessly, produces unintended side effects. 'Inner alignment' is the risk that a system trained on one objective develops internal goals that pursue something subtly different, especially in situations unlike its training data. As models become more capable and autonomous, small misalignments can compound into larger, harder-to-correct behaviour.

The concern sharpens for highly capable, goal-directed systems. Researchers such as Stuart Russell (UC Berkeley) argue that a sufficiently capable system optimising a misspecified objective could resist correction or acquire resources as instrumental sub-goals — not from malice, but as a predictable consequence of competent optimisation. This is why alignment is treated as a prerequisite for, not an afterthought to, deploying frontier AI.

How do researchers try to align AI systems?

Researchers align AI systems today mainly through learning from feedback, rule-based training, and inspection of a model's internals. The dominant method is reinforcement learning from human feedback (RLHF), in which humans rank model outputs, a reward model is trained on those rankings, and the base model is fine-tuned to maximise the reward — the technique behind most commercial chat assistants since 2022.

Anthropic's Constitutional AI, introduced in a December 2022 paper, replaces much of the human labelling with AI feedback (RLAIF): the model critiques and revises its own outputs against an explicit written 'constitution' of principles, and Anthropic reports this produces models that are more harmless than RLHF-only baselines without needing human preference labels for the safety stage. A third, more foundational approach is mechanistic interpretability — reverse-engineering what computations a model performs internally — which aims to detect deception or dangerous capability directly rather than inferring safety from behaviour alone.

None of these methods is a complete solution. Feedback-based training can teach a model to appear aligned (to satisfy raters) without being aligned, and interpretability remains far from being able to fully audit frontier models. The table below summarises the main techniques, what each does, and who uses them.

What are red-teaming and evaluations in AI safety?

AI red-teaming is a structured effort to find flaws and vulnerabilities in an AI system using adversarial methods to identify harmful or discriminatory outputs, unforeseen behaviour, or misuse risks — a definition drawn from the US Executive Order on AI and adopted by NIST. In practice, red-teamers try to 'jailbreak' models past their safeguards, elicit dangerous instructions, or trigger failures that ordinary testing would miss.

Capability evaluations ('evals') are the complementary, more systematic practice of measuring what a model can do against defined danger thresholds — for example, whether it can meaningfully assist in designing a pathogen, execute an autonomous cyberattack, or improve its own code. Evals are the technical machinery that frontier safety frameworks rely on: a lab commits to running specific tests and to withholding or restricting deployment if a model crosses a threshold. Government institutes now run independent pre-deployment evaluations, and NIST's CAISI has published research from large-scale red-teaming competitions against agentic AI systems.

The limitation, documented by Stanford's AI Index 2026, is that safety measured under standard conditions does not survive adversarial pressure: on the AILuminate benchmark, models rated 'Good' or 'Very Good' in normal use degraded across the board when jailbreak prompts were applied. Evaluations also lag capabilities — most developers report capability benchmarks like MMLU and SWE-bench far more consistently than responsible-AI benchmarks.

What are frontier AI safety frameworks and responsible scaling?

A frontier safety framework is a public policy in which an AI developer commits to testing its most capable models for dangerous capabilities and to gating deployment (and sometimes further training) behind defined risk thresholds. Anthropic pioneered the approach with its Responsible Scaling Policy (RSP), which defines AI Safety Levels (ASL-1 to ASL-4+) modelled on biosafety levels; ASL-3, for instance, requires stronger security and a commitment not to deploy if catastrophic-misuse risk is evident under adversarial testing.

OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework, both updated in 2025, follow the same logic with different vocabulary — OpenAI uses 'High' and 'Critical' capability tiers, DeepMind uses 'Critical Capability Levels' (CCLs). Independent comparisons find the three track broadly the same risk areas (CBRN/bio-chem, cyber-offence and AI self-improvement) and all condition deployment on evaluations, but they differ in how prescriptive and binding they are: the RSP pre-commits to specific security postures at each level, while DeepMind's framework is more process-focused on what it will measure and review.

These frameworks were reinforced at the political level by the Seoul Frontier AI Safety Commitments (May 2024), under which 16 companies pledged to publish safety frameworks defining intolerable risks and the thresholds they will not cross. A recurring critique — echoed in the academic literature and by Stanford's AI Index — is that the commitments are voluntary, emphasise misuse over misalignment, and lack external enforcement.

What are AI Safety Institutes and how is AI safety governed globally?

AI Safety Institutes are government bodies created to test frontier AI models and inform policy, and they sit at the centre of an emerging international governance network that began with the UK's AI Safety Summit at Bletchley Park on 1–2 November 2023. There, 28 countries plus the European Union signed the Bletchley Declaration, the first intergovernmental statement recognising the risks of frontier AI and committing to cooperate on safety.

The AI Seoul Summit (21–22 May 2024) extended this into corporate commitments, and the International AI Safety Report — commissioned by the Bletchley nations, chaired by Yoshua Bengio and written by 96 experts — published on 29 January 2025 to give policymakers a shared scientific baseline. But the emphasis has shifted: the UK renamed its institute the AI Security Institute on 14 February 2025, and the US renamed its NIST-based institute the Center for AI Standards and Innovation (CAISI) on 3 June 2025, both signalling a pivot from broad 'safety' toward security-relevant risks and innovation. The table below maps the key institutions and policies to their roles.

These bodies mostly operate through voluntary testing agreements rather than binding law; the principal binding regime is the EU AI Act, which is complementary to, rather than part of, this institute network.

Is AI an existential risk, or a source of near-term harm?

The AI risk debate divides between those who prioritise near-term harms already occurring and those who warn of catastrophic or existential risk from future systems — and the disagreement is about emphasis and probability, not whether AI carries risk. On the catastrophic-risk side, the Center for AI Safety's Statement on AI Risk (30 May 2023) declared that 'mitigating the risk of extinction from AI should be a global priority', signed by Geoffrey Hinton and Yoshua Bengio (both Turing Award laureates), Stuart Russell, and the CEOs of OpenAI, Google DeepMind and Anthropic.

On the near-term side, researchers including Timnit Gebru, Emily M. Bender and Margaret Mitchell — associated with the 'Stochastic Parrots' critique — argue that focusing on speculative extinction can distract from concrete, present-day harms: algorithmic bias, misinformation, labour displacement, surveillance and concentration of corporate power. Prominent scientists such as Meta's Yann LeCun and Andrew Ng are sceptical that existential risk is imminent or well-grounded, viewing today's systems as far from the autonomous agency such scenarios assume.

Both camps largely agree on the near-term harms; they diverge on how much weight to give low-probability, high-severity tail risks and on how soon frontier systems might become dangerously capable. The table below maps major risk categories to their principal mitigations.

Is AI safety being solved?

AI safety is not solved, and by several measures the gap between capability and safety is widening rather than closing. Stanford's AI Index 2026 reports that documented AI incidents rose to 362 in 2025 from 233 in 2024, that the Foundation Model Transparency Index fell to 40 out of 100 (from 58 in 2024), and that responsible-AI benchmarking remains sparse compared with capability benchmarking.

There is genuine progress: RLHF and Constitutional AI make deployed assistants markedly more helpful and harmless than raw base models; interpretability research has begun to identify meaningful internal features; and every leading lab now publishes a frontier safety framework and submits models for external evaluation. Governments have built institutes and a scientific consensus process that did not exist before 2023.

But the honest assessment is that current methods manage risk rather than resolve it. Safety degrades under adversarial pressure, alignment techniques cannot yet guarantee that a model's apparent compliance reflects genuine intent, and the strongest commitments remain voluntary and unenforced. The International AI Safety Report frames the field as one of deep and unresolved scientific uncertainty about the risks of the most capable systems.

What does AI safety and alignment mean for organisations?

For organisations deploying AI, the practical meaning of the safety agenda is that risk management is now an expected, and increasingly regulated, part of using AI — not an optional extra. The NIST AI RMF provides a voluntary but widely adopted structure (Govern, Map, Measure, Manage), while the EU AI Act converts parts of the same logic into binding obligations for high-risk and general-purpose systems.

The concrete implications are procedural: identify where AI systems are used and how they could fail; test them adversarially (red-teaming) before and after deployment; keep humans in oversight roles for consequential decisions; and demand transparency and safety documentation from model providers, whose frontier safety frameworks and evaluation results are becoming a due-diligence input. For providers of the most capable models, the direction of travel is toward capability thresholds, pre-deployment evaluation and — in the EU — legal duties.

Strategically, the unresolved state of alignment argues for humility: the leading laboratories themselves describe safety as an open problem, so organisations should treat frontier AI as powerful but fallible, design for failure, and monitor the fast-moving governance landscape. This report is journalism and analysis, not investment or safety-engineering advice.

Scoreboard (machine-readable data)

Every headline indicator with its value, period, source and confidence. Free to reuse under CC BY 4.0.

↓ CSV · ↓ JSON

Methodology & verification

This report synthesises primary and authoritative secondary sources on AI safety and alignment as of 30 July 2026. Institutional dates and facts were verified against government publications (UK GOV.UK for the Bletchley Declaration and Seoul commitments; NIST for the AI RMF and CAISI), the labs' own published safety frameworks (Anthropic Responsible Scaling Policy, OpenAI Preparedness Framework, Google DeepMind Frontier Safety Framework), the Center for AI Safety's Statement on AI Risk, the International AI Safety Report (Bengio et al.), and the Stanford HAI AI Index 2026 Responsible AI chapter for quantitative metrics. The May 2023 Statement on AI Risk is quoted verbatim. Technical definitions of RLHF, Constitutional AI, red-teaming and evaluations follow the labs' publications and NIST usage. Where the field is genuinely contested — notably the existential-risk debate — named voices are given on each side and no figure was invented or estimated.

Data dictionary

Frequently asked questions

Glossary

AI alignment

The problem of ensuring an AI system reliably pursues the goals and values its human designers intend, rather than a flawed proxy of them. ↗

RLHF (reinforcement learning from human feedback)

A training method that fine-tunes a model to match human preference rankings via a learned reward model; the dominant alignment technique behind commercial chat assistants. ↗

Responsible Scaling Policy (RSP)

Anthropic's frontier safety framework defining AI Safety Levels (ASL-1 to ASL-4+) that gate model deployment and security behind dangerous-capability thresholds. ↗

Red-teaming

Structured adversarial testing to find flaws, jailbreaks and misuse risks in an AI system, as defined by the US Executive Order on AI and NIST. ↗

Sources

• Stanford HAI — AI Index 2026, Responsible AI chapter
• NIST — AI Risk Management Framework (AI RMF 1.0)
• Center for AI Safety — Statement on AI Risk
• UK Government — Bletchley Declaration (AI Safety Summit 2023)
• UK Government — Frontier AI Safety Commitments (AI Seoul Summit 2024)
• International AI Safety Report (Bengio et al., 2025)

Related reports & data

• The EU AI Act, explained
• What is an AI agent
• National AI Power Index