An AI Safety Institute is a government-run body that stress-tests powerful AI models for dangerous capabilities — like the UK AI Security Institute’s recent discovery of a “universal jailbreak” in OpenAI’s GPT-5.6 — and reports what it finds back to policymakers and, often, to the company that built the model. Roughly a dozen governments now run one.
What an AI Safety Institute actually does
Three jobs, roughly:
- Testing frontier models. Institutes run structured red-teaming against the most capable AI systems, before and after they ship. Testers try to get a model to help with cyberattacks, weapons development, or other harmful tasks despite its safety training — a process closer to a security audit than a product review.
- Publishing research on what AI can actually do. Beyond hunting for failures, institutes measure raw capability over time — how good models have gotten at writing working exploit code, for instance — and publish the results so policymakers aren’t relying on vendor claims alone.
- Advising government. Findings feed into export-control decisions, procurement rules, and whatever binding AI law a country is drafting, such as the EU AI Act.
None of this makes an institute a regulator in the traditional sense. It can’t fine a company or block a product launch. Its power is closer to that of an independent auditor: it publishes what it finds, and companies feel pressure to respond because the finding is public and credible.
Where the idea came from
The model traces to the UK and US, which each stood up an institute around the first global AI Safety Summit at Bletchley Park in November 2023. The UK’s institute grew out of an earlier Frontier AI Taskforce; the US version was created inside the National Institute of Standards and Technology (NIST). At that summit, several leading AI labs — including OpenAI, Google DeepMind and Anthropic — agreed to give the UK and US institutes early access to models before public release.
Six months later, at the AI Seoul Summit in May 2024, participating governments agreed to link their institutes into an international network so safety testing wouldn’t be reinvented separately in every country. Members now include the UK, US, Japan, France, Germany, Italy, Singapore, South Korea, Australia, Canada, the European Union, and more recently Kenya and India.
Same idea, different names
The institutes don’t all look alike. The UK’s, now called the AI Security Institute (renamed from “Safety” to “Security” in 2025 to emphasize national-security risks like cyber and bio misuse), sits inside the Department for Science, Innovation and Technology with roughly £66 million in annual funding and access to over £1.5 billion in compute. The US equivalent was renamed the Center for AI Standards and Innovation (CAISI) in June 2025; its mission shifted from safety-first evaluation toward “pro-innovation” voluntary standards, though it kept the national-security testing function. Japan’s, Singapore’s, and other institutes largely follow the UK model — a small technical team embedded in government, working directly with AI labs rather than through formal regulation.
Why it matters
Before institutes like these existed, the only public information about how safe a frontier model was came from the company that built it — a conflict of interest, since the same lab deciding whether to ship a model also decided whether to disclose its flaws. AI Safety Institutes exist to break that loop: an independent, technically capable tester with no commercial stake in the release date.
That independence is also their biggest limitation. Because participation is voluntary and institutes have no enforcement power, a lab can decline early access, or ship anyway after a bad finding. What institutes have proven able to do is create enough public pressure — and enough of a paper trail — that labs generally do fix what’s found, and governments use the findings to justify tougher rules later.
In the news
That dynamic played out directly in July 2026, when the UK AI Security Institute reported finding a “universal jailbreak” in OpenAI’s GPT-5.6 — a way to strip the model’s safeguards that let it move from merely describing a software vulnerability to autonomously writing and running an exploit for it. AISI testers reportedly found the jailbreak within hours. It’s the kind of finding the institute model is built to produce: independent, public, and hard for the vendor to wave away.
FAQ
Is an AI Safety Institute the same as a regulator?
No. It has no power to fine companies or block a release. It evaluates and publishes; enforcement, where it exists, comes from separate laws like the EU AI Act.
Do AI companies have to submit their models for testing?
Mostly not by law — participation has largely been voluntary, based on commitments made at events like the Bletchley Park summit rather than binding statute.
Is red-teaming the same thing as what an AI Safety Institute does?
Red-teaming — deliberately attacking a system to find its weaknesses — is one tool institutes use, alongside capability research and policy advice.
Does every country have one?
No. About a dozen governments run one today, coordinated loosely through the International Network of AI Safety Institutes formed after the 2024 Seoul Summit.
Sources: AI Security Institute (AISI), UK government; Artificial intelligence safety institute, Wikipedia; AISI Frontier AI Trends Report, GOV.UK.