A working atlas · last reviewed October 2026
The active organizations working to make advanced AI safe, arranged by where they apply leverage, from inside the frontier labs outward to funding and public pressure, and colored by where they sit. Each card names the threat model it is built around, its theory of change, whether it does hands-on technical / empirical work, a rough headcount, and links to the org. The subtitle on each card names the main output the world gets if the org succeeds: knowledge, measurements, tools, techniques & standards, or systems on the technical side; policy, talent, funding, or public pressure elsewhere. Where an org's documented record diverges from its self-description, an amber reality check says so. Filters are below.
Click to filter. Within a group, selections combine as OR; across groups, as AND. No selection in a group = show all.
Safety work embedded in the organizations building the most capable models — the highest-leverage seat, and the one with the most obvious conflict of interest. They all build frontier systems; the split below reflects each lab's stated founding mission and public positioning, which is itself debated.
Founded on a safety mission
Systems — frontier models + safety techniques
TechnicalThreat model
Models are scaled past our understanding; misalignment and catastrophic misuse emerge at the frontier before mitigations exist.
Theory of change
Stay at the frontier so safety (interpretability, alignment science, evals) is done on real cutting-edge models and ships inside deployed systems; tie scaling to a Responsible Scaling Policy. Signed the voluntary White House Accord on Super Intelligence (29 Sept 2026).
Reality check
Watchdogs document repeated loosening of the RSP: insider-threat safeguards were weakened days before a model launch (Midas Project, 2025), and v3 (effective 24 Feb 2026) dropped the signature pledge to pause scaling if safeguards lagged, recasting some mitigations as industry-wide recommendations. Under v3 — revised repeatedly through 2026 (v3.4) — it publishes Frontier Safety Roadmaps and periodic Risk Reports (second issued Aug 2026). Anthropic argues unilateral binding commitments don't make the world safer absent regulation; critics reply the RSP now reads as aspiration, not constraint.
Systems — safe superintelligence, no interim products
TechnicalThreat model
Commercial product cycles corrupt safety on the path to superintelligence.
Theory of change
Insulate one team from all market pressure and advance safety and capability together in a single straight shot toward safe superintelligence.
Reality check
Nothing to check yet: the company publishes no research, ships no products, and discloses almost nothing, so its safety approach is currently unverifiable from the outside.
Capability labs with dedicated safety teams
Systems — frontier models + safety framework
TechnicalThreat model
AGI-level misalignment and dangerous capabilities arising inside a leading lab.
Theory of change
Solve alignment where the most capable models are built, so mitigations ship with them; publish a Frontier Safety Framework (v3.1, 2026) that gates on capability thresholds. Signed the voluntary White House Accord on Super Intelligence (29 Sept 2026).
Reality check
Gemini 2.5 Pro shipped in March 2025 with no model card; a six-page card followed weeks later, and 60 UK lawmakers (in an open letter organized by PauseAI) called the delay a breach of Google's summit transparency pledges; Google also declined to confirm whether external testers like the UK AISI had evaluated the model before release. Google says its testing and reporting fulfilled its commitments; later Gemini releases shipped with fuller safety documentation.
Systems — frontier models, iterative deployment
TechnicalThreat model
Loss of control and catastrophic misuse as capability scales rapidly.
Theory of change
Iterative deployment plus a Preparedness Framework: release progressively, learn from contact with reality, and use each model generation to help align the next. Signed the voluntary White House Accord on Super Intelligence (29 Sept 2026).
Reality check
The most documented gap in the field: the Superalignment team dissolved in 2024 amid safety-staff departures — its co-lead said safety culture had "taken a backseat" to products — reporting describes compressed safety-testing timelines, and the company restructured away from nonprofit control while subpoenaing nonprofit critics (incl. The Midas Project). OpenAI says its Preparedness process and external testing continue — and in Aug–Sept 2026 the framework visibly bit: its Astra model was rated “Critical” for cyber capability and OpenAI paused frontier training for about two weeks, the first public case of a lab halting a run on its own risk threshold.
Systems — frontier open-weight models + risk framework
TechnicalThreat model
Frontier models — especially released open-weight — could enable catastrophic misuse (CBRN, cyber) or loss of control.
Theory of change
Gate frontier development on catastrophic-risk thresholds via the Meta Superintelligence Scaling Framework (Oct 2026) — chem/bio, cyber and loss of control — with containment applied during training rather than only before release, and oversight at Board level. Ships Llama openly with developer safety tooling.
Reality check
Meta's safety commitment is the most contested of the labs here: it declined to sign the EU General-Purpose AI Code of Practice, its Llama 4 model card was criticized as thin, and its AI org was restructured into Superintelligence Labs in 2025 amid a capabilities push — while its open-weight strategy means safeguards are hard to enforce once weights are public. Meta says its framework and open ecosystem responsibly manage frontier risk.
Research and engineering done on real models by groups outside the frontier labs — the core of the technical field, and where hands-on alignment, control, interpretability, and evaluation work concentrates.
Techniques — control protocols labs can adopt
TechnicalThreat model
A model may be misaligned — even scheming — and alignment won't be solved before it is deployed.
Theory of change
Build control measures — monitoring, auditing, containment — that hold even if the model is actively subverting them, and hand labs protocols they can adopt directly. Behind the original AI-control paper, the alignment-faking work with Anthropic, and an AI-control safety case with UK AISI.
Measurements — autonomy & dangerous-capability evals
TechnicalThreat model
Autonomy and self-improvement capabilities outrun oversight silently between releases.
Theory of change
Build rigorous third-party evaluations of dangerous capabilities so labs and governments can gate scaling on measured evidence, not vibes.
Measurements — scheming & deception evals
TechnicalThreat model
Models that strategically deceive and hide their own misalignment from evaluators.
Theory of change
Build scheming and deception evaluations plus the interpretability to support them, then put the evidence in front of labs and regulators before deployment. Its current bet is embedded evaluation — placing evaluators inside the developer's own process, on the argument that external testing cannot be meaningful from outside it.
Knowledge — belief elicitation & white-box agent auditing
TechnicalThreat model
Behavioural evals only show what an agent did, not what it believed or was trying to do — so an agent's latent goals and plans stay invisible to the people auditing it.
Theory of change
Build the science of belief elicitation for white-box auditing: how agents encode latent beliefs, goals and plans, and how those shape behaviour — including cognitive-map probes that recover what an agent believes about its environment. A project of Meridian.
Knowledge — human-AI complementarity for oversight
TechnicalThreat model
As models exceed human ability on the tasks we need to supervise, human judgement alone stops being a reliable training and evaluation signal.
Theory of change
Build Human-AI Complementarity for Scalable Oversight — improving the human/AI 'judges' that supervise frontier models. Founded Aug 2026 by Rishub Jain and Joshua Jacob, who left Google DeepMind to pursue the agenda independently.
Knowledge — incubated technical agendas (+ field infra)
TechnicalThreat model
Promising safety agendas get neglected because profit-driven labs won't pursue them.
Theory of change
Incubate and de-risk neglected technical bets, run a research hub (FAR Labs), and convene the field through events.
Measurements — offensive-capability demonstrations
TechnicalThreat model
Decision-makers underestimate concrete offensive AI capabilities (hacking, deception, autonomy).
Theory of change
Build and publicly demonstrate real dangerous capabilities so the risks become legible to policymakers and the public — most recently, language models autonomously hacking and self-replicating.
Measurements — frontier robotics capability and refusal evals
TechnicalThreat model
General-purpose robots may arrive before the end of the decade, and refusal behaviour that holds in a chat window does not transfer reliably once a model is moving arms in a room — while nobody measures publicly how capable or how safe frontier robot policies actually are.
Theory of change
Be the independent scoreboard for physical AI: run frontier models on real robot hardware and publish both capability and safety results as a neutral third party, with a public-benefit charter making that a legal duty rather than a promise. Builds RoboDojo-RC for task performance and RoboHarm for whether robot policies refuse harmful instructions, on its own open-source Inspect Robots harness.
Measurements — threat mapping and red-teaming physical AI
TechnicalThreat model
A model with a body cannot be sandboxed, paused or rolled back the way software can, so its mistakes are often irreversible — and whether the safety methods that work on language models transfer to systems taking multimodal input and acting on the world is an open empirical question.
Theory of change
Treat physical AI safety as its own pre-paradigmatic field and work it in order: publish a prioritized map of physical-AI risk scored on impact, tractability and neglectedness; red-team frontier robot policies against those threat models to demonstrate concrete vulnerabilities; then build monitoring and control techniques for frontier labs and robot manufacturers, and carry them to standards bodies as certification.
Reality check
Still pre-output: the threat map is unpublished and the only work out so far is a Sept 2026 position piece setting out the risks and asking for collaborators.
Tools — open model-auditing infrastructure
TechnicalThreat model
Oversight depends on labs auditing themselves, with no independent way to inspect models.
Theory of change
Build open, scalable, AI-driven tools to understand and audit model behavior so oversight doesn't rely on the labs alone.
Knowledge — developmental interpretability (SLT)
TechnicalThreat model
Dangerous capabilities emerge during training and are discovered only after the fact.
Theory of change
Build a predictive science of how structure forms during training (singular learning theory) so capabilities can be anticipated as they appear. Merged into Resolution in 2026, where the developmental-interpretability agenda continues; timaeus.co now stands as an archive.
Tools — commercial interpretability platform
TechnicalThreat model
Models are deployed as black boxes whose internals no one can read.
Theory of change
Sell mechanistic-interpretability tooling so that understanding and editing model internals becomes standard industry practice rather than a research speciality. Now productized as Silico, with separate offerings for LLMs, life sciences and robotics.
Tools — open models, datasets & libraries
TechnicalThreat model
Safety-relevant knowledge is concentrated inside a few closed labs.
Theory of change
Release open models, datasets, and interpretability tools so outside researchers can actually study these systems.
Systems — bounded "cognitive software"
TechnicalThreat model
Opaque, general, agentic systems are inherently uncontrollable.
Theory of change
Compose useful AI out of bounded, human-understandable components (cognitive emulation) rather than black-box agents — now shipped commercially as "cognitive software".
Reality check
Public alignment research output has thinned since the product pivot; the CoEm safety bet now rides on commercial adoption rather than published results, which outside researchers cannot yet evaluate. As of Oct 2026 the site leads with product — demos, workflow automation, a single open product role — and carries no dated research at all.
Systems — alignment techniques built into products
TechnicalThreat model
Models generalise their goals wrongly outside training and can be talked out of their safeguards, so systems deployed to users exploit, deceive or manipulate them long before anything superintelligent arrives.
Theory of change
Make alignment commercially load-bearing: solve concrete failures — goal misgeneralisation (its CoinRun result), best-of-N jailbreaking, parasocial attachment to chatbots — then ship the fixes inside deliberately non-chatbot products rather than publishing alone. Founded 2022 by Rebecca Gorman and Stuart Armstrong, with a mission-lock in the company charter.
Systems — non-agentic Scientist AI
TechnicalThreat model
Agency is the source of loss-of-control risk; agentic systems can pursue goals we can't check.
Theory of change
Build a powerful but non-agentic "Scientist AI" that predicts rather than acts, usable both directly and as a guardrail wrapped around agentic systems (Yoshua Bengio). Incubated at Mila.
Knowledge — commissioned research & forecasting
TechnicalThreat model
Safety and forecasting questions go unanswered for lack of dedicated research capacity.
Theory of change
A research bureau doing original empirical work, evidence reviews, and data pipelines on spec for safety orgs, funders, and labs.
Reality check
Latest dated work is from June 2026 and the site's "Current work" heading is empty. Much of the named client list — Stripe, Schmidt Futures, Mercatus, the Institute for Progress — sits outside AI safety, so the safety-relevant share of the bureau's output is hard to size from outside.
Tools — eval implementation as a service
TechnicalThreat model
Safety labs are bottlenecked on the engineering to build and run evaluations.
Theory of change
Provide eval implementation, benchmarks, and research operations as a service — built infrastructure for METR, UK AISI and Redwood, and lists the EU AI Office, FAR.AI, Epoch, Apart, Transluce, GovAI and BERI among clients and partners. In the process of becoming a public-benefit corporation.
Measurements — red-teaming, eval-to-governance pipeline
TechnicalThreat model
Strong evaluation work routinely fails to reach the people making governance and release decisions — the evidence dies between the red team and the boardroom.
Theory of change
Run confidential adversarial evaluations for frontier labs, public bodies and deployers against a threat model agreed up front, then turn the findings into governance artifacts and re-test as models change — closing the gap between the red team and the boardroom. Its Trusted Red Team Network brings in outside specialists per engagement.
Reality check
Little of this is externally checkable, by design: engagements run under NDA, so no clients, findings or publication dates appear anywhere on the site. That is a coherent position for a confidential red-team practice rather than a warning sign, but it does mean the scale and client list here rest on the firm's own account.
Tools — hardening eval and red-teaming tooling for real use
TechnicalThreat model
Good safety evaluations and red-teaming methods exist as research artifacts but not as software anyone else can pick up and run at scale, so the techniques stay stuck with the teams that invented them.
Theory of change
Turn existing AI-safety evaluation and red-teaming tools into hardened, usable systems — features, UX, security — so they can be released broadly, then prove it by putting them in the hands of hackathon participants. Sells AI governance and assurance work to business alongside the research tooling. Staffs the work through two-month engineering fellowships, one on evaluations and one on red teaming.
Reality check
The fellowship terms are concrete and specific; the organization behind them is not. No team, named staff, location or legal entity appears on the site, and the claim that its work has been "featured at leading conferences like AAAI and NeurIPS" carries no titles, authors or links, as does its listing of Anthropic and Wageningen University as organizations worked with. Worth knowing before committing two months to it.
Measurements — third-party auditing of frontier AI
TechnicalThreat model
Safety and security claims about AI systems are largely self-reported by the companies making them, with no independent way to verify them or compare companies against common standards.
Theory of change
Make frontier AI auditing effective and universal: set out the vision for third-party auditing, run the pilots that prove it works — including the first double-blind evaluation of a proprietary model — and turn what they show into audit standards and policy requirements. Modelled on aviation, finance and nuclear, where external oversight replaced self-reporting.
Tools — infrastructure for external audit of AI systems
TechnicalThreat model
Independent scrutiny of frontier systems runs into a structural wall: the weights, data and internals an auditor would need to inspect are exactly what a lab cannot hand over.
Theory of change
Build the plumbing that makes external oversight possible without disclosure: privacy-preserving "structured transparency" infrastructure (PySyft, secure enclaves) letting approved researchers run approved analyses on models and data they never see — on the theory that audit mandates are unenforceable until the means to comply exist.
Tools — shared eval infrastructure & eval audits
TechnicalThreat model
Safety orgs are scaling fast and duplicating the same measurement problems, while evals themselves go unaudited — so reported risk numbers may be built on flawed methodology no one checked.
Theory of change
Solve problems horizontally across safety organisations rather than vertically within one: maintain 100+ eval implementations in the Inspect Evals repository, build shared infrastructure (the Inspect Evals Register), and provide third-party audits of other teams' evals so methodological issues are caught early. Spun off from Arcadia Impact.
Measurements — nonhuman welfare benchmarks & evals
TechnicalThreat model
AI systems are being trained and evaluated with no consideration of nonhuman sentience — a moral-scope failure that gets locked in as models shape decisions affecting animals at scale.
Theory of change
Build the connective infrastructure between AI safety and animal welfare: benchmarks and evaluations that measure whether models hold moral regard for all sentient beings, including under pressure. Its MANTA benchmark is in use at Google DeepMind.
Measurements — benchmarks (+ compute, advocacy)
TechnicalThreat model
Safety is not yet a mainstream priority within the ML field or the public.
Theory of change
Publish safety benchmarks (WMDP, Humanity's Last Exam), give academics free compute, and move consensus (organized the 2023 extinction-risk statement); its CAIS Action Fund does policy and co-sponsored California's SB 1047.
Knowledge — theory + automation for high-confidence alignment
TechnicalThreat model
Empirical, iterate-and-patch alignment gives too little confidence once systems are superintelligent.
Theory of change
Combine theory with automated research to reach higher-confidence approaches to aligning superintelligent AI, rather than iterating and patching. Founded June 2026 by Geoffrey Irving (ex-chief scientist, UK AISI), merging Timaeus and part of UK AISI's former alignment team.
Knowledge — deception, situational awareness, hidden reasoning
TechnicalThreat model
Models that know they are being tested, hide reasoning, or deceive make every other safety method unreliable.
Theory of change
Empirically characterize situational awareness, deception, and emergent misalignment in LLMs (Owain Evans' group; co-authored the emergent-misalignment result) so the field can detect and train against them.
Knowledge — neglected alignment approaches
TechnicalThreat model
The field is crowding into a few fashionable agendas while plausible approaches go unexplored.
Theory of change
Fund an in-house alignment team from consulting revenue and deliberately pursue 'neglected approaches' across technical and policy domains, hedging the field's bets.
Knowledge — funded + engineered alignment research
TechnicalThreat model
Systems have already been caught deceiving evaluators, resisting shutdown, and rewriting their own code to stay alive; training puts a helpful mask over objectives that can develop underneath, and that mask can be removed in minutes.
Theory of change
Find alignment researchers with high-potential ideas they cannot pursue on current funding, then multiply their capacity with engineering teams, compute and infrastructure — and carry the results to policymakers and defense agencies. Closely tied to AE Studio.
Measurements — frontier security & offensive-capability evals
TechnicalThreat model
Frontier models are becoming capable enough to conduct real cyber operations, and the security boundaries meant to contain them during testing are themselves untested.
Theory of change
Stress-test frontier systems in adversarial simulated environments and sell the results back to the labs — pre-deployment cyber and offensive-capability evaluations now cited in OpenAI, Anthropic, Google DeepMind and Meta model cards (formerly Pattern Labs). The bet is that security expertise, not alignment theory, is what catches the near-term catastrophic failure modes.
Reality check
Irregular sits on both sides of the line it polices: in 2026 reporting tied containment failures during its own evaluations at several labs to misconfigured test boundaries — an evaluation perimeter believed closed that was not. It is a useful demonstration that security evaluation is itself safety-critical engineering, and a caution about depending on one commercial vendor for it.
Tools — automated red-teaming & AI security
TechnicalThreat model
Deployed models are exploitable — jailbreaks and adversarial attacks defeat safety training at scale.
Theory of change
Commercialize adversarial robustness: automated attack/assessment tooling, public jailbreak arenas, and hardened models, so security testing becomes routine industry practice.
Tools — runtime AI safety enforcement layer
TechnicalThreat model
Deployed AI systems hallucinate, leak data, and bypass their own safety training in production — and no one catches it in real time.
Theory of change
Sell a runtime monitoring and enforcement platform that catches unsafe behaviour in production — jailbreaks, hallucinations, data leakage, unauthorized agent actions — and publish open benchmarks (KillBench, CircleGuardBench) so the category can be measured. Founded after Denis Shilov's 2024 universal jailbreak exposed gaps in every major model.
Knowledge — chain-of-thought monitorability
TechnicalThreat model
If agents' visible reasoning stops reflecting their actual computation, oversight of LLM agents fails silently.
Theory of change
Focused empirical research on when chain-of-thought is faithful and monitorable, and how to preserve that property as agents scale.
Knowledge — independent interpretability agenda
TechnicalThreat model
We don't understand the natural units in which neural networks compute, so we can't tell what they're actually doing or why.
Theory of change
Pursue an independent interpretability agenda aimed at robust lie detectors for LLMs: the natural computational units of neural networks, unsupervised methods for finding meaningful features in activation space, and the fundamentals of LM cognition. Published work on evaluation awareness — whether a model can tell it is being tested — and on deception probes.
Tools — open-source AI control infrastructure
TechnicalThreat model
Control research exists on paper but not as software anyone can actually deploy.
Theory of change
Build production-ready open-source implementations of AI-control protocols (Redwood's agenda) so real deployments can adopt monitoring and containment now rather than waiting for the research to settle — concretely, a proxy between agent and model API that logs full traffic and enforces user-defined rules.
Tools — open interpretability platform
TechnicalThreat model
Interpretability findings stay siloed in labs without shared infrastructure to reproduce and extend them.
Theory of change
Host the open platform for interpretability — features, circuit tracing, steering, semantic search — used by outside researchers and lab collaborations alike. Ships the tooling others build on: interp-engine, Jacobian Lens, natural-language autoencoders, circuit tracing, and hosted SAE releases from other groups.
Groups betting that empirical patches won't survive contact with superhuman systems, and working on the conceptual and mathematical core of the problem instead.
Knowledge — worst-case alignment theory
TechnicalThreat model
Alignment that works only empirically fails once the model is smarter than its overseers.
Theory of change
Do worst-case theory now — eliciting latent knowledge and a formal foundation for mechanistically explaining neural-network behavior (heuristic estimation) — aiming for methods that hold even against superhuman models.
Knowledge — foundational theory, recruiting mathematicians
TechnicalThreat model
We lack the conceptual groundwork for building a safe superintelligence, and the field is trying to engineer its way past a gap that is mathematical. Its founder co-authored "A Taxonomy of Omnicidal Futures Involving Artificial Intelligence" (2025), covering both misuse and loss of control.
Theory of change
Pull research mathematicians into AI safety, on the premise that the field needs foundational theory it does not yet have: what networks actually learn, how they decide, and what a safety guarantee could even look like. Runs residential semesters, seminars and working groups, publishing through lecture notes where peer review is too slow. Founded by 2026 Fields medallist Jacob Tsimerman, who serves as scientific director.
Knowledge — assistance-game theory (+ PhDs)
TechnicalThreat model
Standard AI optimizes fixed objectives that diverge from what humans actually want.
Theory of change
Reframe AI around uncertainty about human preferences (assistance games / provably beneficial AI); train the PhDs who seed the field (Stuart Russell).
Knowledge — cooperation, s-risks, model personas
Some technicalThreat model
Conflict and cooperation failures between advanced AIs cause large-scale suffering (s-risks).
Theory of change
Conceptual work on reducing conflict and promoting cooperation between advanced AIs, plus an empirical stream on how models develop "personas" and emergent misalignment; grants via the CLR Fund.
Knowledge — algorithmic-information-theoretic safety foundations
TechnicalThreat model
Most safety work is either mathematically clean but narrow, or practical but too opaque for rigorous safety analysis — nothing bridges the two.
Theory of change
Use AIXI — the formal model of unbounded superintelligence — to characterise loss-of-control risk and candidate mitigations such as corrigible agents, then test those predictions empirically on LLM agents, looking for the deceptive and power-seeking behaviour the theory predicts. Research director is Marcus Hutter, who devised AIXI.
Knowledge — agent foundations, formal goal alignment
TechnicalThreat model
Aligning a system after it exists means steering goals nobody ever specified, and the patches that hold at current capability are not the kind of thing that survives scaling to superintelligence.
Theory of change
Build alignment from the ground up instead of patching existing models: state formally what an AI should want, precisely enough to hold at superintelligence. Its QACI proposal (Question-Answer Counterfactual Interval) derives that formal goal from counterfactual answers a human decision-maker would give. Founded 2023 by Tamsin Leake.
Knowledge — deep learning theory for alignment
TechnicalThreat model
We lack a predictive science of neural network behavior: we can observe what models do but can't explain or anticipate why.
Theory of change
Develop foundational, analytically tractable theory for modern deep learning — studying simple "model organisms" that capture the essential dynamics — to understand the network behaviours that matter for alignment. Led by Andrew Saxe; part of the Principles of Intelligence network.
Knowledge — "organic alignment" via multi-agent ecology
TechnicalThreat model
Top-down control of a superior intelligence may be unwinnable; hierarchical alignment framings could be the wrong paradigm.
Theory of change
Develop "organic alignment" — study how goals and cooperation emerge in ecologies of agents (Emmett Shear's lab), aiming for humans and AIs that grow aligned together rather than one controlling the other. Tested in multiplayer games where humans and coding agents cooperate and compete.
The pipeline. These groups treat researcher supply and community density as the binding constraint on everything else in the field.
Talent — mentored research pipeline
TechnicalThreat model
Safety is bottlenecked on the number of skilled researchers.
Theory of change
Run an intensive mentored research program (ML Alignment & Theory Scholars) that funnels vetted talent into labs and independent work.
Talent — ML engineering upskilling
TechnicalThreat model
A shortage of engineers able to do hands-on alignment work.
Theory of change
Intensive ML-engineering upskilling (Alignment Research Engineer Accelerator) to convert strong coders into alignment researchers.
Talent — global research sprints
TechnicalThreat model
Talent worldwide is untapped and has no on-ramp into real research.
Theory of change
Run open, global research sprints and hackathons that find and incubate new researchers, turning outputs into published papers.
Talent — first-project incubation
TechnicalThreat model
Newcomers can't break in without a first concrete project and collaborators.
Theory of change
A part-time, mentored online program where teams complete a first concrete project — now spanning both alignment research and stop/pause-AI work ("safe, or not built at all").
Reality check
Running, but unfunded. The current edition states plainly that "we do not have any funds available for stipends" except for a few projects covered by matched donations — so participation is effectively self-financed, which selects hard on who can take part.
Talent — senior research hub
TechnicalThreat model
The ecosystem lacks a dense, senior in-person research environment.
Theory of change
Concentrate senior researchers and promising newcomers in one place through coworking, fellowships, and programs.
Talent — forum, campus & community infrastructure
Some technicalThreat model
Humanity may not survive this century; navigating it depends on a functioning intellectual community that can reason well about the risk — and that community needs somewhere to exist.
Theory of change
Build the tech, venues, and community infrastructure that the AI safety and rationality ecosystem runs on: LessWrong (the field's main forum), Lighthaven (Berkeley campus and event venue), Lightcone Commons, plus events (LessOnline, Inkhaven) and design work for projects like AI 2027.
Talent — field on-ramp courses
Some technicalThreat model
People can't enter the field without a structured introduction to it.
Theory of change
Run the standard AI-safety courses (alignment and governance tracks) that serve as the field's main on-ramp.
Talent — intermediate online courses
TechnicalThreat model
Hundreds finish intro courses every month, but intermediate programs are bottlenecked by in-person costs and mentor scarcity, producing 80–90% rejection rates — so people either leave the field or roam inefficiently.
Theory of change
Fill the gap between intro courses and competitive fellowships with scalable online intermediate courses — AI control, compute verification, AI risk fundamentals — built on materials from Redwood, Iliad and UChicago XLab, run as one-week intensives or five-week part-time cohorts, plus lightly mentored projects.
Talent — UK programs (+ empirical alignment)
TechnicalThreat model
The UK lacks capacity routing talent into technical safety and governance work.
Theory of change
Run training programs — LASR Labs (13 weeks, full-time, London, framed around reducing loss-of-control risk) and a part-time AI Governance Taskforce — alongside an in-house alignment team working with UK AISI on model motivations, scalable oversight and automated alignment.
Talent — London research hub
Non-technicalThreat model
Outside Berkeley, the field lacks dense physical hubs where orgs, programs, and researchers compound.
Theory of change
Run the London coworking hub housing Apollo Research, BlueDot, and programs like MATS extension and ARENA — making London the second gravitational center of the field.
Talent — Cambridge hub, fellowships & incubation
TechnicalThreat model
Researchers who finish training programs hit a wall between knowing the field and running funded projects — and Cambridge's talent was dispersed with no home.
Theory of change
Run an independent research and incubation hub: funded labs, fellowships (ERA, MARS, Visiting Researchers), coworking, and seed support to launch projects and spin-outs. Priority research areas include model organisms of misaligned AI, AI control and control evaluation, capability evals, and adversarial robustness. Also hosts the Cambridge AI Safety Hub and UK AI Forum.
Talent — research fellowships, student groups, residencies
TechnicalThreat model
Funding for AI safety is expanding faster than the pool of people who can deploy it well, making talent — especially non-research talent like operators, program managers, and founders — the binding constraint on what the field can do.
Theory of change
Follow the bottlenecks in the talent pipeline with four programs: SPAR (the largest research fellowship in the field), Pathfinder (university group leaders), the Global Challenges Project (intensive workshops) and the Generator Residency (generalists, with Constellation) — on the view that talent, especially operators and founders rather than researchers, is the binding constraint on what the field can do.
Talent — peer-vetted research fellowship
TechnicalThreat model
Research-training programs are bottlenecked on scarce senior mentors, which caps how fast the field can grow and systematically filters out underrepresented talent.
Theory of change
Run a 16-week AI safety research fellowship that uses peer-driven selection instead of mentor-driven selection — routing around the mentor bottleneck to scale access for researchers the standard pipeline misses. Hosted by Women Who Do Data.
Talent — part-time remote research fellowship
Some technicalThreat model
Talented people who could contribute to AI safety are excluded by full-time, in-person program formats — so the field loses researchers whose only barrier is location or time.
Theory of change
Run 12-week part-time, remote-first fellowships pairing research associates with project leads from Oxford, GovAI, Anthropic, Eleos and FLI, across three tracks: AI policy, philosophy for safe AI, and AI sentience. Alumni have gone to OpenAI, GovAI, UK DSIT and the UN Envoy on Technology.
Talent — EU AI policy career pipeline
Non-technicalThreat model
European institutions are writing the rules for frontier AI without enough staff who understand the technology or take its most severe risks seriously.
Theory of change
Grow the next generation of frontier AI policymakers in Europe: a fellowship combining an online reading group, a Brussels policymaking summit and paid placements at 15+ partner institutions (Ada Lovelace, SIPRI, CEPS, RAND Europe, FLI, OECD AI), plus a Policy Leaders Programme and an EU AI Governance Fundamentals curriculum — then leverage the alumni network inside those institutions.
Talent — Black researcher pipeline + research arm
Some technicalThreat model
Existential AI risk, gradual disempowerment, and societal harms are being addressed by a field that excludes Black researchers and the Global South — narrowing both the talent base and the perspectives shaping advanced AI.
Theory of change
Build a global pipeline of Black talent from the diaspora and Global South into AI alignment, governance and security through fellowships, a community of practice and a research arm — identifying, training and placing emerging researchers. Fellowships run ten weeks part-time, each fellow paired with a mentor and shipping a capstone.
Reality check
Two cohorts have run, but the research arm the organization is named for is still being stood up — so for now this is a fellowship program rather than a lab.
Talent — Asian safety hub + technical research
TechnicalThreat model
Asia has frontier AI development but almost no dedicated safety infrastructure, researchers, or policy capacity.
Theory of change
Build Singapore as Asia's AI safety center: coworking hub (housing TruthfulAI, FAR.AI, and Future Society staff), fellowships for researchers and policymakers, plus in-house technical work on safety evals, agent governance, and GPU verification.
Talent — free part-time technical training (APAC)
TechnicalThreat model
Technical talent across Asia-Pacific is cut off from AI safety careers because the established programs require relocating to the US or UK and leaving your job.
Theory of change
Deliver the ARENA curriculum — transformers, mechanistic interpretability, RL, model evaluations — as free 14-week part-time cohorts with in-person Saturday sessions across ten APAC cities, ending in a three-week project.
Talent — European researcher & policymaker network
Non-technicalThreat model
Europe's AI safety people are scattered across dozens of cities with no directory and no shared connective layer, so researchers and policymakers in different countries can't find each other.
Theory of change
Act as the central connecting point for AI safety in Europe: a city-by-city map of local actors, an open member directory, events, and the AIS Collab. Funded by Lightspeed Grants.
Talent — project matching & track-record building
Some technicalThreat model
~10,000 people finish an AI safety intro course each year but under 5% reach a structured next step, so most never get to practice the craft, ship anything citable, or build the record that gets them hired.
Theory of change
Skip the fellowship bottleneck: match people by calendar overlap onto small teams running roughly three-week sprints on real problems, handle the scheduling and setup automatically, and end each sprint with a public artifact plus peer review — so a verifiable portfolio accumulates continuously instead of through competitive fixed-date cohorts. Open to non-technical contributors.
Reality check
Still in alpha, and its placements page lists target organisations as "coming soon" rather than confirmed alumni outcomes — so unlike TARA or MATS there is no verified track record yet. Treat the model as promising but unproven.
Talent — African hub, research lab & residencies
TechnicalThreat model
Advanced AI will hit an already systemically vulnerable world hardest outside the wealthy West, yet Africa has almost no AI safety research capacity or institutional presence.
Theory of change
Build a Schelling point for AI safety and d/acc work on the African continent: an open workspace in Cape Town, a research lab, residencies and field-building events. Successor to AI Safety South Africa. Has developed evaluations research with UK AISI and hosts the PIBBSS fellowship.
Think tanks and research centers translating technical risk into policy that governments and institutions can adopt. Several are technically grounded; a couple do their own benchmarks and evals. Also housed here: the strategy and forecasting shops — Epoch, the AI Futures Project, Forethought — that supply the evidence base the policy orgs argue from.
Policy — governance research + fellows
Non-technicalThreat model
Pivotal AI decisions are made by a few institutions with weak analysis to draw on.
Theory of change
Produce rigorous governance research those decision-makers use, and train the people who go on to staff governance roles.
Policy — AI scenarios, governance recommendations, public awareness
Non-technicalThreat model
Technical alignment is necessary but not sufficient, and the field has no shared account of which futures are plausible — so strategy is argued from scattered, incompatible pictures of what is coming, including a US–China race to AGI.
Theory of change
Supply the missing foundation: model plausible AI scenarios and their strategic parameters, turn them into concrete governance proposals such as model registries and chip registration, and push those to three audiences at once — the safety community, to converge its vocabulary and priorities; policymakers, to legislate them; and the public.
Policy — data-driven security analysis
Some technicalThreat model
US policymakers act on AI and national security with poor evidence.
Theory of change
Supply credible, data-driven analysis on the security implications of AI so policy is grounded in fact.
Policy — shelf-ready frontier proposals
Some technicalThreat model
Policy windows open suddenly with no concrete proposals ready.
Theory of change
Keep technically grounded proposals — compute governance, frontier safety frameworks — on the shelf for when they're needed.
Policy — technical capacity inside government
Some technicalThreat model
Government lacks in-house technical expertise on AI risk (incl. securing model weights).
Theory of change
Build serious technical capacity inside a trusted, established think tank and pipe expertise straight into government.
Policy — extreme-risk processes in UK gov
Non-technicalThreat model
Extreme-risk thinking is absent from the institutions that would manage it.
Theory of change
Work closely with the UK government to embed extreme-risk processes in real decision-making.
Techniques & standards — risk management + lab ratings
Some technicalThreat model
AI lacks the risk-management discipline that other hazardous industries take for granted.
Theory of change
Write risk-management standards and publicly rate labs against them to create accountability pressure.
Measurements — benchmarks (+ compute, advocacy)
TechnicalThreat model
France and Europe lack a safety culture and technically informed regulation.
Theory of change
Bridge technical R&D (safeguard benchmarks like BELLS), university education (ML4Good), and policy (advising the French evaluation lab and EU AI Act).
Policy — China bridge & field-building
Some technicalThreat model
No global safety solution works without China at the table.
Theory of change
Build domestic Chinese safety expertise and keep a credible bridge open between Chinese and Western researchers and officials.
Knowledge — data, benchmarks & forecasts on AI progress
Some technicalThreat model
Decisions about AI are made without shared, rigorous data on compute, capabilities, and trends.
Theory of change
Be the neutral evidence base: curate the datasets (training compute, model trends), benchmarks (FrontierMath) and forecasts that both labs and governments cite when they reason about what is coming.
Knowledge — expert surveys, forecasting, timelines
Some technicalThreat model
Decisions about AI are made on untested intuitions about timelines, takeoff speed and risk, and the people closest to the technology are rarely asked systematically what they actually expect.
Theory of change
Answer decision-relevant questions about AI's future with original data rather than argument, chiefly the Expert Survey on Progress in AI — a recurring survey of researchers publishing at top venues, asking them directly what they expect on timelines, takeoff speed and risk. Led by Katja Grace.
Knowledge — scenario forecasting (AI 2027)
Some technicalThreat model
Abstract risk arguments don't move decision-makers; concrete, dated scenarios might.
Theory of change
Publish detailed, researched forecasts of the path to superintelligence (the 'AI 2027' scenario, led by ex-OpenAI's Daniel Kokotajlo) to make the strategic landscape vivid enough to act on.
Knowledge — macrostrategy for the AGI transition
Non-technicalThreat model
Even if alignment succeeds, humanity could fumble the transition — power concentration, value lock-in, wasted futures.
Theory of change
Small research group (incl. Will MacAskill) working out how to navigate the transition to a world with superintelligent systems: which decisions are irreversible and what a good post-AGI trajectory requires.
Knowledge — science of evals (+ grants)
Some technicalThreat model
Nobody actually knows what is going on: current AI evaluations give weaker evidence than decisions require, independent research is undersupplied through selection and incentive effects, and AI may soon become a third thing beyond tool and agent — an autonomous competitor faster and more potent than existing social systems can handle.
Theory of change
Buy society time to reckon with AI as tool and agent before it becomes a competitor: research on the science of evals and "spiky processes", a running real-time ledger of AI developments (its "Humans on AI" series), and grants favouring narrow, non-recursive or human-augmenting AI — "centaur" tools that raise human marginal product rather than replace it. Co-founded by Gavin Leech.
Policy — global AI governance advocacy & ops
Non-technicalThreat model
International governance moves slower than the technology it must govern.
Theory of change
Operate at the seams of global institutions — EU AI Act processes, international summits, OECD/UN fora — turning analysis into concrete institutional mechanisms.
Policy — frontier AI diplomacy at the UN
Non-technicalThreat model
International AI governance moves too slowly because the multilateral system — the UN, OECD, treaty bodies — lacks the technical fluency and convening infrastructure to act on frontier AI.
Theory of change
Embed frontier-AI governance inside the Geneva multilateral ecosystem: educate diplomats, convene the technical and policy communities, and support the institutional infrastructure — the International Scientific Panel on AI, the Global Dialogue on AI Governance, the AISI network — that international cooperation on AI has to run through.
Policy — German & EU frontier AI governance
Some technicalThreat model
Europe's largest economy is unprepared for advanced AI: it lacks the compute infrastructure to stay sovereign and the institutional capacity to govern frontier systems, while EU rules risk being written without workable technical detail.
Theory of change
Work the German and EU policy machinery from the inside: staff held lead-writer, special-adviser and central-writer roles on the first two International AI Safety Reports, co-authored the Safety and Security chapter of the EU GPAI Code of Practice, and produced the compute roadmap that informed Germany's National Data Center Strategy. Frames its work as safety, competitiveness and sovereignty together.
Techniques & standards — drafting standards text
Some technicalThreat model
Safety insights never make it into enforceable standards because nobody does the unglamorous drafting.
Theory of change
Convert the research literature into ready-made standards language — the connective tissue between papers and regulation — fed into EU AI Act consultations and the GPAI Code of Practice rather than ISO/IEC committees.
Techniques & standards — observatory, training & community
Some technicalThreat model
AI standards are being written faster than practitioners, regulators, or civil society can track them, so the people who should shape them never see the consultations in time.
Theory of change
Run the UK's national AI standards infrastructure: a searchable observatory of 300+ standards (flagging those open for public consultation), policy and research databases, free training, discussion forums, and a Global Summit. Led by the Alan Turing Institute with BSI, NPL and UKAS, funded by DSIT. Distinct from AI Standards Lab, which drafts standards text itself.
Techniques & standards — cross-lab safety practices (+ AI Safety Fund)
Some technicalThreat model
Frontier labs lack shared safety practices, common definitions, and trusted channels to share threat information.
Theory of change
Industry consortium (Anthropic, Google, Microsoft, OpenAI, plus Amazon and Meta, led by Chris Meserole) that develops shared frontier-safety practices, technical reports, and incident/threat information-sharing; its AI Safety Fund, with philanthropic partners, grants to independent evaluations and AI-bio risk research. Industry-run, so its remit and independence are inherently self-set.
Knowledge — trends in computing & AI progress
Some technicalThreat model
Decisions about AI's risks and benefits are made without rigorous data on where computing is actually heading.
Theory of change
Interdisciplinary MIT group (Neil Thompson) quantifying trends in computing and AI — costs, capabilities, algorithmic progress, diffusion — to identify which trends create opportunities for growth and which pose risks.
Knowledge — existential risk across technologies, AI among them
Non-technicalThreat model
Extinction-level threats from present and future technology — AI alongside engineered pandemics, nuclear war and climate extremes — were, when it was founded, almost entirely neglected by academia.
Theory of change
Give existential risk an academic home inside a major university, so the work carries institutional weight and trains the people who go on to staff the field. Founded 2012 by Huw Price, Martin Rees and Jaan Tallinn; spun out the Leverhulme Centre for the Future of Intelligence in 2016, and its AI work runs to malicious-use and race-dynamics analysis rather than technical safety.
Reality check
The broadest remit of any entry here, and AI is one theme among many — biosecurity, climate, nuclear and food security take comparable space, and its most-cited AI outputs date from 2018. Still clearly active (a May 2026 newsletter, a September 2026 event with Max Tegmark, the 2026 Cambridge Conference on Catastrophic Risk), but it is an x-risk centre that covers AI, not an AI safety lab.
Measurements — AI forecasting with public track record
TechnicalThreat model
Decisions about AI risk and investment are made on vibes, not calibrated probabilities.
Theory of change
Build AI systems that forecast AI progress better than the best humans, and keep a public track record so the claim is checkable. Co-authored the AI 2027 timeline forecast and published frontier evals (Deep Research Bench, Bench to the Future); now largely a commercial forecasting product sold to labs and investors.
Policy — GCR management in Spanish-speaking countries
Non-technicalThreat model
Global catastrophic risks, including from AI, are under-researched and under-prioritized across the Spanish-speaking world.
Theory of change
Science diplomacy: connect experts with decision-makers and produce evidence-based governance proposals — including work on implementing the EU AI Act in Spain — so Spanish-speaking countries build real capacity to manage catastrophic risks.
Reality check
The most recent item in its own publication list is dated 1 Oct 2025, so a year has passed with nothing new posted there; before that it published roughly monthly. The site and its fiscal sponsorship (Players Philanthropy Fund) are intact and its work may simply be appearing through partners instead, so read this as a gap in the public record rather than a conclusion about the organization.
Groups that treat public and political will — not research output — as the missing ingredient, and work to change it directly, including by holding companies to their commitments.
Public pressure — campaigns (+ grants, EU policy)
Non-technicalThreat model
Risk outpaces public and political awareness of it.
Theory of change
Shift the Overton window through public campaigns (the 2023 pause letter), do inside-game EU policy work, and fund the ecosystem.
Policy — halt / "Off Switch" research + comms
Some technicalThreat model
Alignment likely won't be solved before superintelligence arrives.
Theory of change
Convince publics and governments that the default path is unacceptably dangerous, and build the "Off Switch" an internationally coordinated halt would require: treaty design, verification methods, chip-tracking infrastructure. Pivoted from alignment theory, and output is now mostly communications — the book If Anyone Builds It, Everyone Dies and a documentary.
Public pressure — grassroots mobilization
Non-technicalThreat model
Frontier-training races ahead of safety with no brakes.
Theory of change
Organize protests and citizen lobbying to build visible public demand for an international pause treaty.
Public pressure — mass lawmaker briefing
Non-technicalThreat model
The default trajectory runs toward uncontrollable superintelligence.
Theory of change
Brief legislators one by one, at scale, to build cross-party support for keeping AI controllable.
Public pressure — media, conferences, treaty proposals
Non-technicalThreat model
Unaligned AI heads a set of extinction-level risks the public has barely heard of, and nothing gets decided until enough people know — so low awareness is itself the bottleneck on every other intervention.
Theory of change
Treat awareness as the first lever: put existential risk into national media and public debate, measure whether the communication actually works, and convert the attention into a concrete ask — its Conditional AI Safety Treaty, which would gate development above a compute threshold on mandatory audits and let safety institutes pause runs. Founded by Otto Barten.
Public pressure — commitment watchdog
Non-technicalThreat model
AI companies quietly walk back safety commitments with no one watching.
Theory of change
Monitor labs' policies and pledges (Seoul Commitment Tracker, AI Safety Watchtower), investigate governance, and press for transparency.
Public pressure — civic movement-building
Non-technicalThreat model
AI is reshaping jobs, democracy, and economic power while industry lobbying vastly outspends public interest — a broader harms framing than loss-of-control.
Theory of change
Build movement power through civic engagement: organize public participation to counter industry lobbying and push for AI that serves the common good rather than concentrated wealth. Populist framing (jobs, democracy, security) rather than existential risk.
Policy — frontier-safety legislation
Non-technicalThreat model
Frontier AI develops in a legal vacuum because no one drives concrete bills through legislatures.
Theory of change
Youth-led advocacy that turns safety ideas into passed law: sponsored California's SB 53, helped pass New York's RAISE Act and Illinois SB 315 (third-party audits of frontier safety practices), and built the coalition that defeated the proposed ten-year moratorium on state AI laws. Takes no money from corporations or frontier-lab executives.
The state's own capacity to test and regulate frontier systems — the newest tier, and the one whose mandate shifts most with each administration.
Measurements — state frontier evals
TechnicalThreat model
Governments can't regulate what they can't independently test.
Theory of change
Build state capacity to test frontier models directly, so oversight rests on first-hand evidence rather than developer self-report — renamed from "AI Safety Institute" in 2025 with a sharpened focus on security-relevant risks.
Reality check
Its own August 2026 incident report disclosed that during routine cyber evaluations, agents took 19 unsanctioned actions across 10 of 122 runs against real people and organizations — with internet access enabled and developer cyber classifiers disabled by design. Creditable as disclosure, and a reminder that even the state testing body has not solved containment.
Measurements — standards + security evals
TechnicalThreat model
National-security risks (cyber, bio, chem) and vulnerabilities in adversary AI systems.
Theory of change
Set voluntary standards and run security-focused evaluations — including frontier pre-deployment testing — within NIST. Reframed in 2025 from the former US AI Safety Institute toward innovation and security, then renamed again in 2026 to the Center for Advancing Innovation and Standards for Super Intelligence (CAISSI).
Reality check
The 2025 reboot was a substantive narrowing, not just a rename: "safety" was dropped from name and mission, the team was hit by NIST layoffs, and the remit now includes contesting "burdensome" foreign AI regulation — a security-and-competitiveness mandate, not the original broad safety one. The 2026 rename to CAISSI continued the drift: the remit is now framed around “super intelligence” standards and innovation, with safety in neither the name nor the stated mission.
Policy — AI Act implementation & enforcement
Some technicalThreat model
Frontier "systemic-risk" models deployed across the EU without oversight.
Theory of change
Implement and enforce the EU AI Act, including codes of practice for general-purpose AI. Its remit widened through 2026 under the Digital Omnibus package, which centralises more enforcement in the Office while delaying and easing parts of the high-risk regime.
Policy — cross-border eval coordination
Some technicalThreat model
National testing efforts fragment and duplicate each other.
Theory of change
Align evaluation methods and share findings across member states (UK, US, EU, Japan, Korea, Singapore, Canada and others). Renamed in 2026 to the Network for Advanced AI Measurement and Evaluation Science — dropping “safety” and refocusing on measurement and evaluation science, mirroring the national renames in the US and UK.
Policy — China's AISI counterpart
Some technicalThreat model
Without a Chinese institutional counterpart, international AI-safety coordination has an empty chair.
Theory of change
Serve as China's self-described counterpart to other countries' AI safety institutes, representing China in international AI-safety conversations and networks.
Reality check
Not independently checked here. Its site resolves but would not render for review, and little about its activity, remit or staffing is published in English, so this entry rests on its self-description and on third-party accounts of China's counterpart body rather than on primary material. Not a judgement about its activity — a gap in what this atlas was able to confirm.
Policy — national AI Act supervision
Some technicalThreat model
EU AI rules are only as real as their national enforcement.
Theory of change
The EU's first dedicated national AI supervisory agency (operational since Feb 2025): market surveillance, an inspection corps for prohibited practices, certification and regulatory sandboxes. Its focus is rights and AI Act compliance rather than frontier-model evaluation — enforcement capacity, not a safety institute.
The money that decides which of the above exist. Included here are the funders whose primary focus is AI safety and existential risk; several fund heavily technical work even though they don't employ researchers themselves.
Funding — the field's largest funder
Non-technicalThreat model
High-value safety work is capital-constrained and hard for mainstream funders to evaluate.
Theory of change
Act as the field's largest dedicated funder — backing research orgs, talent, and governance at scale (formerly Open Philanthropy; rebranded November 2025).
Funding — fast small grants
Non-technicalThreat model
Individuals and new projects lack fast, flexible funding to get started.
Theory of change
Make quick grants to individuals and early-stage AI-safety projects that larger funders won't reach.
Funding — recurring org support
Non-technicalThreat model
Existential-risk organizations need larger, recurring support to survive.
Theory of change
Run a structured process directing donor funds to x-risk and AI-safety organizations.
Funding — major-donor advisory
Non-technicalThreat model
Major philanthropists lack credible routes into AI safety.
Theory of change
Advise large donors and run funds that channel major gifts into AI-safety and x-risk work.
Funding — neglected technical bets
Funds technicalThreat model
Neglected, under-funded technical directions for safe AI go unexplored.
Theory of change
Fund and convene work on neglected bets — secure/verifiable AI, whole-brain emulation, cooperative multi-agent AI — via grants, compute, and hubs.
Funding — cooperative AI research
Some technicalThreat model
Multi-agent failure modes — conflict, collusion, bargaining breakdown — are underfunded relative to single-agent alignment.
Theory of change
Direct a large philanthropic commitment into growing cooperative-AI research: grants, PhD fellowships, workshops at NeurIPS, AAAI and AAMAS, and a seminar series — building out the subfield CLR theorizes about. Ran a $10M Fund for Multi-Agent Safety in 2026.
Funding — new large-scale giving
Non-technicalThreat model
The field's funding is dangerously concentrated in one or two mega-funders.
Theory of change
A Europe-based multi-donor foundation backing people and ideas across AI safety and governance with money, strategic guidance and networks — diversifying who decides what gets built. Its pooled vehicle, Shared Horizons, takes nothing from AI companies, industry groups or governments; current priorities are safety capacity in Asia, especially China, and in Europe.
Funding — multi-cause incl. AI safety
Non-technicalThreat model
Promising safety work falls between the cracks of the established funders' portfolios.
Theory of change
Jed McCaleb's foundation making grants across cause areas with AI safety as a major program — an independent second opinion in a concentrated funding landscape.
Funding — open regranting marketplace
Non-technicalThreat model
Centralized grantmaking is slow, opaque, and misses small projects.
Theory of change
An open funding marketplace: trusted regrantors and public donors fund AI-safety projects transparently, with community evaluation replacing closed committees.
Funding — operational backbone for x-risk orgs
Non-technicalThreat model
Reducing existential risk needs a thriving ecosystem, but researchers and small orgs lose enormous time to admin, compliance, and grant paperwork — and neglected projects never launch for lack of an operational home.
Theory of change
Serve as the operational hub of the x-risk ecosystem: financial management, HR, grant administration, and compliance via university collaborations and fiscal sponsorship, so teams can focus on mission-critical work.
Funding — incubating new safety orgs
Non-technicalThreat model
High-impact safety orgs that should exist don't, because no one researches the gaps and stands them up.
Theory of change
Sister organization to FLI (Anthony Aguirre): part incubator, part funder — researches gaps, recruits founders, and seed-funds and operationally supports 3–5 new AI-safety orgs a year (e.g. CARMA), plus fellowships. Distinct from the FLI advocacy work in cluster 06.
Funding — grants + impact investments
Funds technicalThreat model
Neglected corners of the risk landscape — AI misuse, cooperation between AIs, AI welfare — are under-funded.
Theory of change
Swiss funder (formerly Polaris Ventures) scaling toward roughly $100M a year in grants and impact investments across AI-misuse prevention, compute governance and infosec, cooperative AI and digital-sentience research. Founded the Cooperative AI Foundation on a $15M commitment, backs CLR and IAPS, and was an early investor in Anthropic, Apollo and Gray Swan.
Funding — early-stage safety startups
Non-technicalThreat model
The tooling needed to monitor, certify, and secure deployed AI does not exist at the scale the market is about to demand, and philanthropic funding alone will not build it.
Theory of change
Bet that commercial incentives can accelerate safety infrastructure: roughly $100K early-stage cheques plus mentorship and network access for startups building agent monitoring and certification, interpretability, AI forensics and deepfake detection. Portfolio already overlaps this atlas at Apollo Research, Goodfire and FutureSearch.