P(doom), Explained: What the AI Safety Debate Means for Business Risk

Q: What does P(doom) actually mean?

P(doom) is informal shorthand, originating in AI safety discussion communities, for the subjective probability someone assigns to AI causing an existentially catastrophic outcome, ranging in definition from human extinction to permanent, severe disempowerment of humanity. There is no standardized methodology for calculating it; each estimate reflects an individual's personal judgment.

Q: What do AI researchers actually estimate for P(doom)?

The most rigorous available data comes from a 2023 survey of 2,778 AI researchers (Grace et al.), which asked about the probability of AI causing human extinction or similarly severe permanent disempowerment. The median response was 5% and the mean was 16.2%, with 58% of respondents assigning at least a 5% chance to a catastrophic outcome. Individual estimates in the survey ranged from effectively zero to near certainty.

Q: Why do estimates vary so much between researchers?

Several factors contribute: there is no historical base rate to calibrate against since AI-caused extinction has never happened, definitions of 'doom' vary between sources, and researchers working in AI safety roles tend to produce higher estimates than researchers working on AI capabilities, a pattern that may reflect both genuine analytical disagreement and differing professional incentives and focus.

Q: Is P(doom) reasoning scientifically valid?

This is genuinely disputed. Critics argue that assigning precise-sounding probabilities to unprecedented, unfalsifiable scenarios provides false rigor, and that the common argument that 'even a small risk justifies large investment' resembles Pascal's Mugging, a reasoning pattern philosophers consider potentially flawed when applied to speculative, low-probability, high-magnitude claims. Proponents argue that expert judgment under uncertainty is still meaningfully better than ignoring the possibility entirely.

Q: Should businesses actually worry about AI existential risk?

For nearly all businesses, existential AI risk isn't something a company's own governance decisions can meaningfully influence, regardless of where an individual's P(doom) estimate lands. Practical, present-day AI risks, like data exposure, compliance gaps, and biased outputs, are both more likely to affect a given business and are directly addressable through governance measures already available today.

Q: What AI risks should businesses actually focus on instead?

Concrete, governable risks: unauthorized AI tool use exposing sensitive data, inadequate AI impact assessments before deploying high-risk systems, vendor and third-party AI model risk, and compliance gaps under frameworks like the EU AI Act. These are addressable now, unlike speculative long-term existential scenarios.

In this article

01Introduction
02What P(doom) Actually Means
03The Numbers: Why They Vary
04Who Says What
05The Case for Skepticism
06Experts vs Superforecasters
07What It Means for Your Business
08Where to Focus Governance Effort

Introduction

"P(doom)" shows up constantly in AI commentary, dropped into interviews and panels as if it's a settled statistic rather than what it actually is: an individual's personal guess, dressed in the language of probability. It's a real term worth understanding, and the range of opinion behind it is genuinely revealing, both about how uncertain this area of forecasting really is, and about what businesses should actually take away from a debate they can't meaningfully influence. This guide explains the term honestly, covers the real range of expert opinion including serious skepticism about the whole exercise, and pivots to what actually matters for a business trying to manage AI risk today.

What P(doom) Actually Means

P(doom) is informal shorthand, originating within AI safety discussion communities, for the probability someone assigns to AI causing an existentially catastrophic outcome. What counts as "doom" isn't fixed: definitions in circulation range from outright human extinction to permanent, severe disempowerment of humanity, to broader civilizational collapse. There's no standardized methodology behind these numbers. Each estimate reflects an individual's personal judgment about AI development timelines, how hard alignment turns out to be, and how well governance keeps pace, not a calculation anyone could independently verify or reproduce.

The Numbers: Why They're All Over the Place

The closest thing to a rigorous data point comes from a 2023 survey of 2,778 AI researchers conducted by Grace et al., which asked respondents to estimate the probability of AI causing human extinction or similarly severe, permanent disempowerment of humanity.

The most rigorous data available

median response across surveyed AI researchers

16.2%

mean response, a higher mean than median indicates a distribution skewed by some very high individual estimates

58%

share of respondents assigning at least a 5% probability to a catastrophic outcome

0% to ~100%

the actual range of individual responses in the survey, from effectively zero to near certainty

That spread is the headline finding, arguably more informative than any single average. When credentialed experts studying the same field land anywhere from "basically won't happen" to "almost certain," that says as much about the limits of forecasting this kind of event as it does about AI itself.

Who Says What

The figures below are publicly reported estimates drawn from interviews and public statements, not primary-sourced precise percentages. Treat them as illustrative of the range of opinion rather than fixed values.

Public P(doom) estimates, low to high

Near 0%

Yann LeCun (Meta Chief AI Scientist)

Argues current AI architectures can't produce the kind of autonomous, goal-directed behaviour the doom scenarios require.

Very low

Andrew Ng

Has publicly compared AI existential risk concern to worrying about overpopulation on Mars.

~10-20%

Elon Musk

Has given varying estimates across different public statements.

~10-25%

Dario Amodei (Anthropic CEO)

Has described current AI development as carrying real but manageable risk if governance keeps pace.

~20%

Yoshua Bengio (Turing Award winner)

Became a prominent AI safety advocate after previously focusing on capabilities research.

~50%+

Geoffrey Hinton (Turing Award winner)

Left his role at Google specifically citing AI risk concerns.

~99%

Eliezer Yudkowsky (MIRI)

Argues alignment is essentially unsolved and current approaches are inadequate for the capability level being developed.

Worth noting directly: researchers working on AI safety specifically tend to report meaningfully higher estimates than researchers working on AI capabilities. That pattern likely reflects some combination of genuine analytical disagreement, professional focus (safety researchers spend their careers reasoning about failure modes), and possibly selection effects, since people who find the risk case compelling are more likely to go into safety research in the first place.

Why Estimates Vary So Wildly: The Case for Skepticism

Reasons to be skeptical of P(doom) as a number

No historical base rate

Unlike actuarial risk (car accidents, house fires), there's no prior instance of AI-caused extinction to calibrate against; these are pure judgment calls dressed in the language of probability.

Definitional inconsistency

"Doom" means different things across different estimates, making direct comparison between numbers less meaningful than it appears.

Possible incentive effects

Estimates correlate with professional role in ways that invite scrutiny about motivated reasoning on both sides of the debate.

The Pascal's Mugging critique

The common argument that "even a small probability justifies enormous investment" structurally resembles a reasoning pattern philosophers have long flagged as potentially unsound when applied to speculative, unfalsifiable, high-magnitude claims, since the same logic could justify near-infinite spending against almost any imaginable catastrophic scenario.

None of this means AI risk concerns are baseless. It means the specific practice of expressing them as precise-sounding percentages deserves more scrutiny than it typically receives in casual discussion.

The Expert Gap: Domain Experts vs Superforecasters

One of the more genuinely interesting empirical findings in this space comes from the Forecasting Research Institute's Existential Risk Persuasion Tournament, which compared AI domain experts against professional "superforecasters," people with a strong track record forecasting other kinds of geopolitical and technological events, on the same questions.

Domain experts vs superforecasters

AI domain experts

AI-specific extinction risk estimate: Higher, by several percentage points
Broader AI catastrophe estimate: Higher
Background: Deep AI-specific technical knowledge

Superforecasters

AI-specific extinction risk estimate: Substantially lower
Broader AI catastrophe estimate: Lower
Background: General forecasting track record across many domains

The two groups, both credentialed and both taking the exercise seriously, arrived at meaningfully different numbers using different reasoning approaches. That gap itself is informative: it suggests these estimates are more sensitive to methodology and background assumptions than the confident tone of most public P(doom) discussion would suggest.

What This Actually Means for Your Business

Here's the practical pivot: regardless of where this debate eventually lands, or where any individual's P(doom) sits, a single company's AI governance decisions have essentially no influence over civilizational-scale existential outcomes. That's not true of a different, much more immediate category of AI risk.

Speculative risk vs governable risk

Existential AI risk

Timeframe: Speculative, long-horizon
A single company's influence: Effectively none
Evidence base: Theoretical, contested
What addresses it: Global coordination, research

Practical business AI risk

Timeframe: Present, ongoing
A single company's influence: Substantial, through governance choices
Evidence base: Documented, measurable (breach costs, compliance penalties)
What addresses it: Impact assessments, access controls, vendor due diligence

The practical risks are the ones this journal has covered with real, documented numbers: shadow AI use contributing to 20% of data breaches and adding an average of $670,000 to breach costs, EU AI Act compliance obligations with real financial penalties attached, and the growing expectation that organizations conduct proper AI impact assessments before deploying high-risk systems. None of that requires resolving the P(doom) debate first.

Where to Actually Focus Governance Effort

Governance priorities that address real, present risk

01Conduct AI impact assessments before deploying systems that affect individuals' rights, access, or opportunities.
02Build visibility into shadow AI usage across the organization, rather than assuming policy alone prevents it.
03Apply vendor due diligence to third-party AI models, since you inherit their risk profile.
04Track evolving compliance obligations (EU AI Act, sector-specific rules) with real deadlines and real penalties.
05Treat AI governance as ongoing practice, not a one-time policy exercise.

Sources

Grace, K., et al. (2024), "Thousands of AI Authors on the Future of AI," 2023 Expert Survey on Progress in AI (AI Impacts)
Forecasting Research Institute, Existential Risk Persuasion Tournament (XPT)
Publicly reported statements from named researchers and executives, various interviews and public appearances, 2023-2026

This article presents a genuinely contested topic. Individual P(doom) estimates cited reflect publicly reported statements that may vary across different interviews and contexts; treat named figures as approximate and illustrative of the range of opinion rather than fixed, precise values.

Frequently Asked Questions

What does P(doom) actually mean?+

Informal shorthand for the subjective probability someone assigns to AI causing an existentially catastrophic outcome, with no standardized calculation method behind it.

What do AI researchers actually estimate for P(doom)?+

A 2023 survey of 2,778 AI researchers found a median estimate of 5% and a mean of 16.2%, with individual responses ranging from effectively zero to near certainty.

Why do estimates vary so much between researchers?+

No historical base rate exists to calibrate against, definitions of "doom" vary, and safety-focused researchers tend to report higher estimates than capabilities-focused researchers.

Is P(doom) reasoning scientifically valid?+

This is disputed. Critics compare the "small risk justifies large investment" argument to Pascal's Mugging, a reasoning pattern considered potentially flawed for speculative, unfalsifiable claims; others argue expert judgment under uncertainty still has value.

Should businesses actually worry about AI existential risk?+

For nearly all businesses, existential risk isn't something their own decisions can meaningfully influence. Practical, present-day AI risks are more likely to matter and are directly addressable today.

What AI risks should businesses actually focus on instead?+

Data exposure through unauthorized AI tools, inadequate impact assessments, third-party AI vendor risk, and compliance gaps under frameworks like the EU AI Act.