AI-generated draft · This entry was drafted by AI from the original work and public summaries, and will be replaced by a human-edited version. The original has not been fully checked for this revision; interpretations and missing causal links remain open to review.
01 / The scenario
What happens?
This entry covers “AI as Normal Technology,” the essay Arvind Narayanan and Sayash Kapoor published with the Knight First Amendment Institute in April 2025, and not their book “AI Snake Oil” or their newsletter except where those explain it. It is an argument, not a story: the authors state a worldview and describe the “median outcome,” with no dates, named actors, or turning event, so this entry records its forces as steps. Essay; announcement; follow-up guide
Part I separates invention (new methods), innovation (applications), adoption, and diffusion, and argues that impact arrives only at the end of that chain. In high-consequence domains such as medicine, criminal risk, and insurance, deployed systems still use decades-old regression, because complex models fail in ways validation misses (Epic’s sepsis tool) and because regulators hold them to existing standards; the authors expect this to continue. Elsewhere adoption is slower than headlines suggest (40 percent of U.S. adults using generative AI came to under 4 percent of work hours), and diffusion waits on people, organizations, and institutions changing, as electrification waited some forty years for factories to be rebuilt around it. Innovation is capped by the external world too: the “capability-reliability gap” meant self-driving cars needed two decades for the feedback loop AlphaZero runs in hours, tacit knowledge must be learned sector by sector, and society will not permit rapid experiments on people. Economic impact is therefore gradual and uneven; as tasks are automated, humans move to what is not yet automated and the AGI goalpost keeps moving. Even recursive self-improvement, granted as possible, is expected to arrive gradually, and methods speed is not what sets impact.
Part II replaces intelligence with power, “the ability to modify one’s environment.” Since capabilities will keep rising, the authors intervene between capability and power: keep capable systems from acquiring power rather than try to align arbitrarily powerful ones. They predict AI will not meaningfully beat trained humans at forecasting or at persuading people to act against their own interests. Control has many forms between full autonomy and a human in every loop: auditing, monitoring, fail-safes, least privilege, and simpler AI overseeing more capable AI. As more tasks become automatable, a growing share of human work becomes AI control and task specification, as factory work became machine control, and market forces drive this because “poorly controlled AI will be too error prone to make business sense.”
Part III takes accidents, arms races, misuse, misalignment, and systemic risks in turn. Accidents fall to deployers, disciplined by markets and regulation. Races to the bottom are an old, sector-specific problem solved by making firms internalize safety costs; self-driving shows safety and market success moving together, and the authors see no straightforward reason for arms races between countries over adoption (military AI is excluded). Model alignment is a brittle defense against misuse because harm depends on context the model lacks, so defenses belong downstream, where AI helps defenders more than attackers. Catastrophic misalignment is “by far” the most speculative risk: adoption requires demonstrated reliability in low-stakes settings first, and deception is an engineering problem. What they take most seriously are systemic risks (bias, job losses, inequality, concentration of power, a polluted information ecosystem, democratic backsliding) that arise from people using AI for their own ends: serious, but unlikely to be existential.
Part IV treats policy under disagreement that will not resolve. Defenses against superintelligence concentrate power and defenses against normal-technology risks disperse it, so compromise fails, and expected-value arguments on subjective probabilities are rejected. The authors ask for reducing uncertainty through research and monitoring, and for resilience: democratic foundations, state capacity, defenses, and, more cautiously, competition, open models, and polycentric regulation. Nonproliferation (export controls, licensing, bans on open weights) is called infeasible, because the knowledge is widespread and costs fall, and harmful, because it creates single points of failure; “nonproliferation is a mindset,” and its measures build the very autonomy and resource access that make AI powerful. Benefits are not automatic: diffusion needs enabling regulation, investment in complements such as literacy, and safety nets.
The essay explicitly rejects “fast takeoff,” superintelligence as an agent or a species, and the compressed timelines of Situational Awareness and AI 2027, while forecasting large change over decades and declining to look past its own world. It leaves open how much unemployment the transition brings, noting occupations already lost while expecting adaptation, and whether power concentrates despite resilience policy, which the authors call a bigger problem than accidents. Safety-community critics reply that gradual does not mean continuous and that the forecasting and persuasion predictions are already strained. MIT Technology Review; AI is Not Normal Technology
For the library, the mechanism is a gate: capability advances faster than the world can absorb it, adoption and diffusion set the pace, humans keep control because uncontrolled AI does not pay, and the remaining risks are old ones amplified. The forces are explicit and tied to evidence, so specificity can be rated high on the causal claims even though nothing is dated or personified; rate plausibility on whether the diffusion argument holds against faster capability growth and whether the persuasion and control predictions survive. It is the counter-scenario to the AI 2027 and Situational Awareness entries and the policy mirror of Machines of Loving Grace, which shares its bottleneck framework but keeps chip controls and a compressed timeline.
Library summary · Not the full original work
02 / The mechanism
How does it happen, step by step?
- 01
AI methods keep advancing, and the authors grant that capabilities may rise indefinitely.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 02
Impact requires applications to be invented, adopted, and diffused, and each stage has its own speed limit, so faster methods do not mean faster change.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 03
Regulators hold medicine, criminal risk, and insurance to existing standards, so deployed systems there still use decades-old regression.
- Locus of agency
- Human-directed
- Intentionality
- Intended
- Human actor type
- State actor
- 04
Complex models fail in ways validation misses, as Epic’s sepsis tool did, which keeps high-consequence adopters cautious.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 05
Adoption measured in hours of use stays low long after access is universal: 40 percent of U.S. adults using generative AI came to under 4 percent of work hours.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 06
Diffusion waits on people, organizations, and laws changing, as electrification waited some forty years for factories to be rebuilt around it.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 07
Reliable applications need a deployment-and-improvement loop that safety keeps slow; self-driving cars needed two decades for what AlphaZero did in hours.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 08
Tacit knowledge must be learned sector by sector, and society will not permit rapid experiments on people, so benchmark gains overstate real progress.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 09
As tasks are automated, humans move to what is not yet automated and the AGI goalpost keeps moving; economic impact is gradual and uneven.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 10
Even recursive self-improvement, granted as possible, arrives gradually, and the speed of methods is not what sets impact.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 11
The relevant question becomes power, the ability to modify one’s environment, rather than intelligence; capability need not become power.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 12
AI does not meaningfully beat trained humans at forecasting or at persuading people to act against their interests, which keeps control tractable.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 13
Deployers keep capable systems from acquiring power through auditing, monitoring, fail-safes, least privilege, and simpler AI overseeing more capable AI.
- Locus of agency
- Hybrid / distributed
- Intentionality
- Intended
- Human actor type
- Corporation or lab
- AI autonomy level
- Agentic under oversight
- 14
Market forces drive this, because poorly controlled AI is too error prone to make business sense.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 15
A growing share of human work becomes AI control and task specification, as factory work became machine control.
- Locus of agency
- Hybrid / distributed
- Intentionality
- Accepted trade-off
- Human actor type
- Diffuse / uncoordinated
- AI autonomy level
- Agentic under oversight
- 16
Accidents fall to deployers, who are disciplined by markets and regulation.
- Locus of agency
- Human-directed
- Intentionality
- Intended
- Human actor type
- Corporation or lab
- 17
Races to the bottom are sector-specific and are solved by making firms internalize safety costs; self-driving shows safety and market success moving together.
- Locus of agency
- Human-directed
- Intentionality
- Intended
- Human actor type
- State actor
- 18
The authors see no straightforward reason for arms races between countries over adoption; military AI is set outside the analysis.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 19
Model alignment is a brittle defense against misuse because harm depends on context the model lacks, so defenses go downstream, where AI helps defenders more than attackers.
- Locus of agency
- Human-directed
- Intentionality
- Intended
- Human actor type
- Diffuse / uncoordinated
- 20
Catastrophic misalignment stays speculative: systems get consequential access only after demonstrated reliability in low-stakes settings, and deception is an engineering problem.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 21
People using AI for their own ends amplify bias, job losses, inequality, concentration of power, a polluted information ecosystem, and democratic backsliding.
- Locus of agency
- Human-directed
- Intentionality
- Accepted trade-off
- Human actor type
- Diffuse / uncoordinated
- 22
These systemic risks are the ones the authors take most seriously; they call them serious but unlikely to be existential.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 23
Disagreement between the superintelligence and normal-technology worldviews does not resolve, and since one’s defenses concentrate power while the other’s disperse it, compromise fails.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 24
Nonproliferation is judged infeasible, because the knowledge is widespread and costs fall, and harmful, because it creates single points of failure and builds the autonomy that makes AI powerful.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
- 25
Policy pursues reduced uncertainty through research and monitoring, and resilience through democratic foundations, state capacity, defenses, competition, open models, and polycentric regulation.
- Locus of agency
- Human-directed
- Intentionality
- Intended
- Human actor type
- State actor
- 26
Benefits are not automatic; diffusion needs enabling regulation, investment in complements such as literacy, and safety nets.
- Locus of agency
- Human-directed
- Intentionality
- Intended
- Human actor type
- State actor
- 27
How much unemployment the transition brings, and whether power concentrates despite resilience policy, are left open.
- Locus of agency
- Structural / no discrete agent
- Intentionality
- Emergent
03 / The premises
What must be true?
These describe what the work relies on, not the library’s beliefs.
Physical and organizational bottlenecks will substantially limit how quickly AI transforms society.
The central claim of Part I: transformative impact is gated by innovation, adoption, and diffusion, which are limited by safety regulation, organizational change, tacit knowledge, and the social cost of experimentation, so it unfolds over decades as it did for electricity and the internet.
A software-only intelligence explosion is possible.
The authors reject “fast takeoff” scenarios and expect a gradual increase in the role of automation in AI development rather than a singular moment of recursive self-improvement; they add that even arbitrarily fast methods progress would not speed impact.
Current language-model approaches can scale to superintelligence.
The essay accepts that AI capabilities are likely to increase indefinitely but calls superintelligence incoherent as usually conceptualized and denies there is a useful sense in which AI is more intelligent than people acting with AI; it does not rule on how far current methods scale.
Developing superhuman AI capabilities does not require open-ended autonomous goals.
AI is “a tool that we can and should remain in control of”; capability keeps rising while deployers withhold power and autonomy, and the authors warn that nonproliferation-minded designs are what would add open-ended autonomy and resource access.
Human oversight will remain effective for AI systems substantially more capable than their supervisors.
Part II argues that AI will not be meaningfully more capable than humans acting with AI assistance, that control has many flavors beyond a human in every loop, and that human control will remain central to deployment.
Restrictions on computing resources will effectively constrain frontier AI development.
Export controls on hardware are listed as nonproliferation, which the authors call infeasible to enforce because the required knowledge is widespread and training costs keep falling, and harmful because it concentrates the market and creates single points of failure.
Instrumental convergence can occur across different ultimate goals.
The paperclip maximizer argument is said to rest on assumptions that may or may not be true, and catastrophic misalignment is classed as a speculative risk whose underlying uncertainty research can reduce.
Superhuman persuasion will significantly affect society.
The authors predict that AI will not meaningfully outperform trained humans at persuading people to act against their own self-interest, and note that the control problem is far more tractable if superhuman persuasion is an unfounded concern.
Machines can perform enough cognition for people to become dependent on them.
The world of Part II has most cognitive and physical tasks automated and human work redefined as AI control and specification, so the economy depends on machine cognition even though humans hold the levers.
Unlisted shared premises are not applicable. Absence does not imply disagreement.
04 / Reader assessment
How convincing is the account?
These are participating readers’ judgments, equally weighted. Ratings can refer to earlier revisions. Well-being and open-future scores describe the future the work arrives at, not the path there; high specificity does not establish plausibility.
Rate this scenario
Sign in to rate this scenario. Anyone can browse the library.
05 / A living entry
Suggestions & revision history
No suggestions yet. Help make this account more precise.