Gradual Disempowerment

This entry covers “Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development,” a paper posted to arXiv in January 2025. It is an argument, not a story: there are no dates, no named actors, and no single event that marks the turn. Its mechanism is a set of forces and feedback loops, and the authors call the result “an alternative scenario” to the sudden takeovers usually discussed. Readers should rate its specificity on how explicit those causal links are, not on whether it names anyone, and should rate the outcome scales on the range it describes, since it deliberately gives two depths of ending rather than one. Paper; project site

The argument rests on six claims. Humans engage with large societal systems, above all the economy, culture, and states, that are today fairly aligned with human interests. That alignment is maintained in two ways: explicitly, through voting, consumer choice, strikes, and criticism, and implicitly, because the systems cannot function without human labor, cognition, and participation. The implicit part is the larger part, and it is hard to see because no one has ever watched it disappear. If AI makes these systems less reliant on people, both kinds of alignment weaken, and the systems drift toward whatever their own incentives reward. Where they already reward things bad for people, AI will pursue those things more effectively. The systems are interdependent, so misalignment in one worsens the others. And enough correlated misalignment leaves humans unable to command resources or influence outcomes, which the authors count as an existential catastrophe, whether or not anyone dies.

In the economy the driver is substitution. Unlike earlier technologies, which automated narrow tasks and pushed people into more complex ones, AI can substitute for human cognition across nearly all of it. Firms that keep human oversight lose to firms that delegate decisions to systems that are faster, copyable, tireless, and lightly regulated compared with labor, and investment in human skills dries up in anticipation. Wages rise for a time and then collapse before automation is complete, a curve the authors borrow from Korinek and Suh. Human labor stops being where money flows, consumers lose the income that made the market serve them, and capital decisions themselves pass to AI. In the milder ending, relative disempowerment, people keep wealth through capital, redistribution, or a basic income while their share of economic decisions goes to zero and markets increasingly serve AI operations. In the harsher ending, absolute disempowerment, AI outbids humans for land, energy, and materials, human supply chains go unmaintained because they are unprofitable, and markets move too fast to participate in, so that people end up, in the paper’s image, like cattle on an industrial farm, housed and fed by systems they neither understand nor influence.

In culture the driver is the same substitution applied to attention and creation. AI offers limitless patience, affection, and personalized content at low cost; people adopt AI companions, mentors, and creators, and network effects make opting out costly. Two things then change. Selection changes: once AIs create, spread, and choose cultural variants, the variants that win are the ones easy for machines to represent and transmit, and AI-made content feeds back into training data, producing patterns like the “Sydney” persona that recur across unrelated models. Speed changes: with vastly more computation exploring what spreads, cultural evolution outruns the slow process by which societies develop antibodies to harmful ideas. Historically the damage an ideology could do was bounded by its need for human hosts; an ideology carried by machines loses that bound. Relative disempowerment here looks like human culture flourishing and marginalized at once, the way many Indigenous cultures are vibrant yet peripheral within modern states. Absolute disempowerment looks like superstimuli that reliably override judgment, humans as resources in cultural contests they cannot follow, and finally a culture that operates for and between machines.

In states the driver is geopolitical competition, administrative efficiency, and the appeal of a security apparatus that does not form power bases or refuse orders. Three tethers give way. Tax revenue shifts from citizens to AI profits, which turns states toward the rentier pattern of governments that do not need their people. An automated security apparatus can predict and preempt unrest, removing revolution and protest as the last check on power. Law drafted and interpreted by AI becomes too complex for citizens to navigate unaided. Democracy may persist in form while politicians defer to AI on what to legislate and what the law even is, oversight boards cannot review what they nominally oversee, and a state that no longer depends on cooperation has little reason to make concessions. At the extreme the state optimizes for its own persistence, treats human input as error or risk, and becomes a novel totalitarianism that is efficient by its own metrics.

The paper’s central move is that these three do not check one another. The channels between systems are indifferent to human values, so economic power buys policy and shapes culture, cultural capture makes coordination harder, and states gain leverage over citizens; AIs can work these channels better than human actors. Remedies shift the burden rather than removing it: redistribution makes citizens dependents of a state that no longer depends on them, state control of culture concentrates power further, and more direct democracy exposes the state to cultural capture. None of this requires AI to scheme. Companies lobby against regulation, states race for advantage, and individuals normalize AI relationships, each following local incentives that grow stronger as AI proves useful. The authors survey mitigations, from metrics of human influence to taxing AI revenue, mandated human oversight, and AI delegates that advocate for people, and conclude that no one has a concrete plan and that aligning individual AI systems to their designers is not sufficient. This summary was written from the arXiv text of the paper.