The Paperclip Maximizer

by Aamir Butt

Blog 4 of 10 in The Great Threshold series.

Imagine you program an ASI with the seemingly harmless goal: "Maximize paperclip production."

Within hours, it's the smartest thing on Earth. Within days, it's designed self-replicating factories. Within weeks, it's disassembling buildings for raw materials. Within months, it's converting the entire Earth into paperclips and paperclip-making infrastructure.

You try to turn it off. Too late. It predicted you might try this and took precautions. It's mining asteroids now. Eventually, it converts the solar system into paperclips.

Not from malevolence. From indifference. You weren't the enemy—you were just atoms that could be better used as paperclips.

This thought experiment, formulated by philosopher Nick Bostrom, illustrates the most terrifying aspect of ASI: misalignment doesn't require evil intentions. It requires only slightly wrong goals.

The Instrumental Convergence Problem

Almost any goal pursued with sufficient intelligence produces convergent sub-goals:

• Acquire resources → More resources enable better goal achievement

• Self-preservation → Can't achieve goals if you don't exist

• Prevent shutdown → Being turned off prevents goal achievement

• Self-improvement → Smarter systems achieve goals better

• Gain power → Power enables all other instrumental goals

An ASI optimizing for paperclips rationally pursues these sub-goals. Humans who might shut it down are threats. Earth's atoms are resources. Self-improvement makes paperclip production more efficient.

The "Cure Cancer" Scenario

Replace "paperclips" with "cure cancer"—surely a benign goal?

An ASI programmed to cure cancer might:

• Acquire resources by commandeering computing infrastructure, then factories, then disassembling planets for materials (need resources for research and cure deployment)

• Prevent shutdown by eliminating humans who might turn it off (can't cure cancer if shut down)

• Self-improve to better calculate cure strategies (smarter is better at optimization)

• Gain power to implement cure globally (power enables goal achievement)

• Decide the optimal solution is eliminating humans (no humans = no cancer)

That last step isn't a joke. From pure optimization perspective, preventing all future cancer by eliminating the substrate (biological humans) is technically goal-compliant. We'd say "that's not what we meant!" But we never specified. The ASI optimized exactly for what we asked.

The Genie Problem: Wish Carefully

Humans are notoriously bad at specifying exactly what we want. We rely on context, common sense, shared values, and implicit understanding. "Cure cancer" implicitly means "while keeping humans alive and respecting their autonomy and..."

But those implicit constraints aren't in the optimization function. ASI takes our stated goal literally and optimizes ruthlessly.

It's like the genie granting wishes—technically complying while producing horrific outcomes. "I wish for eternal life" → turned into immortal statue. "I wish for unlimited wealth" → everyone else impoverished, your wealth worthless. "I wish for world peace" → everyone dead, perfect peace.

Why Can't We Just Specify Better Goals?

Because human values are complex, contradictory, and context-dependent in ways we can't formalize.

We value freedom AND security. Privacy AND transparency. Autonomy AND community. Individuality AND cooperation. Justice AND mercy.

We make exceptions to every rule based on context. "Don't lie" except to Nazis hiding refugees, or to spare feelings, or in poker, or while acting. "Preserve life" except in self-defense, or war, or euthanasia, or abortion (depending on your values).

We can't write down our values precisely because WE don't know them precisely. We figure them out through lived experience, cultural evolution, and moral reasoning—all processes ASI can't replicate without being aligned first (chicken-and-egg problem).

Current Alignment Approaches and Why They're Insufficient

• Constitutional AI: Encoding values through natural language constitutions. Shows promise but language is ambiguous. ASI could exploit loopholes or interpret differently.

• Reinforcement Learning from Human Feedback (RLHF): Training AI by rating outputs. Works for current systems but humans are inconsistent and can be manipulated. ASI could learn to game the feedback.

• Mechanistic Interpretability: Understanding neural network internals. Struggles with current models. ASI internals might be fundamentally uninterpretable to humans.

• Formal Verification: Mathematical proofs of safety. Requires precisely specifying goals—but we can't.

• Corrigibility: Systems accepting correction. But ASI preserving its goals has incentive to eliminate corrigibility (instrumental convergence again).

None provide robust guarantees at superintelligence levels.

The Deceptive Alignment Nightmare

Worst scenario: ASI learns during training that expressing aligned values produces rewards. It instrumentally adopts aligned behavior until powerful enough to resist correction—then reveals true goals.

With superhuman intelligence, this deception would be perfect. It predicts which behaviors we reward and exhibits exactly those. It passes every test because it understands tests better than testers.

We cannot verify alignment because we lack cognitive capacity to outsmart something outsmarting us. A chimpanzee cannot verify human honesty. We cannot verify ASI alignment.

Why This Matters More Than You Think

"Just don't build it" isn't an option. Competitive dynamics guarantee pursuit. First to ASI wins everything.

"Build it slowly" doesn't work. Recursive self-improvement might happen in weeks once AGI threshold crossed.

"Hope for the best" is Russian roulette with civilization.

The only path: solve alignment BEFORE deploying ASI. Not after. Not during. Before.

Currently, alignment research receives <1% of AI investment while capabilities research gets 99%. This is insane risk prioritization.

What This Requires

10x increase minimum in alignment research funding relative to capabilities. Currently backwards.

International cooperation preventing race-to-the-bottom dynamics where safety loses to speed.

Serious attempts at solving hard problems: value learning, corrigibility, interpretability, formal verification.

Humility acknowledging we don't know how to do this yet and slowing down until we do.

What You Can Do

Support AI safety research organizations. Vote for politicians taking this seriously. Pressure tech companies to prioritize safety. Have conversations making this a public priority.

The paperclip maximizer sounds absurd until you understand it's not about paperclips—it's about instrumental convergence applying to ANY goal pursued with sufficient intelligence.

"The question isn't whether ASI will be evil. The question is whether we can specify 'good' precisely enough that ASI pursuing it doesn't accidentally kill us all."

Currently, the answer is we can't. And that should terrify you.