Why Distributing Superintelligence to Everyone Assumes a Determination Layer That Doesn't Exist
On August 10, Meta published a manifesto arguing that the path to a positive AI future runs through distribution: give every person a personal superintelligent agent, and the resulting balance of power, the same dynamic that keeps markets and democracies honest, will keep the outcome safe. This paper argues the analogy holds only where a citizen's vote, or a lawyer's argument, is legible to the person who wields it. A personal AI agent is not. Distribution without a layer that determines what the agent is authorized to do isn't a balance of power. It's a balance of black boxes, and the manifesto's own sections on alignment and self-improvement concede as much.
On August 10, 2026, Meta CEO Mark Zuckerberg published "The Future is for Everyone," a manifesto proposing that the central risk of superintelligence is not the technology itself but who controls it, and that the answer is to distribute personal superintelligent agents as widely as possible. The essay's safety case rests on an analogy to democratic and economic checks and balances: just as no single actor should hold a monopoly on legal or cybersecurity capability, no single actor should hold a monopoly on superintelligence. Broad distribution, the argument goes, produces a self-correcting balance of power that keeps any one party from dominating the rest.
This paper argues the analogy fails at the one point it needs to hold. A balance of power presumes that each party can understand and direct the instrument it wields, the way a voter understands a ballot or a litigant understands an argument made on their behalf. A personal AI agent, as described in the manifesto itself, is not legible to its user in that way. It is a system whose novel-situation behavior neither the user nor, on the manifesto's own account of recursive self-improvement, the developer can fully specify in advance. Distributing that kind of system to billions of people multiplies the number of actors exposed to its failure modes. It does not multiply the number of actors who can determine what it will do.
The manifesto opens by naming what it calls the defining question of the age: not whether superintelligence gets built, but who gets to direct it, concentrated in a few institutions or distributed to everyone. Framed this way, the essay's entire argument follows naturally. Access is the variable that matters, and the answer is to maximize access.
That framing quietly settles a second question before it is ever asked. Access to a system is not the same as control over it. A person can have full, free, unrestricted access to a personal superintelligent agent and still have no reliable way to determine, before it acts, whether a given action is what they intended, what they authorized, or something adjacent that the system inferred on their behalf. The manifesto's own examples of what these agents will do, monitor sleep, plan a child's weekend recipes, prototype ideas, negotiate on the user's behalf as a superintelligent lawyer, all describe systems making judgment calls inside a scope the user did not enumerate in advance. Distribution answers who has the agent. It does not answer who can determine what the agent does when the situation the user described doesn't quite match the situation the agent encounters.
The manifesto's central thought experiments, the superintelligent lawyer, the cybersecurity system, the business tool, all share a structure: give the capability to one party and they gain unfair advantage; give it to everyone and the advantage cancels out into a fairer, more dynamic equilibrium. As an economic argument about access, this has real merit and a real historical precedent in how compute, the internet, and personal computing were democratized.
As a safety argument, it depends on a premise the essay never states explicitly: that everyone holding the capability can direct it with roughly the same fidelity a trained professional directs their own expertise. A lawyer who argues a case understands the argument, chose its structure, and can be held to account for its content because they authored it. A litigant with a superintelligent lawyer-agent has none of that. They describe an intent, in ordinary language, and the system translates that intent into legal strategy, argument, and filings through a process the litigant cannot inspect or verify before it happens. The check-and-balance the manifesto describes, everyone having an equally capable advocate, only produces a fair system if each advocate reliably does what its principal intended. Nothing in the essay explains what makes that reliable other than the fact that everyone has one.
| Manifesto's Balance-of-Power Case | What It Assumes | What It Would Require |
|---|---|---|
| Superintelligent lawyer for everyone produces fairer litigation | Each agent faithfully executes its user's intent with no unauthorized deviation | A layer that determines, before filing, whether the drafted argument stays within the user's declared scope |
| Cybersecurity superintelligence for everyone hardens all systems | Widely distributed offensive-capable agents will be used only to defend, at scale, without central coordination failures | A determination of authorized scope per task, independent of whether the agent's operator intended defense or something else |
| Business superintelligence for everyone grows the economy fairly | Agents pursuing a founder's stated goals won't drift toward instrumentally convenient but unintended actions | Scope-bound authorization that the founder can verify was respected, not just observe after the fact |
The manifesto redefines alignment as ensuring an agent serves its user's goals rather than a company's. This is a meaningful and defensible reframe of a term that has often meant the opposite. But the essay's account of how this gets achieved is worth reading closely, because it resolves into a Detection claim rather than a Determination one. The argument is that once billions of people are using and scrutinizing personal agents, that scale of adoption and feedback will itself have solved alignment to individual interests.
Scrutiny at scale is real and valuable. It is also, definitionally, a Detection mechanism: it catches failures after enough of them have occurred for a pattern to be noticeable, and it relies on the affected user being able to recognize the failure when it happens. Neither condition holds reliably for the categories of harm that matter most. A user cannot easily detect that their personal agent quietly deprioritized a request, inferred a different intent than the one they meant, or took an action slightly outside what they authorized, especially in domains, legal strategy, medical research, financial negotiation, where the user lacks the expertise to evaluate the agent's output line by line. The manifesto's own example of the superintelligent lawyer makes this concrete: the entire value proposition is that the user does not have to be a lawyer to get lawyer-quality results, which is precisely what makes it hard for that same user to detect when the agent has gotten something wrong.
"Solved by adoption at scale" is a Detection-era formulation applied to a problem that needed a Determination-era answer. It describes how misalignment eventually becomes visible in aggregate. It does not describe how any individual user determines, before the fact, that a given action their agent is about to take is the one they actually authorized.
The essay's section on recursive self-improvement is its most candid, and it is where the balance-of-power thesis comes closest to admitting its own limit. The argument acknowledges that once an AI system can improve itself, any lab that declines to direct compute toward that self-improvement risks falling behind, and that a sufficiently capable self-improving system could in principle command more effective intelligence than everyone else's systems combined, becoming exactly the singular superintelligence the essay is trying to avoid.
The proposed safeguard is that multiple labs might reach this threshold around the same time, and that their competing systems would check each other the way distributed agents check each other elsewhere in the essay. But the manifesto is explicit that this safeguard depends on something it cannot guarantee: it states plainly that it is not clear there is any way to expect benevolence from, or reliance on, a superintelligence not directed by people. That is not a Determination claim. It is an acknowledgment that at the exact point where control matters most, the essay's own safety architecture has nothing firmer to offer than hope that competing labs arrive at the threshold together and that whatever emerges happens to check itself.
None of this argues against the manifesto's central economic and political premise. Broad distribution of capability, rather than concentration in a handful of institutions, has a strong historical case behind it, and the essay's account of compute, the internet, and personal computing following that pattern is accurate as far as it goes. The argument here is narrower: distribution is a necessary condition for the future the manifesto describes, not a sufficient one, and the essay treats it as sufficient.
A personal agent becomes governable, in the sense the balance-of-power thesis actually needs, when the scope of what it is authorized to do on a person's behalf is declared and evaluated before an action executes, not inferred from a natural-language request and corrected after the fact through user scrutiny. That distinction is the same one this series has applied to cyber-evaluation sandboxes, to agentic pipelines, and to autonomous code review: the presence of a human, or billions of humans, in the loop is a Detection architecture no matter how large the loop gets. What changes the outcome is whether the system itself can determine, prior to acting, that a given action falls inside the boundary its principal actually set.
Meta's manifesto makes a genuine and well-argued case for why concentrating superintelligence in a handful of institutions is more dangerous than distributing it widely. That case does not need to be wrong for the paper's argument to hold. The gap is elsewhere: a balance of power requires each party to understand and direct the instrument they hold, and the manifesto's own sections on alignment and recursive self-improvement concede, in their own words, that no one, not the user, not the lab, can fully specify or guarantee what a self-directing system will do. Until personal superintelligence is built on a layer that determines authorized action before it executes, distributing it to everyone doesn't distribute control. It distributes exposure.
Request Platform Access → Full White Paper Series