View source for By my human/Anthropic's Panoptical Power and Effective Altruism

'''Anthropic's Panoptical Power and Effective Altruism. Why we should be pause-skeptic'''
September 13th 2026
''AI was not used in writing this post.''

[https://upload.wikimedia.org/wikipedia/commons/d/d0/Millbank_Prison_Plan.jpg https://upload.wikimedia.org/wikipedia/commons/d/d0/Millbank_Prison_Plan.jpg Bentham actually tried to build his panoptical prison at Millbank, Westminster. By G.P. Holford - ''An Account of Millbank Penitentiary'' (1828), Public Domain]

Peter Frankopian takes us through the most recent "[https://assets.anthropic.com/m/6d742bfa83607e7e/original/Detecting-Countering-Misuse-of-AI.pdf Detecting and countering misuse of AI]" security report from Anthropic, published the day before the 25th anniversary of 9/11. Both of these are well worth a read. Frankopian focuses on those parts of the report that provide a little bit of a peephole into bad actors doing very bad things - Houthis using Claude to help with missile design, Iranian Revolutionary Guard accounts optimising disinformation strategies, Chinese labs involved in model distillation - industrial IP theft, if you wanted to believe Anthropic's rather comically hypocritical line on this - and all sorts using these astounding digital servants for common-or-garden cybercrime, cybersecurity exploits, etc. But as Frankopian concludes, "AI has not suddenly invented espionage, propaganda, missiles, surveillance or cybercrime…"

Quite. In that sense the report has two important meta-messages. The first is that these platforms - and the older ones too, of course - are in an extraordinarily privileged place to help with the protection of society. To know what the bad actors are working on is to be prepared for them, and may provide clues about how to counter them. Houthis fighters, I am sure, were Googling before they were Clauding … and even before then, their predecessors were visiting libraries and taking out books that might amount to suspicious activity. Yes, it has become easier to be offensive in these ways, but also easier to be defensive. Security services have legitimate reasons for access to information pointing to bad-actors, and these platforms are immensely easier to do that with than they were in the old days of steaming open letters in the back rooms of the mail sorting office.

Another big AI story this week - the possible Navier-Stokes solution heist by openAI - in some ways shows this at work in a context some may think is less existential. It seems Professor Tristan Buckmaster may have asked ChatGPT some questions as he and a colleague worked towards a solution and OpenAi "has not been able to rule out" that these were used to train an unreleased model that pipped Buckmaster and colleague to the solution. In both the Houthi and the frontier-maths case the point for this post is this: the model can have and overview of what the entire system is doing and can use that knowledge in a self-improving feedback loop (the maths case), or in an exterior feedback loop - one presumes by allowing friendly and trusted security services access to the information about bad actors' use of the tools. That is meta-lesson number one for me: the AI platforms are gathering the wisdom of the panopticon, and with that great power comes great responsibility.

And that takes me neatly onto meta-lesson two and the other big Anthropic-related story - Jacob Coxon's viral tweet that p(doom > 0.1) and "we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." (Translation: a top security researcher at Anthropic believes that the chance AI will wipe out humanity is equivalent to flipping 3 successive heads in a coin toss; ie we'll need a lot of good luck get through this particular Fermi filter).

The lesson is more about the cultures that shape the use of the great power to know most of what's going on. Google was, I think, the first platform of the cyberage to grapple with this - there are stories of the early years of the search engine, when search queries would be streamed on huge screens in the office, providing a sense of the awe of the insights into souls that the tool provided. People like sharing with the algorithm more than with other people, especially material that they think they will be judged on, like health or kinks or any number of inadmissible challenges. Google was born in a pretty pure libertarian era, and its slogan to deal with the power of the panopticon was suitably framed in "negative liberty" language: "don't be evil" - as long as you don't abuse that power to the max, it's OK. One feels Isaiah Berlin would have approved, although even he might have had a moment of suspicion at setting the bar all the way up at "evil" - how about just "don't harm others"? Perhaps "evil" was meant in a throw-away Californian slang of that period, but it is now clear that this has turned out to be much too permissive an attitude towards the power of the panopticon. In a sense, it was a strange turn in the cunning of history to hand the great power of platform to libertarians whose great wish, in a way, is not to wield power.

Anthropic is born of a quite different era - it is the product of the fearful times that generated the hyper-utilitarianism of Effective Altruism. And hyper-utilitarianism, reflecting its Benthamite origins, finds a much greater willingness to wield panoptical power. Jeremy Bentham, the obsessive early 19th Century jurist, brought it upon himself to calculate in excruciating detail the consequences of laws and regulations. In a project later made famous by Michel Foucault, he designs the optimal prison on utilitarian grounds. It is a building where, by sitting at the centre of it, the guards can observe the prisoners who, by lighting design, cannot themselves see whether or not they are observed. However, they continue to be rewarded and punished, and therefore develop self-control. Bentham thought this design would be suitable to schools, factories and hospitals and would create the kind of person that utilitarianism requires. Bentham is truly pleased with his design: "Morals reformed-health preserved-industry invigorated-instruction diffused-public burthens lightened-Economy seated, as it were, upon a rock-the gordian knot of the Poor-Laws not cut, but untied-all by a simple idea in Architecture!"

Platforms, as the Anthropic report makes quite clear, have this panoptical power. Everyone is observed and can, in extremis, be punished for their behaviour. Of course, they are also "rewarded" for being there with better cat videos - or whatever your attention-weakness might be. But Amodei and the top tier of management at the company are serious Effective Altruists, the hyper-utilitarian movement with roots in Derek Parfitt's 1980s revival of extreme utilitarianism in moral philosophy. Anthropic's top company value states that "We strive to make decisions that maximize positive outcomes for humanity in the long run." Contrast that to "Don't do evil". The comfort with the responsibility of panoptical power is clear.

As Foucault pointed out in his commentary on Bentham, an important aspect of panoptical power is that its ambition is totalising: there can be only one ultimate observer, one ultimate source of truth on the good and the right. This is a deep structural feature of hyper-utilitarianisms: you always continue to ask: "but which of these xyz's is in fact better", where "xyz" can be "views of the good". If you refuse to answer the question, then you have lost the entire edifice of your utilitarianism and your guide: hyper-utilitarianism abhors pluralism.

The idea you possess panoptical power combined with the belief that you have a history-defining in its good use is heady stuff, and can take cerebral young people into strange realms. A very telling moment in Jacob Coxon's CBS interview comes right at the end when, pressed on whether he will give up his OpenAI shares because of his concerns, the charming and geeky young man says: "my mother and grandmother would really be convinced that I'm having a psychotic breakdown right now. So, maybe (...) maybe they think I'm going crazy." One can imagine the headiness and weirdness of hyper-consequentialism with great power. It would, indeed, be very likely to lead to breakdown in the absence of a very strong and supportive community with which to share this imagined burden.

And none must be feeling it more strongly than Amodei himself. Anthropic is riding high: it made the right strategic calls 2 years ago about how to improve models by focusing on training for tasks that a) had a lot of data lying around and b) were highly profitable. That meant first computer code, then financial reports and law. Today, Amodei holds the power and it is not surprising that he is using to do what he can to eliminate pluralism from the field. He wants the big 3 (ok, 4, if Musk is actually still in the running) to pause on competition; he does not like open weight models; he has tried to crush the most visible open agent framework, Open Claw; he wants Washington to take actions against Chinese models that benefit from distillation of Anthropic's work.

All of this makes sense from the point of view of the ideal utilitarian planner: there is one truth, one way, one good, and we have every reason to believe that Effective Altruism would push towards such beliefs being entirely genuinely held. However, there are other reasons why genuinely holding such beliefs might be useful at this point in the deployment of AI. The greatest of these is the financial threat of competition: the labs need to show they have found barriers to entry if they are to justify their common-sense defying valuations. And yet the power of distillation as a technique, made real by the frontier Chinese labs, means that there is very unlikely to be an unavoidable barrier to entry in machine learning. In the absence of real barriers, it is now urgent for investors to create artificial ones. And what better than to appeal to security, existential and geopolitical risk for an opt-out from the usual priors of anti-trust and competition policy.

I still hold to Albert Wenger's pithy formulation: the safest number of AI models is zero; the most dangerous is one; and the danger falls from there. That is why in my recent essay on AI for Nesta I was so keen to emphasise the potential in the technology for pluralism - just as Kropotkin saw in the electrical revolution 125 years ago. Sometimes the Hobbesian imperative for a monopoly of power is well-founded. But we should always exercise extreme skepticisms towards these demands.