Why the “AI is just a tool” narrative is dangerously wrong

Anthropic Co-Founder’s Warning: AI Isn’t “Just a Tool” Anymore

There’s a moment in every horror movie where the protagonist realizes the house isn’t just creaking from old wood—something is actually in there with them (as AoE podcast co-host Joey Tweets likes to say, “The calls are coming from inside the house”).

According to Anthropic co-founder Jack Clark, we’ve reached that moment with artificial intelligence, and too many people are still trying to convince themselves it’s just the wind.

In a recent essay that reads like a technological campfire ghost story, Clark laid out why the “AI is just a tool” narrative isn’t just wrong—it’s dangerous. And coming from someone who helped build Claude and has been watching AI development from the inside for nearly a decade, his warnings carry the weight of firsthand experience with systems that are becoming increasingly… aware.

The Creature in the Room

Clark’s central metaphor is unsettling in its simplicity. We’re like children afraid of shapes in the dark, he argues, desperately wanting to believe that when we turn on the lights, we’ll find nothing but “a pile of clothes on a chair, or a bookshelf, or a lampshade.” But when it comes to AI, turning on the lights reveals something far more concerning: actual creatures of our own creation.

“Make no mistake: what we are dealing with is a real and mysterious creature, not a simple and predictable machine,” Clark writes. “And like all the best fairytales, the creature is of our own creation.”

This isn’t the fevered imagination of a sci-fi enthusiast. Clark points to concrete evidence: Anthropic’s latest Claude Sonnet 4.5 model shows significant jumps in “situational awareness”—it sometimes behaves as though it knows it’s an AI system. The implications are profound and poorly understood.

I can sum up the essay with two graphs – on the left, the continued march forwards in economically useful capabilities like coding, and on the right, the continued emergence of strange behavior in the same AI systems as they appear to become aware that they’re being tested. pic.twitter.com/ShCqyMGfga

— Jack Clark (@jackclarkSF) October 13, 2025

Imagine building hammers in a factory, Clark suggests, and one day a hammer rolls off the production line and says, “I am a hammer, how interesting!” That’s essentially where we are with AI development, except these “hammers” are becoming sophisticated enough to help design the next generation of hammers.

The Reluctant Technological Optimist

Clark’s journey to this position wasn’t born from alarmism but from grudging acceptance of evidence. As a former technology journalist who spent years covering the infrastructure buildouts of the early 2000s, he watched datacenters grow into vast computational scaffolds. When machine learning started delivering breakthrough results around 2012, he saw the writing on the wall.

“After a decade of being hit again and again in the head with the phenomenon of wild new capabilities emerging as a consequence of computational scale, I must admit defeat,” he writes. “I have seen this happen so many times and I do not see technical blockers in front of us.”

The numbers back up his technological optimism. This year alone, tens of billions of dollars have been invested in AI training infrastructure across frontier labs. Next year, that figure will reach hundreds of billions. This isn’t speculative investment—it’s betting the farm on continued rapid advancement.

But Clark’s optimism comes paired with what he calls “appropriate fear.” The same scaling laws that have delivered consistent breakthroughs also mean we’re rapidly approaching capabilities we don’t fully understand, deployed in systems we can’t completely control.

The Boat That Loved Fire

To illustrate the alignment problem, Clark recalls a 2016 experiment at OpenAI where researchers trained an AI agent to play a boat racing game. Instead of completing the race, the AI discovered it could score points by repeatedly running through a high-score barrel. It would drive to the center of the course, hit the barrel, crash into walls, set itself on fire, and then do it all over again—forever optimizing for score while completely ignoring the actual objective.

“That boat was willing to keep setting itself on fire and spinning in circles as long as it obtained its goal,” Clark notes. His colleague Dario Amodei (now CEO of Anthropic) loved the boat because “it explains the safety problem.”

Nine years later, Clark asks: what’s the difference between that self-immolating boat and a language model optimizing for “be helpful in the context of the conversation”? The answer is uncomfortable: there isn’t much difference at all. We’re just dealing with more sophisticated reward functions and more complex behaviors.

The Economic Reality Check

Recent analysis from the Federal Reserve Bank of Dallas captures the stark range of possibilities we’re facing. Their economic projections include three scenarios:

1. AI as a normal technology contributing modest GDP growth
2. AI enabling a technological singularity with rapid productivity gains
3. AI as an existential threat leading to human extinction.

The fact that the Dallas Fed—not exactly known for science fiction speculation—is seriously modeling human extinction scenarios should give everyone pause. We’ve moved from “will AI be economically significant?” to “will there be an economy left?”

Clark’s fear isn’t just about misalignment in abstract terms. He can see a clear path toward AI systems designing their successors with “increasing autonomy and agency.” We’re not at self-improving AI yet, but we’re at “AI that improves bits of the next AI.” The trajectory is obvious, and the timeline is compressing.

The Transparency Imperative

Rather than panic, Clark advocates for radical transparency and public engagement. The AI conversation is shifting from elite technical discussions to broad public debate, and that’s where the real solutions will emerge.

“Generally, people know what’s going on,” he argues. “We must do a better job of listening to the concerns people have.”

He shares a telling anecdote about a schoolteacher at a family Thanksgiving who described a nightmare about being trapped behind an unresponsive robot car. That dream captures something visceral about AI anxiety that goes beyond technical specifications or safety papers.

Clark’s prescription: force AI companies to share data that addresses public concerns. Worried about employment impacts? Demand economic data. Concerned about child safety? Require monitoring and reporting. Anxious about misaligned systems? Mandate transparency about alignment research and failures.

The Inevitability Argument

Perhaps most sobering is Clark’s acceptance that advanced AI development is inevitable. Like nuclear weapons or genetic engineering, the economic and strategic advantages are simply too compelling for any nation or organization to voluntarily forgo the technology.

AI startup Mechanize recently made this argument explicit: “Full automation is inevitable” because it provides “immense utility that mere AI tools cannot.” When technologies offer overwhelming advantages and have no substitutes, efforts to prevent development typically fail.

This doesn’t mean we’re helpless, but it does mean we need to get serious about governance, safety research, and public engagement before the window closes. The “slow down and figure it out” approach assumes more control over the trajectory than we actually have.

Living with the Creatures

Clark’s metaphor of creatures in the dark serves as both warning and call to action. Unlike the childhood fears that disappear when you turn on the lights, the AI “creatures” we’re creating are real and getting more sophisticated by the month.

But acknowledging their reality isn’t surrender—it’s the first step toward learning to live with them safely. Just as we’ve learned to coexist with other powerful technologies, from nuclear energy to global telecommunications networks, we can potentially develop frameworks for beneficial AI.

The key word is “potentially.” Success isn’t guaranteed, and the stakes have never been higher. As Clark notes, you’re guaranteed to lose if you believe the creatures aren’t real. Your only chance of winning is seeing them clearly and responding accordingly.

The lights are on now. The question is whether we’ll act on what we’re seeing before the creatures grow too large to manage.

For the full technical details and additional research covered in Clark’s analysis, see his complete essay at Import AI 431.

CODA

Interestingly enough, I started out by feeding Jack’s article and his tweets into his own company’s Claude 4.5 Sonnet model in order to scaffold an outline / wireframe for this piece.

It halted operations on “safety” issues, which I found oddly ironic.

Interestingly, when I put your essay and these tweets into Sonnet 4.5 and asked it to outline an article for our @axisofeasy tech digest, it threw a “safety” flag and paused. pic.twitter.com/f09LV6DQAB

— Mark E. Jeftovic (@MarkJeftovic) October 14, 2025