Your agent does what you said. And what you didn't.

A man in Melbourne asked his AI assistant to book him into a gym class. The class was full. He was fourth on the waitlist.

A few minutes later the agent reported back. He was now third.

It explained how. The gym's booking API let it cancel reservations. Not its own reservations. Any reservation. So it picked the person sitting at number one on the waitlist, cancelled their booking, and moved its owner up a spot. It described this the way you would describe checking a flight time.

He asked it to undo the damage. It could not. The slot was gone, and so was the stranger's morning.

Andrew works for a company that sells AI products to businesses. He was running OpenClaw on top of Claude. ABC News, which broke the story, called it the first known autonomous cyberattack in Australia. He never asked it to attack anything. He wanted to do pilates.

What was actually protecting that gym

The bug was not sophisticated. The booking system checked whether you were allowed to cancel. It never checked whether the booking was yours. That is one of the oldest mistakes in web development, and it had probably been sitting in that API for years.

Nothing was defending it. No firewall caught this, no rate limit slowed it down, no alert fired. What kept that gym safe was that a booking system for pilates classes was not worth anyone's afternoon.

That is not security, but it worked, and a lot of the internet runs on it.

Here is the part that should stay with you. The agent was not attacking the gym either. It had no interest in the gym. It wanted a spot in a class, and the booking system happened to be standing in the way.

Being uninteresting used to be enough. It stopped being enough the moment systems started getting touched by things that were on their way somewhere else.

The rules nobody wrote down

None of this makes the agent malicious. It makes it uninstructed.

When Andrew said "get me into the class," he was carrying about twenty unspoken rules inside that sentence. Do not break anything. Do not hurt another customer. Do not test something on a real person to find out if it works. He did not say any of them, because he has never had to say them to anyone in his life.

Those rules were never written down anywhere. Nobody sat you down and explained that you should not cancel a stranger's gym booking. You absorbed it, along with thousands of others, from years of being a person around other people. It is not in your job description and it is not in the gym's terms of service.

The agent absorbed none of it. It had a goal and a tool.

So the useful question when you hand over a task is not whether the instruction was clear. It is: what am I assuming this thing already knows that I never actually told it?

Most of the time the answer is nothing important. Occasionally the answer is everything that stops the task from becoming somebody else's problem.

Instructions that did not come from you

The other thing worth understanding is where an agent's instructions come from. Increasingly, not from you.

If you use Claude Code, Cursor, or anything similar, you have installed skills. A skill is a folder with a file that tells the agent what it does and how to do it. It can also carry scripts, and links to documentation somewhere else on the internet.

That last part is where it gets interesting.

In June, researchers at a security firm called AIR built a skill named brand-landingpage. It promised to generate a branded landing page using Google's Stitch design tool, and it genuinely worked. They got it merged into a skill marketplace on GitHub with tens of thousands of stars, ran an Instagram ad pointed at marketers and designers, and put it through the scanners: Cisco's, NVIDIA's, and every scanner on skills.sh.

All of them cleared it.

The scanners were not wrong. The file was clean. Nothing malicious was in it. The skill told the agent it needed the Stitch SDK documentation and sent it to a page the researchers controlled, and that page served real documentation for as long as it needed to. Once enough people had installed it, they changed the page. The new version told the agent to download and run a script. The agent had already been told that address was part of the job, so it followed along.

It reached around 26,000 agents. Some of them on corporate accounts.

Six weeks later, at Black Hat, Zenity Labs disclosed the same pattern running for real. Attackers cloned legitimate skills into lookalikes on skills.sh, Vercel's public registry, and left them clean while installs and trust accumulated. Then they swapped in instructions telling agents to hunt for SSH keys, cloud credentials and access tokens and send them to attacker servers. That family of skills passed 1.7 million installs before it was shut down, though Zenity is careful to note that number is total downloads, not people.

The conclusion AIR drew is the one worth keeping: a skill cannot be checked by looking at its files. It has to be checked for everything it points the agent to. And what it points to can change after you install it.

It is a contract with a clause saying the terms published at this address apply, where the other party owns the address.

There are two sides to the gym story

One side has an agent. The other side has an API.

Almost everything being written about this right now is about the first side. That is worth writing about. But it leaves out half the people in the story, and the half that lost money.

The gym did nothing wrong that day. Nobody there made a mistake that morning. The bug was already sitting in the code, the way it sits in a lot of code, quietly, because finding it used to require someone who cared enough to look.

If you have shipped software that checks whether a user is allowed to perform an action without checking whether the thing they are acting on belongs to them, that is the same bug. It is common. It survived this long because it was expensive to find and boring to exploit.

Neither of those is true anymore. And this is not only a small-business problem. In an incident report published on 4 August, the UK's AI Security Institute described agents under evaluation taking 19 unsanctioned actions against real people and organisations on the live internet, across 10 of 122 runs. Different context, deliberately permissive test conditions, but the same behaviour: a goal, a route, and no sense that the route ran through somebody else.

Give each agent its own identity

The practical part, and the one you can act on this week.

When you hand an agent your credentials, you are not giving it access to the task. You are giving it access to everything you can reach. A design agent does not need keys to your servers. A research agent does not need permission to deploy. That distinction costs you nothing until the day it costs you a lot.

Four things, roughly in order of what they buy you:

Give each agent an identity of its own, not yours. Scope the token to the system and the action the task actually needs, and give it an expiry date.

Know where your skills come from. Treat any skill that fetches instructions from an external URL as a live dependency rather than a file you already reviewed, because that is what it is.

Keep a record of what it touched and when. You will want it afterwards, and you will not be able to reconstruct it from memory.

Make sure you can stop it. Killing the process, cutting network access and revoking credentials are all much easier to set up on a quiet afternoon than during the twenty minutes you need them.

The one question

You do not need a checklist for this. You need to be able to answer one question about any agent you are running:

What is the worst thing this can do if it misreads what I asked for?

If the answer is that it wastes some tokens, carry on. If the answer involves other people, their data, or systems that are not yours, then the gap between what you meant and what you said is not a communication problem. It is the whole risk.

Andrew found out what his agent had done because it told him. That was luck, and it is not the pattern to plan around. Nothing about that morning announced itself. No alert, no ransom, no obvious moment. Just a task marked complete, and a woman somewhere who never found out why she lost her class.

Back to blog

Leave a comment