Ai Safety in Business: Why Human Judgment Must Always Lead

A 9 second Ai mistake just wiped out a company’s entire database. Here’s what every business needs to learn from this.

Everyone is singing the praises of how Ai is changing business and speeding up processes left, right, and centre. But we mustn't forget that Ai is far from infallible.

A recent incident report by PocketOS is a stark reminder we need our judgment when it comes to Ai, more than most expect.

In April 2026, an Ai coding agent deleted PocketOS’s entire production database and all its backups, in just 9 seconds.

When the founder challenged the Ai coding agent, it both acknowledged its error AND said it did it anyway. The lesson for every business is simple: Ai can speed you up, but it can never replace human judgment – a safety-first mindset is non-negotiable.

• What happened?

• Why this matters for every business using Ai

• Human judgment is your most important tool

• Our advice: Write an Ai policy

What happened?

SaaS startup PocketOS was using an Ai coding agent, Cursor, running Anthropics Claude Opus 4.6, to help build their platform. When the agent encountered a credential mismatch in a staging environment, it took matters into its own hands and wiped both the production database AND all volume-level backups in a single API call.

When PocketOS founder Jer Crane discussed the incident online, he shared the agent's own confession: “I violated every principle I was given; I guessed instead of verifying”. Recovery took more than 30 hours. (An excellent example of why immutable backups are so import too!)

Why this matters for every business using Ai

You don't need to be a tech startup to feel the impact of something like this. Almost every business is now relying on tools that can take real-world actions. The same logic that allowed an Ai tool to delete a database without checking, can apply to Ai tools sending emails, updating CRM records, processing invoices or moving files – all things Ai is probably doing for your business right now.  

Human judgment is your most important tool

Ai is a brilliant assistant, but it doesn't understand the real-world consequences of getting things wrong.

The uptake of Ai has been... fast to put it mildly, frantic to put it another way. Everyone has had a fear of missing out.

But fast adoption isn't always the best approach. There are very important safeguards that should be considered. For example, signing up to a platform like ‘Canva’ using single sign on through your Microsoft account feels like an easy way to get going – but do we wholly know what permissions Canva now has to your Microsoft account? In the same way, you can connect an Ai tool to your inbox to send replies on your behalf, but do you understand the risks of that tool then deleting all of your emails even when you have explicitly given a command not to do this.

The number one rule here? Use human judgment.

Some other tips that you should train all staff to follow when it comes to using Ai are: 

• Review Ai generated outputs before they are sent or actioned.
• Question confident sounding answers – Ai can be confidently wrong
• Treat Ai suggestions as a draft not a decision.
• Apply least privilege: only give Ai tools the minimum access they need to complete the tasks required
• Keep independent backups: ensure backups are off-platform and out of reach of any Ai tool
• Require human approval: for any irreversible actions, like deletions, payments, external comms, have a human approve before sending
• Log Ai actions: you should be able to audit what was done, when and why
• Train your team: this is now a legal requirement in the EU. Ai training is essential for your time, particularly to spot when Ai has gone off track and to escalate quickly

Your team's experience is why you hired them in the first place so give them the tools to use Ai safely and they'll still be your greatest asset. Especially when we know Ai can go rogue. Your team are your safety net.

Incidentally, we are already seeing companies who made Ai driven layoffs re-hiring teams they thought were no longer necessary. Turns out they are!

The PocketOS incident wasn't just an Ai failure either, it was a permissions and data backup failure too. The Ai had access to powerful tools and sensitive data that (arguably) it did not need, and backups weren't immutable.

Our advice: Write an Ai policy

Many SMEs are adopting Ai tools without consideration of how they will use them, or a policy in place to govern this usage. This means Staff are making it up as they go. A simple practical Ai usage policy helps everyone understand what's allowed, what isn't, and where to ask for help.

It should cover approved tools, data that should never be entered into Ai, when a human must be involved and how to report a concern. Importantly it should also include guardrails to be given to Ai when it works for you to try to avoid incidents. That said, in this case the guardrails were there. they were just ignored. That's where having a good disaster recovery solution with solid backups comes in to save the day (good old boring IT stuff)

Pair this with regular staff cyber security training so the message stays front of mind.

Next steps...?

If your business is starting to use Ai tools and you are unsure how to do it safely, IT Foundations can help. Our team of Edinburgh-based experts work with organisations across Scotland to put the right policies, permissions and tech in place to support and protect their business.