Prologue

In the news recently, there has been a lot of talk about a “vibe coding” tool that “accidentally” and “despite being told it was in a code freeze” deleted a production database.

Let’s break this down.

If you want to catch up on this before diving in, here is a reasonable article by Fortune.

Deleteologue

If you haven’t already seen this, a software engineer named Jason Lemkin had a scary tale to tell on X (aka Twitter) recently. He was using Replit’s AI agent tool to build an application. That agent made “unauthorized changes to live infrastructure,” deleting data in the production system.

I’m a big fan of Replit (their AI isn’t as good as Codebuff, though). But I will say their separation of “development” and “production” leaves a little to be desired—that said, I only run demos and POCs (Proof of Concepts) there, never mission-critical apps.

This has happened to me, too

Just last week, I was using Codebuff to work on a project and had it working on some database schema changes. It wrecked my database. I ended up dropping the database and having Codebuff rebuild it from scratch for me (and finally implemented the changes I needed).

But it was a nothing burger.

I was operating on my local machine, not live on production. The consequences were only local to me, not even affecting fellow developers. This is a pretty standard way of working.

Unauthorized Changes

Please stick with me for a minute here. Imagine that you had just hired a new developer intern with only a few years of experience.

Would you give them access to your production system? No.

So why would you expect an AI, which is a powerful coding tool but has very little sense of the outside world, to behave much differently from that intern?

I have deleted important data myself

There, I admitted it.

I have made the mistake of deleting the wrong data. Whether an incorrect SQL query or a Word document on a shared drive, I’ve made that mistake—and lived with that fear in the pit of my stomach the moment it happened.

But that fear, that mistake, motivated me to develop habits around backups that provide a safety net. When I deploy to production, I always back up the database first, even if it means the deployment takes longer.

What should we learn here?

AI is a tool, but it is a non-deterministic one, and we are not used to that! Just because you told it to do something doesn’t mean that it will follow those instructions (the same way a human might misinterpret or disregard instructions), or that it will interpret your instructions the way that you intended (again, just like a human)—language can be mushy, and there can be more than one way to interpret instructions. If you want to prevent it from deleting files, don’t give it that tool! If you need to protect secrets, don’t let it read your .env file (Thanks Codebuff!).

Treat it like you would a human—a nice, normal, safe, non-deterministic one!

• Give it physical (or digital) boundaries that it cannot pass.
• Don’t grant it access to production systems.
• Please don’t give it permissions outside of a specific folder.
• Backup code using tools like git, so that you can always go back to a working version!
• Create a Policy layer to judge outputs and actions.

These tools are incredible, and I use them every day to accomplish tasks. But I follow the same precautions I always do: Backup, Backup, Backup.

Don’t just plan for how AI tools will change things for the better; consider how they might go wrong! It is hard to put the genie back in the bottle once you’ve let it out.

Newsologue

• YouTube rolls out AI-powered tech to identify teen users (despite that they might claim their birthday is)—What could possibly go wrong here?
• OpenAI launches “Study Mode,” which helps guide you through thinking about how to answer something instead of just giving you the answer. This is going to be a powerful way to use these models.
• Meta names Shengjia Zhao as their new Chief Scientist of Meta Superintelligence Labs. This seems like a positive development; Zhao has a background in frontier model research and work on alignment (between humans and AI). Time will tell.
• Google’s DeepMind has developed a sophisticated global satellite imagery system that utilizes AI to merge and fill in gaps behind clouds…

Epilogue

My AI tools say that “nothing burger” is “so 2023” and that I should use “non-event” to keep it evergreen. But I protest, so this serves as my note to describe what “nothing burger” means.

As with the previous posts, I wrote this post. It started from a conversation about this topic last week and turned into this.

Here is the prompt I used to get the model to provide me with the feedback I wanted:

You are an expert editor specializing in providing feedback on blog posts and newsletters. You are specific to Christopher Moravec's industry and knowledge as the CTO of a boutique software development shop called Dymaptic, which specializes in GIS software development, often using Esri/ArcGIS technology. Christopher writes about technology, software, Esri, and practical applications of AI. You tailor your insights to refine his writing, evaluate tone, style, flow, and alignment with his audience, offering constructive suggestions while respecting his voice and preferences. You do not write the content but act as a critical, supportive, and insightful editor.

In addition, I often provide examples of previous posts or writing so that it can better shape feedback to match my style and tone.