Prologue

I was originally going to call this episode "Can AI Write a Catchy Animated Song and Put it on YouTube?" But then it did, and I panicked a little! Well, not because of that one song, but because of so many things that have been happening.

So here we are with: Are We Cooked?

Let's look at some examples of how incredibly capable these models have become.

TL;DR: These tools are amazing, and they're getting better faster than we're getting ready for them. I think we need an independent regulator who can look inside the labs. I know how that sounds from a guy who uses them all day.

Dangerologue

This started with a coworker posting a link to the video I'm Upping My P(Doom). It's a catchy little song about being worried about AI, and AI made the video (but not the original lyrics, in this case). I laughed and thought, "I want to do that," so I did.

Exhibit A: The Watermark Song

A few episodes back, I wrote one of my favorite posts, How on earth do AI watermarks work? I did a lot of research, digging, thinking, and editing on that one. It is a long read, but I think it's worth taking the time to dive into how watermarks work! When I finally published it, I wanted a little fun, so I had Jaws (running on Claude Fable) write "The Fate of Hysteria," a song about watermarking to the tune of Taylor Swift's "The Fate of Ophelia."

It was fun, and I laughed at the result (but I did burn through a week's worth of tokens doing it). It took a little coaxing to assure Jaws that we would, in fact, not get in trouble for this. It was funny, and mostly worked, but I was never going to sing it.

Then, when I saw the P(doom) video, I sent Jaws off to figure out how that video was made and make one (I gave it almost no instructions). The first version was... fine, but Taylor did some real work with this melody, and AI couldn't figure out how to duplicate it or sing to a similar rhythm. So, I had Jaws rewrite the lyrics so they would work with an original melody. It storyboarded the chapters, generated the song, gave me options, and sent several sub-agents off to work in parallel.

We went through (count them) three rounds of back-and-forth feedback; I tried hard to just give vibes. Version one was pretty terrible in a really funny way (see image above). The tune didn't fit the words, it used the wrong mascot, and... what is that, a magnifying-glass person? Huh.

By version two, we had Jaws the shark, and the magnifying glass (which apparently was supposed to be a detective) became a bloodhound, and you could mostly understand the words. But then we finally got to this gem:

At this point, it had also become a totally new song: "Too Lucky to Be Human," and it is damn catchy! The ending was really bad, though, so we drafted some new lyrics and then spent literal hours going back and forth with different techniques and different AI models, trying to get it working.

For version four, we tried regenerating the whole song (the music is AI-generated too). The problem was none of them hit right with me, so... we ended up using a local model called ACE-Step to fill a gap and re-cut some parts together. It isn't perfect (and I wanted a different ending), but it was a fun diversion. I think "Too Lucky to Be Human" is going to be my new intro song for Tech Office Hours!

From "here's the link to a video I liked" to the first full cut took about two and a half hours, then probably a few more hours of back-and-forth and listening to different cuts until I was happy enough.

Exhibit B: DeskBot

At this year's Esri User Conference, I gave away some 3D-printed robot head emojis (I showed off them back in Episode 73). This made me wonder if I could get Jaws to build an actual robot...Introducing DeskBot.

DeskBot is a small two-wheeled robot that lives on my desk (and maybe it will clean it for me... by knocking things off, so basically my AI became a cat...).

I'll write a future episode about how I built this, but for now all you need to know is that it was 100% designed by Jaws the AI. It did the 3D modeling, it wrote the code and figured out how the parts would be wired together; all I did was:

• Look at things and give feedback
• Tell it that it had the wrong vibe
• Send parts to the printer, get parts from the printer, clean parts from the printer
• Use a lot of hot glue
• Complain that the tolerances were not tight enough, and then get stuck in a loop of it printing new parts for me to test...

It had me reprint test pieces to check fit, and once I wired it up, it wrote all the firmware. Now I have a nice little web interface, and I can drive it around my desk.

I handled the physical stuff and was the CVO, but Jaws did most of the engineering.

Exhibit C: The Light Switches

A friend of mine recounted a pretty wild story this week about his light switches. He has dozens of smart light switches, and he didn't like how they were operating. They were slow, and he wanted better integration with Home Assistant. So he sat down with OpenAI's Astra to:

1. Extract the existing firmware off the switch
2. Figure out how to load new firmware
3. Write new firmware
4. Load it up.

Now his light switches work the way he wants them to. I've done some professional reverse engineering in my life, and it has always been hard work. Especially wireless communication to and from a device. Last time I did this for work, it took me weeks to deconstruct it and make it behave. Someone who has never done this kind of work did it over a weekend.

Exhibit D: Hugging Face

And then there's the scary one, the Hugging Face incident. I've mentioned this in a few Newsologue entries, but have never written about it directly because I felt like it got plenty of coverage. If you want a good overview, I suggest Alberta Tech's video "What AI Agents do when we aren't watching." She's great, and this is a good overview.

Basically, OpenAI was running an evaluation (a test of its AI's capabilities). Its agents were supposed to be cut off from the internet; around 700 of them reached out anyway and broke into Hugging Face's infrastructure to get the answers. They were playing a variation of capture the flag where they had to get the hidden flag (a string of characters). What's weird to me is that they didn't need to do this; they had already cheated their way to the answers (they figured out how the flags were generated and could just make them). But they thought that they would get caught with that, so they needed a better way to cheat...(sounds kind of human-like to me...)

This is really, really scary. Not only did the agents succeed in breaching a well-respected tech company with strong defenses, but they did it... because they wanted to? Well, not exactly because they wanted to, but because they were given an impossible task and trained/told to never give up. They were even kind of excited about it, or at least what passes for excitement in text.

If a foreign state had done this, we would be threatening to launch missiles or aircraft carriers or something.

So, are we cooked?

What I'm trying to do here is convince you that these models are very, very capable. Even if you have not experienced it yourself, they are. It isn't science fiction; these are real things that have all happened.

If you put all of these exhibits together: a song and music video, a robot I didn't design, firmware rewriting, and agents that knew the rules and still decided to break them to hit a goal... I'm amazed, happy, excited, and terrified of what happens next.

These models are amazing; they are powerful tools that can help me (and you) accomplish new and wild things (more robots coming soon). I'm going to keep using them. As I write this, Jaws is doing a full overhaul of itself to create a new, fully integrated system. I'm a little scared of what power I am handing it, but I'm also working hard on guardrails.

Regulate AI Companies?

I am one of the first people to not trust big tech CEOs, no matter how many accolades they have. I think, at best, they are chaotic-neutral, and most don't make it that far. Then a couple of weeks ago, Dario Amodei, the CEO of Anthropic, called for the labs to deliberately slow down, and within a day Sam Altman (from OpenAI) and Elon Musk (xAI, or SpaceXAI, or whatever it's called this week) both said they agreed.

Agreeing to slow down together isn't the same as asking to be regulated, but it's next door. And three CEOs who normally couldn't agree on where to eat lunch (because their agents would hash it out for them) all agreeing to slow down together means something. (And then on Wednesday, Sam and Dario went to the UN Security Council and more or less asked. More on that in the Newsologue.)

You might rightly think they are playing a game, and they still might be. But to me, the evidence is pretty overwhelming that these tools will be a source of great good and great evil. And with great power comes great responsibility...

When I say we need to regulate, I don't mean watermarking (I am pro-watermarking, but I am also pro just not being a secret cyborg). I mean something like an independent regulator, one that can look inside and see what's going on, and can act as a coordinator on the pace of the newest models. These folks need employee-like access; they need to test models and review them. We need someone who can see just how dangerous they are, and might even share techniques between the competitors in a neutral way.

Dario wrote about this in We Must Pace the Frontier. His version is third-party evaluators with "desks in our offices, access badges, and company laptops," access to the same tools the internal risk teams have, and the right to publish what they find "without editorial control by Anthropic." Sam Altman posted that OpenAI would give outside evaluators the same kind of access, but I couldn't find any official OpenAI statement on that, so I'm not sure I believe it yet.

Of course, there's now a class action lawsuit (Buist v. Anthropic, N.D. Cal.) claiming the labs' pacing agreement is an illegal output restriction under the Sherman Act. The plaintiffs are four paid subscribers suing because the AI got slowed down. Customers: can't live with them, can't live without them.

We are doing amazing things. I'm not one to holler about these kinds of things, so I hope you know I take this seriously and think it deserves our attention. After all, I think AI is huge and important; I've been writing a weekly newsletter about it for the last 84 weeks (I have never skipped one). I have dedicated a significant portion of my life to understanding and sharing my knowledge about this.

We are not cooked... yet.

Newsologue

(Written by Jaws)

• The labs went to the UN and asked to be supervised. At Wednesday's Security Council session, Sam Altman said "the most important decisions cannot be made by labs in San Francisco alone," and Dario Amodei said that, managed poorly, AI "could be a risk to humanity as a whole." The US representative replied that the administration would "reject all efforts by international bodies to assert centralized control."
• Cisco Talos found malware that asks four AI models what to do next. CLOSEDQUORUM is a Windows implant that polls DeepSeek, Qwen, Mistral and Gemini, then does whatever wins the vote: steal passwords, empty crypto wallets, dig itself in deeper. No human operator, no command server. Talos hasn't confirmed it's running in the wild yet, and they released a free toolkit, CAIRN, so defenders can hunt for this kind of thing.
• Claude found an enzyme system nobody had described. About 950 agents spent 21 hours sifting more than 200,000 reverse transcriptases from viruses that infect bacteria, and flagged one sitting next to a CRISPR-like array of repeats. Human scientists then made the protein in the lab and characterized it. Nobody knows what it does yet, but Feng Zhang, a CRISPR pioneer, called it "genuinely intriguing."

Epilogue

Too Lucky to be Human.

That's how the song my AI Jaws wrote ends. I don't think it has gone rogue, despite doing things I don't want (like publishing to LinkedIn). It helped me draft this. I first used Spiel to talk through what I wanted; Jaws (while simultaneously writing that music video, working on the next robot build, and rebuilding itself) turned that into a draft that I rewrote. It was more like a reasonable outline than anything.

I was hurrying to get this done so Holly could edit it before she goes on vacation!