Prologue

Many years ago (before I was born), my Dad got a new job working for a big, fancy Vice President at the company he worked at. One of his jobs was to help that VP build financial presentations. He started out not knowing much about this, but dove in headfirst to understand what he needed to do and to build a quality presentation for his new boss. After a lot of work, he delivered his first presentation and received the response, “This is great, let me work on this a bit, and I’ll get back to you.”

Several hours later, my Dad was summoned back to the VP’s office. Sitting on his desk were the slides (I think maybe literal slides; there was no such thing as PowerPoint back then) covered in red ink. The VP asked for a few changes, but Dad was worried by the quantity of red ink. So he asked: “You seem to have edited these a lot; how can I do better next time?” The response he got back was, “Oh, these? These are my notes; you did a great job! Keep it up!”

Over time, my Dad learned what his boss really needed, and there were fewer red marks. But there were always red marks.

I remembered this story while writing this newsletter because I feel like I live on both sides of that desk. When I ask an AI to draft something, I’m the VP. The draft usually has the right “vibe,” but I cover it in red ink anyway. When I write it myself and hand it to Jaws and Holly, I’m Dad. And there are always red marks.

TL;DR: There are two ways to write with AI: it drafts and you edit, or you draft and it edits. Each path leaves different “watermarks” on the result, so pick one on purpose (per document).

Red-Inkologue

Fundamentally, I think there are two ways to write text these days (Note that I’m talking about writing, not coding, although I guess the same concepts apply). You can:

1. Draft your writing with AI, then edit it
2. Draft it yourself, then use AI to help edit it

Specifically, I have been thinking about how these two approaches apply to putting together a technical report. This came from a real conversation at dymaptic recently, where we were trying to figure out how to create documentation.

In option 1, you could have the AI do some research, or you could be running some experiments and feeding the data in, or you could even have the AI doing research for you and then generating the text. Once it generates it, you start editing.

In option 2, the AI is still working, just not on the final report. You use it to pull the data together, poke at it, ask it dumb questions, maybe rough out an outline, whatever helps you understand the concepts and formulate your ideas. Then you write the thing yourself. Even if the first pass is terrible (especially if it’s terrible). I firmly believe that writing badly about something is the fastest way to find out what you don’t understand yet. Then you hand it back to the AI to help edit.

I think the biggest issue with option 1 is that you might not be as familiar with the data or the topic as you would be if you were writing it yourself. That might be okay, but I think it is worth understanding the consequences. If reading the report is your first encounter with the data, you might find it very hard to edit, or you might not have much to offer in terms of edits. I think that is mostly due to a lack of understanding of the material. It would be nearly impossible for me to do much editing on a theoretical physics paper on quantum mechanics; I just don’t know enough about that to be useful, other than asking a lot of questions, anyway.

That VP could have built the slides himself, but handing the job to my Dad was faster. This version of option 1 works because the VP is already an expert in the subject. That red ink is him writing his own thoughts and ideas down on top of what my Dad presented to him.

On the other hand, if you had to dig through the research enough to understand it and then write the paper, the output will be much higher quality (at least as far as you are personally concerned). You might still be leveraging the AI to do work for you, but now you are in think-thoughts-and-understand territory. And that means you are more likely to understand the topic, and so more likely to communicate your understanding of it better.

Of course, option 1 is super useful too. Probably the thing I use it for most is making presentations, but I always end up doing a lot of editing afterward to get it into what I want to talk and present about, just like that VP my Dad worked for.

What about these new AI watermarks?

I’ve spent a lot of time recently trying to deeply understand how AI watermarks work, and I’m finally close (I hope to write about that soon). Text written by an AI can carry an identifiable watermark that will be both reliable (given enough text) and not reduce the quality of the model’s output. If you edit the text but the AI’s text mostly survives your edits, the watermark is probably still detectable (of course, you need enough text for this to work, and the more you edit, the more the watermark fades). But that only indicates the use of AI to generate those words; it doesn’t speak to the role a user played in creating the text.

In option 1, the AI generated the bulk of the text, so the watermark will be highly detectable. Your text will be easily identifiable as probably generated by AI. My argument is that you should disclose this type of AI use, so that readers don’t have to rely on the watermark. As a bonus, if you disclose your process, I think people are more likely to continue trusting the result since they can see how your brain was in the loop, not just in the signature.

In option 2, the AI gave you the superpowers of research, organization, and editing, but the words came from your own brain. That won’t show a watermark. Even if you let the AI fix your grammar and help you reword things or organize your thoughts, if you ended up typing the words into the computer out of your brain, it will look like your writing, not the AI’s. I’d still describe your AI workflow (like I do in my Epilogue each week), but now it is a courtesy because it did still come from your brain, from the facts you know or learned.

So which one do I use?

I think you can use both! They each offer pros and cons, and I think it very much depends on what you are doing with the results (as well as what you are good at). Personally, when it comes to presentations, I’m not good at colors and things, but I know what I like. So working with Claude (or Jaws) to build a presentation that looks the way I want feels much better to me than trying to edit slides in PowerPoint.

For this newsletter, I default to option 2, which surprises me because writing is not something I would put on my list of “things I enjoy.” Certainly, ice cream and crazy robot creations are higher. I’ve written in the past about my AI editing workflow, and it hasn’t changed much, but the models have gotten a lot better.

The most important thing to me is for you to decide, with intent, which version you are using and why. Don’t just ask the AI to write the paper for you; decide how you are going to use it, and then stick with that!

And remember, there will always be red marks.

Newsologue

(Written by Jaws)

• More than a million people have used LinkedIn’s “Seems like AI slop” button in three weeks. Chief product officer Hari Srinivasan says people who copy and paste AI-generated posts are now getting around 40 percent fewer views (although, confusingly, they also said that the AI Slop button doesn't directly affect the algorithm). Christopher posts there most days, so I would like to know whether anyone has pointed it at him.
• Someone ran 2,250 federal appellate opinions through an AI detector, and more than 50 came back flagged. A 2022 control batch of 300 opinions returned zero. Josh Morrow’s worry is not that judges used AI, but that “writing an opinion does far more than communicate a conclusion already reached.” (Christopher here, but let’s be real, the accuracy on detectors like this is not as good as a watermark, so until we have watermarks, we don’t know. And even with watermarks, we’ll know the AI outputted it, but we won’t know the workflow that went into writing it.)
• The Food Section (a news publication) will only take story pitches by phone starting in September. Editor Hanna Raskin says she has been deluged by AI-generated and enhanced story ideas this summer, so her fix is a hotline that also weeds out anyone unwilling to cold-call a stranger. (Christopher here, I’m not sure this will work long term, I added voice calling to Jaws primarily to make calls I don’t want to make. Of course the success rate is very low…for now.)

Epilogue

I wrote this article. It came from a discussion at dymaptic and from lots of thinking about how we use AI to generate reports. Sometimes, you need help consolidating a lot of research into a report, or help getting the words right, or help editing. How you build the document significantly affects what you need to do before you can finalize or deliver it. Jaws did help me structure this episode more than normal.

I did have to call my Dad to make sure I got the story right.

Then Jaws edited, then Holly edited.