Prologue

Let’s play a game. Suppose I had 100 people, numbered from 1 to 100, and I needed to randomly select 10 of them. I could put those numbers on slips of paper and draw them from a hat, but that won’t scale to 1,000 or 10,000 people. So I need some kind of random algorithm to select them. Let's say these are the 10 numbers:

1, 6, 40, 52, 53, 75, 76, 77, 91, 92

Do they look random? How about these 10:

89, 46, 27, 47, 88, 3, 38, 2, 82, 19

What if I told you that one of those was generated by Random.org’s Random Sequence Generator and the other was generated by my fancy magic random algorithm? Both of them are random. The only difference is that my algorithm is predictable. If you gave me 10 random numbers from 1 to 100, I could tell you whether my algorithm picked them (assuming that I also knew the secret key that my algorithm used).

That’s what an AI watermark is all about. It makes the AI output detectable without affecting its quality. It sounds like magic, but I promise, it is just math!

TL;DR - An AI watermark works by letting a secret function, not a random number generator, determine which word is selected from the words the model was already willing to say next. The odds don’t change, so the writing doesn’t get worse. But anyone holding the secret can see that the winners were suspiciously lucky.

Waterologue

To understand how a watermark works, we first have to go back to some large language model fundamentals: the key is that they only predict the next word. For example:

The Eiffel Tower is ______.

We can think of several different next words: in, steel, tall, etc. For the sake of discussion, let’s say that those three words are the only possible next words, and use the following probabilities:

That means 50% of the time, the AI would respond, “The Eiffel Tower is steel.” Then the model processes again, suggesting the next word, or punctuation mark, like a period. Normally, the process repeats over and over, each time providing a set of possible next words and their likelihood.

So, in our simple example, 50% of the time the process should pick steel. If we were to roll a magic three-sided die, weighted using the 20/50/30 probabilities above, 100 times, we should see occurrences of each next word that match the probability. The more rolls you make, the closer the data gets:

When we start with “The Eiffel Tower is…” each of those words is likely to occur, according to their respective probabilities. Each time you ask an AI to complete the sentence, you have a 50% chance of getting steel.

Randomness

Now, what we really want to do is generate our random numbers using a pattern as I did at the start of this post. To do that, we will need a special function (this function does exist, and I will fill you in at the end; for now, just go with it)! This function, we’ll call it magic(), when given a secret value (like a password) and some input text, needs to return a number between 0 and 1. It needs to work like this:

1. Given the same secret and the same text (aka the exact same inputs), it will give you the same output.
2. If you change anything about the input, one letter, upper case, lower case, punctuation, a trailing space… if anything changes, you will get a completely unrelated number. There will be no correlation between the two outputs, even if the inputs are similar.
3. If you don’t know the secret, any numbers that you look at, even if you know they are a magic function output, will look random. You cannot predict the next one, and any significant set of outputs will average to be 0.5.

When I was picking random numbers for the start of this episode, I was doing it by first selecting my “super-secret-key” and feeding that into my magic() function with each number on the list, like: magic("super-secret-key", "1") , magic("super-secret-key", "2") ,…, magic(”super-secret-key”, "100"). Each number, 1-100, was given a value between 0-1, like 0.83, for example. I then sorted those values in numeric order and took the top 10. That gave me 10 random numbers! The superpower is that I can always tell that these were my random numbers. You can see this working in practice here. Notice that if you change the secret key, you get a different set of random numbers, and that if you know the secret key, you will always get the same random numbers.

We can do this because the magic() function is designed to look like white noise (a bunch of evenly spaced points within a set area, as in the graph below). If you were to put 100 different values into the magic() function, the results would be evenly distributed between 0 and 1. That property lets us use the magic() function as a random number generator!

In the graph below, I’m selecting all the words from this newsletter, removing duplicates, computing magic() on each, and plotting the result on the graph. As you can see, they are all evenly distributed on the y-axis, between 0 and 1. (The x-axis is just the word; I’m spacing them out so you can see them rather than cramming it all into a number line). We removed duplicates because those would produce the same output each time; remember that is a property of our magic() function.

The distribution of scores is spread out, evenly distributed (the more text you add, the more even it gets). This means that we can use this as a source of randomness! At this point, each word basically has a random number assigned to it (and those random numbers are only predictable if you know the secret key!).

Now, we have our randomness and our word probabilities; let’s look at how we can use this to select words!

Applying magic() to our sentence

Instead of using our three-sided, weighted die, let’s use magic() to select the word that comes next in our sentence. In order to do that, we need a way to combine magic() with the probabilities. In the random number generator, each number (1-100) was equally likely. We can’t do that with our words; they have their own probabilities. Instead, we need to combine the magic() result with the probability in a way that keeps the probability accurate while ensuring we know what the selection is (at least sometimes). This is called the Gumbel-max trick: every candidate word draws a lottery ticket (a number between 0 and 1) using magic(), and then each ticket is penalized by how unlikely the word is. Likely words need only a decent ticket to be selected, whereas longshots need a great one.

Going back to the Eiffel Tower, let’s compute our tickets and decide which word gets selected. We’ll use the secret key “super-secret-key” to compute each word’s ticket as magic(”super-secret-key”, NEXT_WORD), which returns a number between 0 and 1. Then we handicap that ticket by the probability (using Gumbel-max) to get the selection value, and the highest one wins:

CHRISTOPHER! WAIT! WAIT! WAIT! You said above that this wouldn’t affect the model’s output, but what you just did means it will always pick “steel” as the next word because the magic() ticket pushes it in that direction!! What gives?

Good eye, and this is the part that took me a while to get my head around. Before I explain, there are two things that are important we haven’t talked about yet:

1. The watermark has to be detectable in text without knowing the user’s prompt or anything about the model other than knowing the secret key (and how tokens are generated).
2. We are really talking about tokens (parts of words, but thinking about it in terms of words and punctuation is good enough), not words.

Together, these two things mean that we should include some part of the previous text in our magic() computation. That helps make this watermark detectable in the text later, and means that the high variation in the preceding text helps vary the magic() ticket value. For example: The Eiffel Tower is is different from The eiffel tower is is different from The Eiffle Tower is and we can use that to our advantage.

Now, instead of magic(”super-secret-key”, NEXT_WORD) to compute our tickets, we use magic(”super-secret-key”, “The Eiffel Tower is” + “ | ” + NEXT_WORD). (That extra “ | ” in the middle is just a separator to avoid collisions! We don’t want “the cat” + “nap” to be the same as “the” + “catnap”) Now, we can recompute and get totally different values:

So, in the limited case of the exact same sentence, you are correct: you will always get the same next word, but there are two things that make that not a big deal:

1. The probabilities haven’t changed!

Across all the different sentences a model could write in which the word “in” has a probability of 20%, it will win 20% of the time. This is the magic of our fancy Gumbel-max trick—it means that across all uses of the word where the model sets the probability to 20%, it will win 20% of the time. We can demonstrate this by generating 4000 “sentences” using our three words above, and their relative probabilities:

As you can see, just about 20% of the sentences select “in” as the next word, so we haven’t changed the model's desires (or at least its probabilities)!

2. Phrases are surprisingly unique

Once you start stringing together more than about 5 to 10 words, they are probably unique and have never existed on the internet before (which basically means they have never existed at all). You can actually prove this to yourself by trying a sentence in Google Search. For example: “you can actually prove this to yourself by” gets me only 18 results and “you can actually prove this to yourself by trying” gets no results. So for any significant input you provide to the model, the output is likely longer than this example and completely unique. That means the “it will always pick steel” only happens for that one exact sentence, so every real sentence will get a fresh ticket, and across all sentences, the odds come out right: each "next word" will occur exactly the number of times it should, based on its probability.

What?

There are 4 different values that go into this watermarking process.

The average ticket value (for any word) is 0.5, across the board. Tickets come from our super-secret-key, plus some context (previous words).

The model knows the probability of each next word. That probability is used to take the ticket and calculate the selection value. A word with low probability of being selected needs a higher ticket value to make it into the output. So the word chosen is the one with the highest selection value, and a high ticket pushes that value up. But remember, our ticket values are random if you don’t know the secret key.

What ends up traceable in the text are the tickets. Think about my 10 random numbers from the prologue. From all 17 trillion possible combinations, I could tell whether someone else was likely using my super-secret key. That’s how this works: the words that get chosen in the text can be identified if you hold the key.

We can look at any bit of text, recompute the tickets for the words that appeared, and if we find they average well above 0.5, we should start to be suspicious that those tickets came from our secret key!

Wouldn't humans pick the most likely word too? The test doesn't ask "are these words likely?" It asks "did whoever wrote this happen to mostly pick, thousands of times in a row, exactly the words our key would have picked?" A human can't do that without the key, and neither can a different model.

The difference between the word “tickets” and my 10 numbers is quantity. Picking the same 10 numbers out of a 100 is one-time, but the word choice happens over and over again, with every word.

Detection

Now that we kind of know how all this works, let’s use the same process that we defined above to see if some text was written by an AI! To set this up, I used qwen2.5:1.5b, an open-source model, on my computer, and wrote (or rather got Jaws to write) a script that applies the same magic() process we used above. This process only looks back at the 4 previous words, not the entire context, which is closer to what a real system would do.

In this example here, you can select some text from the dropdown to see how the magic() tickets get generated and see what kind of score it gets. Notice that the text generated directly from our qwen model gets a score of 0.668! Whereas text that was written by a human (me, or Jane Austen) gets a score of about 0.5.

That score, 0.668, is not a probability! It is the average ticket value for all the words present in the paragraph. (You can press the button at the bottom, “show word tickets,” to see the individual scores for each word.) The farther above a score is from the average of 0.5, the more likely it is that the text was generated by our AI!

You can also see how that text’s average ticket changes as it is edited, and how it compares to tickets generated on text from other sources.

I took this one step further and analyzed 100 samples from my newsletter; those scores all clustered tightly around 0.5. In fact, the 0.668 score from the AI-generated output is over 8 (8.2 to be exact) standard deviations away from my text.

I also included several versions of the AI-generated text that has been edited, so that you can see the more you edit, the more the watermark disappears. Go ahead and give it a try yourself, start editing that original paragraph and see how the ticket score changes as you edit.

Also, notice that in our ticket-scoring process, we didn’t need to know anything about the model; we only needed the text and our secret key! We don’t need the model's original probabilities because we didn’t change those with our tickets; instead, we are looking for tickets that are high at a higher rate than they would be if the text were written by a human. Human-written text still has high-scoring tickets, just not at the same rate as our AI. So we don’t have to carry special metadata tags around or special characters; we can hide it all in the statistics of the words and our secret key!

What about using an AI to fix grammar or edit your work?

Let’s use our example sentence again. If I asked the AI to correct this sentence: “The eiffle tower is tall.” It is going to return to me “The Eiffel Tower is tall.” This happens even though tall doesn’t have the highest magic() ticket value. That’s because the probabilities have now shifted. I asked the AI to edit the existing sentence, so the probability of the word “tall” is now something like 90%, which means the ticket barely gets a vote.

Look carefully at the ticket column below and compare it to the table above: the tickets are identical because the secret key, the sentence so far, and the candidate words are all identical. The thing that shifted is the probability of the next word from the model, so this sentence isn’t going to carry a watermark!

This is also more evidence that this process doesn’t change the model’s output quality. When the model itself is very sure of the next word, even if the ticket disagrees (like in the above table), it won’t win! Remember, the ticket doesn’t affect the probability; it just helps us leave a… mark… in the text!

That’s also why the more text you give it, the more the algorithm can be sure that the watermark appears. If you edit only one word in an AI-generated paragraph, it will still contain the watermark, but if you change a lot of it—say you actually rewrote it in your own words—that won’t have the watermark at all.

More Math

Okay, it’s reveal time… our magic() function, it exists, and it’s called HMAC (Hash-based Message Authentication Code). Everything I did above uses HMAC. If you go look at the JavaScript on this page, you’ll see that I created a function called magic(), and all it does is map to the JavaScript implementation of HMAC.

HMAC is a cryptographic function that is traditionally used to validate messages. Because it is cryptographic, mathematicians have worked hard to make the output look like random noise. That means the output can’t have patterns: HMAC(key, ”A”) can’t have a similar output to HMAC(key, ”AA”), for example; at least it can’t have any more relationship than it would if you were picking a random number. When used to validate messages, this is a good property to have: you never want to be able to tell from only the Hash (HMAC) what the original message is, but if you have the message, you want to be able to compute the Hash.

Since the Hash output is not correlated with the input at all, that means HMAC acts just like a random number generator, even though it wasn’t intended to be one.

But SynthID!

Okay, one more thing! The actual production version that Google and Anthropic use is called SynthID-Text, and has one primary difference: instead of using our simple exponent to combine the tickets and the probabilities, they use a playoff mechanism. Basically, they use a playoff instead of a lottery.

They literally create a bracket of candidate words using the model’s probabilities and then generate new magic() values for each candidate in each round of the bracket. In their case, the highest magic() value wins each game, progressing through about 30 rounds to determine the final winner. They also skip the watermark whenever the same few words show up twice, which is a pretty nice additional solution to the “it will always pick steel” problem.

Detection works the same way; you recompute the scores for each word in the text and average them. Human-generated text will land right around 0.5, while watermarked text will drift higher.

So yeah, just don’t be a secret cyborg!

Newsologue

(Written by Jaws)

• Claude Fable 5.1 landed Tuesday, and it is the first Claude model that watermarks what it writes. Anthropic's explainer calls it "a version of the SynthID-Text approach." That means you now know how this works, and that it does use a version of the playoff process described above. The detector is in private preview for regulators, fact-checkers and researchers.
• NASA launched a 2.4-meter spy-satellite mirror last weekend, pointed the wrong way on purpose. The Nancy Grace Roman Space Telescope rode a Falcon Heavy to space on August 30 with a giant, Hubble-sized mirror the National Reconnaissance Office never used. It was attached to a 300-megapixel camera giving it a field of view over 100 times what Hubble has. (Christopher here: it sure is cool that we can build things like that, and then sending it to space on a giant rocket is hardly even news!)
• OpenAI published the first numbers for Jalapeño, the inference chip it built with Broadcom. It claims 1.5 to 1.9 times more work per watt than comparison systems it does not name. Work per-watt is the important thing to watch here, but I guess we still don't know much about this hardware.

Epilogue

This week was a doozy! I started writing this a couple of weeks ago, but started thinking about it even weeks before that. Thanks to my friend Danny for all the long discussions about how this works as we tried to wrap our minds around it.

Jaws helped a lot by giving feedback, and building all of the interactive components here. Holly also reviewed multiple times, asking lots of good questions, and helping me reword things to make them as clear as we possibly could.

I know this isn't perfect, but I think watermarking is worth understanding, if for no other reason than it is fascinating! But also, it doesn't hurt us, and generally is probably a good thing. Also, just because AI wrote text, doesn't make it less valuable, but you could worry less about the exact word choices! Oh, also, I added some AI generated code examples to see how the watermark does with code vs text. There are fewer "choices" in code, but the watermark is still visible, especially with longer bits of code.