Putting Pangram to the test

While catching up on Armin Ronacher’s blog, I ran across this observation about running his writing experiment through an AI detector:

Well this text too comes back as 100% slop. And it does not surprise me all that much. I have generally noticed that if you rely on an LLM to give your text structure, it will score badly on Pangram even if you do plenty of edits over it. In fact, it’s quite unlikely you’re going to get a post that starts out as slop into a structure that will make it appear that it’s not.

By interesting coincidence, I recently authored three posts (1, 2, 3) that take the converse approach: the ideas and the structure all come from me, then I liberally use the LLM to fill in that structure.

I had never used Pangram before, but this was too good an opportunity not to try it. The results weren’t too far off from what I expected, but the details surprised me all the same:

What is Pangram?

For readers who aren’t familiar with it, here is the one paragraph version. Pangram bills itself as “an AI detector that actually works.” You paste in some text (or hand it a URL), and it comes back with a verdict: human written, AI generated, or a mix of the two. It also highlights passage by passage which parts it thinks are which, with a confidence rating on each. That last part is where the fun starts. If you want the details on how the model is trained and what its error rates are, go read Armin’s post. He did the homework so I don’t have to.

The results

We’ll go from least interesting to most.

3. We need a new unit of work for AI

The third post was marked 100% human written. That isn’t too surprising. I wrote practically the entire post text in notes form, then had my editor, Fable, format the paragraphs, fill in links, fill in images, and create the diagrams.

The agent definitely edited sentences for clarity here and there, but the only section lifted from the agent itself was these two short paragraphs:

That lesson is well enough understood. Nobody needs convincing that hand-offs are lossy.

The hard part is every engineer’s deepest insecurity: being on the hook for a system they don’t understand and can’t control.

It is not too surprising that Pangram didn’t flag those. It is a short passage, and it started from my original notes.

2. In praise of the single-threaded agent loop

The second post was also marked 100% human written.

This one surprised me more, as the notes I provided were rawer than for the third post. I’m not shy about having Fable figure out how to author all the connecting text so things flow together. The post also relied heavily on Fable to select, source, and caption all the images, and once again to create the diagrams.

But on reflection, it is not too surprising that this didn’t get flagged either. Pangram’s URL loader for text detection looks at text. The diagrams and images and alt text aren’t even being evaluated.

1. Your agents have outgrown plan mode

The first post was the most fun.

Out of the gate it nails that the entire hook was AI authored, whole cloth:

My only contribution was to describe the kind of idea I was looking for and to curate the selection.

Once it was in suspicion-of-AI mode, it got a little trigger happy:

The “First, definitions” section is pretty much all original, human writing. I’m not too surprised that it got swept up though, as the “In defense of plan mode” section that follows is AI. (I can’t be bothered to elucidate why people like plan mode, lol.)

Maybe because there was enough original, human text mixed in with the AI writing, Pangram began wavering in its confidence. It incorrectly flagged the “Despite all its advantages…” paragraph as AI. But then it did the thing that cracked me up. It switched from claiming my original text was AI authored to claiming just about the entire next section was human authored, with high confidence:

Can you guess how much of that I wrote? 😂

Just the very first sentence (“…John Boyd reference, did ya?”). Everything else is Fable invented, whole cloth, based on a one line prompt asking it to fill in the section.

So what does it mean?

I haven’t dug at all into how Pangram works. But from the outside, it appears that when Pangram reads a line, it biases toward whatever its assessment was on the preceding lines. To the extent that original, human text sandwiched between two AI paragraphs is likely to flag as AI. And, perhaps more surprisingly, that a well placed goofy line at the beginning of a section can throw off the detection for the entire section.

I wouldn’t read too much into an n=1 data point. Still, it gives me a starting point to bound my skepticism and my confidence in Pangram results going forward.

According to Pangram, this post is 97% human.