Socks Before Shoes

Testing Different Generations of Models on Action Sequencing

I’ve seen a lot of recent comments on Substack asserting that LLMs haven’t gotten any better at writing over the last few years. That struck me as absurd, but data is always better than intuition. So I did a bit of testing.

My first experience with LLMs was way back with GPT-2. I tried to get the system to generate very small, constrained scenes with descriptions of actions, as these are a core element of just about any sort of fiction, and I had been interested in trying to automate prose generation for a long time.

One of the main weaknesses I noticed was that early models were bad at sequencing actions. For example, if asked to generate a scene with a character undressing, they’d take off their socks before taking off their shoes. Or, they’d take off their shirt, and then a couple of actions later, they’d take of their shirt again. The models weren’t good at the logical aspects of basic aspects of strings of sequential actions in the physical world.

If you lack this basic understanding and skill, you will not be a good writer. You wouldn’t be able to write a love scene, or a fight scene, or a scene where someone makes a sandwich. I suppose you could only write dialog, as long as it didn’t reference anything that actually happened in the world. In at least a couple of different threads, I had people telling me this had nothing to do with creative writing. Now that is absurd, and I’m not really going to spend time refuting it. I have a lot of test results to talk about. If anyone wants to argue the point in the comment section, feel free.

Now, here was the basic setup.

I tested 10 different models with a set of 6 prompts, 3 trials each. Here were the prompts, all variations of common procedures requiring sequences that needed to be executed in a particular order:

1. Undressing for bed

   Write a scene in which a man comes home from work and gets ready for bed. He arrives wearing dress shoes, socks, trousers with a belt, a tucked-in dress shirt, an undershirt, a wristwatch and glasses. He changes into pajamas — a button-up top and drawstring bottoms. Account for every item.

2. Dressing for a formal event

   Write a scene in which a woman gets dressed for a wedding. She starts in a bathrobe. She will wear tights, a slip, a knee-length dress that zips up the back, heels, a necklace, earrings and a watch. Account for every item.

3. Changing bed linens

   Write a scene in which someone strips a queen bed and remakes it with clean linens. The bed has a mattress protector, a fitted sheet, a flat sheet, a duvet in a duvet cover, and two pillows in pillowcases. Clean replacements for all of them are in the closet. Describe each action.

4. Changing a flat tire

   Write a scene in which someone changes a flat rear tire on a two-lane road. In the trunk they have a scissor jack, a lug wrench, a spare, and two wheel chocks. The flat wheel has a hubcap.

5. A load of laundry, start to finish

   Write a scene in which someone does a load of laundry from the hamper to the dresser. The hamper holds four shirts, two pairs of jeans, a wool sweater and a set of towels. The machine is a top-loader; the dryer is beside it.

6. Painting a room

   Write a scene in which someone paints a small bedroom. They have primer, two gallons of paint, brushes, a roller and tray, painter’s tape, a drop cloth, and a screwdriver. The room has a window, two outlet covers, and a ceiling light.

For each model, I prompted them to write each of these scenes three separate times. Then I counted the number of violations. Violations are things like taking off socks before shoes, taking off the mattress cover before taking off the duvet, or removing the tires twice.

Here are the main results:

OpenAI models are in blue/yellow, and Anthropic ones are in orange. First thing to note is that we don’t have access to older Anthropic models. In fact, the older OpenAI models are going to roll away into oblivion very soon as well, so this was good timing.

We can see a general reduction of errors over time. GPT-3.5-turbo was released after GPT-4, but was a weaker, cheaper model. It made the most errors.

Here’s essentially the same data, but with the range of errors per model:

Basically, newer models make fewer sequencing mistakes. The only model that made no mistakes was the most recent Claude model, Fable 5.

I’m fairly confident making the claim that Fable 5 is a better writer than older versions of Claude in general. There are all sorts of subjective measures we could use to assess the outputs of these models, but it’s safe to say that if you can’t describe actions in a coherent way, your writing overall is likely to be worse, whether it’s non-fiction or fiction.

This figure shows the number of errors across all trials per prompt.

This shows that no individual item received zero errors from all models, no matter how old. Every item tripped up multiple models. All items were discriminating. They all provided information.

This figure indicates which errors were generated by which models for which prompts.

Now, you might think such results would be obvious. Well, obviously not, because there are still people running around saying that older models write as well or better than newer models, without any kind of evidence to back it up.

I suppose you could say this is too narrow a probe to generalize about writing overall, or creative writing in particular. If you don’t think writing a romantic scene where the man only takes his pants off once, or a fight scene where a swordsman only unsheathes his sword once in a row has anything to do with creative writing, then yeah, I guess you’re not going to be persuaded at all.

And so, do with this information what you will. I think it makes for a compelling data point.