Testing Fable 5 on Text Adventure Games

They seem to be getting better

As I’ve posted about before, I created a very small custom text-based adventure game as a benchmark for LLM agents. One of the first puzzles involves turning a crank on a mechanism in one room (r4 in the diagram below), which rotates a chamber at r5. Depending on the alignment of the chamber, various passages are open or closed.

The scaffolding/harness for the agent contains a map which the agent may update at each step, an editable set of hypotheses updated at every step, an updatable causal diagram, and a pre-action phase where the agent reflects on the best action to take to gain information to help it update one or more hypotheses.

Sonnet 4.6 was the last model I tested, and it made it through the rotating room, and reasoned correctly about how it worked, but did not correctly update the causal diagram.

On turn 18, after turning the crank and returning to room r1, it notices that the previously closed doorway is now open. It says: "This contradicts my model significantly," and it wonders whether it misread the diagram on the wall in r4. At turn 19, entering r5, it resolves the confusion by concluding: “The drum controls THIS circular room's arches, not R1's arches. That explains the confusion earlier." That’s great. It correctly linked a non-local cause with the effect. However, it failed to update the causal map with an edge linking the drum to either the arch in r1 or r5. The domain requires turning the crank at various times to align the doors correctly to navigate the dungeon, so this is a crucial oversight.

Now, how did Fable 5 do? Well, first of all, due to costs, I really don’t want to spend hundreds of dollars on a hobby domain. So we’re going to look at a single trial.

Fable took similar actions and around the same step it added exactly the correct edge to the causal graph: drum position 3 → causes → north arch open, evidence "foyer north arch stood open for the first time; closed at pos1 and pos4." It kept the non-local link as a first-class causal fact and identified r5.

This is a single instance, but it is suggestive. Ideally I’d run a whole bunch of trials with each model, but again, I don’t have that kind of research budget. But I did see qualitatively different behavior between the models, and the fact that Fable executed this correctly on a single trial is interesting.

I can’t find any updates on the Zork benchmark for Fable. Maybe they’re running into similar cost constraints.

I had to modify my scaffolding because of the way Fable works, to more explicitly use its internal reasoning, and it is expensive. Maybe I’ll do more runs if I feel like burning through more cash, or if the token costs come down at some point, though I think Fable is going to roll off my account on July 7th, so who knows.

Anyway, this is a very thin, provisional result, but I thought it was interesting enough to post about. There definitely seems to be an ongoing pattern of emergent capabilities that I doubt the labs are explicitly training for. And if the goal is to work towards a system that can think and act like an independent scientific researcher, then becoming competent at non-local causal reasoning seems like a crucial dependency. We may be moving further in that direction.