⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins

Hey, everyone. Welcome back to a Latent Space Lightning Pod. This is Alessio, partner and CTO at Decibel, and Swix is running late, so we'll just get started, and maybe he'll join. Today, I'm joined by Jack and Mart, who are the two researchers behind the Factorio learning environment.

Since we had multiple guests on the podcast before bring up Factorio as a-- both as a hobby and a way to think about building startups, we thought we would do this episode today to figure out which models are best at Factorio and some learnings in every model learn through games.

We talked about the Voyager paper with Minecraft in the past, so welcome to the show.

Hello.

Hi there.

Excited to be here.

Were you guys Factorio players before, or did you just pick Factorio post-fact as the best environment for this?

No, I played Factorio for years, and about eighteen months ago, I was looking into, uh, this idea of the paperclip maximizer, you know, that kind of cautionary tale where if you set a model to maximize paperclips and ma-optimize a factory, what kind of bad stuff do they end up doing in furtherance of that goal?

Um, and it got-- kind of got me thinking about the applicability of Factorio as an environment for benchmarking models. One of the kind of motivations for it is that when you compare Factorio against Minecraft, for instance, the complexity scale is drastically different.

In order to complete Factorio, you know, complete and launch a rocket, you need to mine maybe about a million raw resources to get to that point.

Mm-hmm.

Whereas the same kind of point in Minecraft, you have to just get maybe two hundred resources. So, you know, we have orders of magnitude difference in, in Factorio comp-compared to some of these other environments. And if you wanna go beyond just launching a factory and, you know, see some of the huge factories that real players have managed to make, I think the biggest one at the moment is generating something like nineteen million resources a second.

So we have factories that, you know, stem from, you know, making, uh, maybe one resource every couple of seconds all the way to eighteen, twenty million a second. And that range gives an enormous amount of signal for comparing models, uh, you know, from tiny models that aren't very good all the way to superhuman models and, and enable us to differentiate them kind of at any step along the way.

And that, and that I guess, was the core idea. And, you know, I, I kind of set out on a Friday night about eighteen months ago, uh, you know, as a, as a casual w-weekend project, you know, in and out operation, get it done, and eighteen months later, we kind of got it into a position to actually release.

Mark kind of joined, uh, working on it about nine months ago or so, uh, or so, and we really started focusing on this about three months ago and, and just trying to clean it up and get it, get it into the public, basically.

I wa- I was on the complete opposite side. I had never played Factorio before, uh, before joining, but I was kind of sold by Jack because of all the reasons that he just brought out. And then I remember I had to do the full tutorial before even starting on actually building this thing.

Um, and already during the tutorial, I could kind of see the complexity and the, the opportunities for this, and it was then really cool to kind of see as we build this thing where, where these models get to and, and where do they, um, where do they kind of fall short.

Yep. Um, it would be great if you start sharing your screen, we can kind of run through your FLE blog post paper/um, however you-- whatever the right nomenclature is nowadays.

Sure.

So yeah, at a high level, I guess, yeah, for people that have never played Factorio that are watching this on YouTube, you can kind of see what the environment looks like. It's kind of like a cyberpunk industrial game.

I don't know how else you guys would, would describe it. Le-let's start from the game harness. Was it easy to plug a model into the, the game? Like, does it already have like a API? I know Minecraft has very good support for kind of like shell commands and all that.

So it does to an extent. Exactly the same as Minecraft, Factorio exposes a low-level Lua API to interact with the game. The, the way you mod Factorio traditionally is you, you know, you write some Lua scripts, you hook that into the game, and then when events happen, your script runs.

But this doesn't really work for doing what we were interested in, which was kind of running this at scale and getting, you know, a cluster of servers running. So we had to develop a, an entirely different, uh, way to mod the game by exploiting the admin console of multiplayer Factorio servers.

So, you know, you wanna run a Factorio server with your friends, one of you is the admin, and as an admin, you have a little console where you can ban players, you can, you can change the permissions of different players a-a-and just basically modify the state.

And what we realized is we could hook into this admin console over the network using, uh, a protocol called RCon, uh, you know, carried by TCP. And this basically enables us to execute actions in the game remotely, um, and, and, and kind of shard them across dozens or hundreds of Factorio servers, which is necessary if you want to do training or any of the other, um, uh, workloads you might need at scale.

So going back to your question, you know, what's the harness like? It was very easy to get a, a low-level harness working. You know, tell the character, "Move one step to the left, move one step to the right," that kind of thing.

But that type of approach is probably a lot, uh, much better suited to training models from scratch. You know, I guess a Markov decision process architecture where, you know, you give it a bunch of these low-level actions and you train a network just to, you know, maximize the order in which these low-level actions are executed in order to get an outcome.

But such an approach would prevent you using any of the fine-- uh, of the pre-training that large language models have got. So that forced us to think a bit more carefully about what interface we could expose to these models, and we ultimately settled on, uh, a code synthesis approach, where these models would write Python code, and those Python code-- in that Python code would invoke certain high-level actions, such as move to this distant point or place an entity next to another entity, and you'd leave it up to the model to think about how to abstract these tooling, uh, abstract these actions into a structure that would give it An ability to manage factories of arbitrary size.

You want the model to be able to come up with classes, with functions, with data structures, such that when it has a factory that might be producing millions of resources a second, it might be sprawled over, over a huge amount of the game space, it needs to be able to reason over these different parts of the factory efficiently.

So we decided Python, 'cause the models, uh, pre-trained models are pretty good at Python, and that was, you know, a reasonable choice. Um, so yeah, we, we kind of set up th-this harness. The, the models write Python. They execute the Python in the game, which results in factory segments like you can see on screen kind of coming into being.

And then the models get given a score, which is basically the production statistics, you know, what it's producing a second. Uh, and we also keep track of milestones, such as technologies the re-- the model was able to come up with

or new en-- uh, new items that they were able to create for the first time. So we have these kind of two reward signals, uh, that complement each other to pr-- um, and the reason why this is necessary is to avoid certain, I guess, behavioral collapses where a model might choose, for example, to mine coal and mine a billion or a trillion coal, and that's all it does.

And that would give a good score. Like, that, that would, uh, that would have a high production score, but it would probably indicate something has gone very wrong in the evaluation or training process. So we have, we have a couple of different ways of measuring model performance.

Yeah. And then you have-- Can you talk about the lab-play versus open-play structure of the two tests here, right?

Yeah, I think one of the biggest questions for us was to figure out what do we actually want to evaluate in this, in this environment to... Because lab-play i-is simpler. Lab-play is you give it, you give the model a task to create a factory of some entity, and then the model just needs to create the factory using our API.

That gives you an easy way how to evaluate the upper bound on the, on the, on the performance because we have twenty-four target entities and each, each new entity is more difficult than the previous one. And then we quickly saw essentially the upper bound of, like, how large factories can these models create because you need to, you know, for every, uh, input requirement, you need to create another subsection and then connect that section to all that.

So lab-play is simple in that sense, and it very specifically measures the spatial reasoning of the models, can they use the API, and how well can they work in constrained environments. Now, open-play is the unbounded create the biggest factory that you can think of type of, type of environment.

Lab-play is, is in that sense a subset of open-play, where open-play, in open-play now the models have to actually create their own objectives as well. Because if you just give it, if you just give it the, the goal of create the biggest factory, it needs to make that large goal into sub-goals.

You know, first let's create drills, then let's create inserters, then let's put them down. Then let's create an, a mine of this resource, that resource, all that. And, um, so you have these two, let's say, abilities at play there of, one, is the model actually good at using the, the API and working in, in Factorio?

And two, what objectives does the model set for itself? So for instance, one of the interesting things we saw was that DeepSeek was quite decent at lab-play. It was kinda on the level of GPT. But in open-play, DeepSeek was terrible because what it often just decided to do was, "Let's just create more chests."

Yes. Let's, let's create two hundred and fifty chests and put them in a nice little line-

Exactly

... and that is my factory.

Exactly. So you can really see the myopic and short-term objective setting at play in the open-play because a lot of the times the weaker models, they didn't think four or five steps forward. They just created-- like, fell down to just creating, crafting entities, making, putting down one drill versus actually creating an automatic setup.

Mm. Like integrating all of the components together.

Yeah, exactly.

I'm curious, what is the real world analogy to lab-play and open-play? Um, you know, it, it w-- I, I think it's very significant that Claude is so much better than DeepSeek in open-play, and is it just, are they just missing a post-training step?

And what is the task that, that this corresponds to?

Um, so I, I think that the, the thing that--

The task is long-term planning. Ba-basically being able to reason over things, figure out an appropriate objective, and maintain that objective for an extended context, uh, without kind of losing the plot and diverting into different areas. Whereas the lab-play is much more about spatial reasoning and the kind of low-level figuring out how to compose stuff that, that that has the appropriate space and precision around it such that it works.

As you might know from Factorio, much like writing a program, if you put one thing in the wrong place, the whole system grinds to a halt. So it's-- that's slight- a slightly different skill set that we're evaluating, uh, than the long-term planning stuff.

It's not obvious to us why Claude is so, so much better than the other models, uh, in both settings. I mean, our, our suspicion is it is better trained for, for this kind of long-term

planning, uh, aspect. Um, one thing we have found in preliminary results is that the reasoning models don't seem to do as well as you'd expect in this setting. And I think that's probably because the way we set this up, it's a bit like we're already making it do reasoning traces over an extended period of time.

Asking the model to do reasoning over reasoning kind of eliminates a lot of the performance improvements you get from applying the reasoning models alone to problems like Math Frontier, where there's a lot of benefit from just writing stuff down and figuring out what works.

Sorry-

Oh.

Go ahead, Shun.

Sorry if I missed this, but in the original publication, there was-- there were no reasoning models in there. Is this just results-

Oh-

... you haven't published?

Th-this is unpublished. This is our preliminary results. But we're gonna, uh, our plan is to, um, uh, publish additional results with the reasoning models, and we've also added in a, uh, an adapter to support VLMs. Now, a lot of the models we did evaluate, uh, do have vision.

In our, in our, uh, published results, the models are interacting symbolically with the environment. It doesn't actually see the pixels on the screen. So one thing we wanna bring out in our next work is how much, how much improvement can you get from allowing the models to take screenshots of part of the factories they want to debug.

A quick, quick response to that and then, and then, uh, and then Alessio. Typically, the results here, not having seen your results at all, typically everyone's very excited about adding vision, and then they add vision, and they find that it doesn't add anything.

Yes. Uh, I mean, that, that, that was our point as well. We w-- No, no, truly, truly. 'Cause we, we had this great idea about adding vision right at the beginning, and then you, you see those experiments where they have two lines crossing each other in an X, and you ask the model, you know, "W-what is this showing?"

And, and it can't even say the, the, the basic geometry of the picture. They can't-- It can't intuit the two lines are-- have an intersection. So to kind of get around this, we've scrapped in, in the vision component, we've scrapped using the tile maps and the graphics of Factorio and have instead fallen back to a very simple geometric renderer which the models can kind of choose to configure when they invoke it as a tool.

And the hope there is that we can stumble acron-- stumble upon the right vision mode that these models are reasonably good at. But by-- I think your intuition is right. I would be surprised if we get that much from it at this stage, but it would be, it would be cool if we do.

We did, we did try it out. We did try out sending in the screenshots at some point, and the, the problem is, because you have so much going on in larger factories, that the models start, start to hallucinate things.

They don't correctly understand what is broken, why a factory doesn't work. It gets very detailed very fast and in our testings when we, when we made this decision, as you said, adding this niches didn't improve things, and it sometimes made it even worse because they started hallucinating entities that weren't there and started doing often-- went on tangents that weren't actually real in the game state.

Right now, at least, the, the vision actually wouldn't improve. Now, of course, as we go on, the models get better and better. It might improve in the future, but that's at least the current state what we found.

Yeah. Um, and if you guys can scroll down to the environment part, there's a couple of things that compared to the Voyager paper were interesting. So in Voyager, there was this idea of like self-discovery of actions, where the model could like then save skills in a skill library and then reuse them later.

It s-seems like here you have memory, but it's unclear if the model can save actions or if they can only use the tools that you specified. What was the design decision there?

Sure. So, uh, apologies for not being, um, clear in the, in the publication. But the models can define classes, they can define functions and then execute them later. Only, uh, Google models tend to do this, um, which is quite interesting.

Another finding we, we discovered was that each of the models we evaluated have f-- had fundamentally different coding styles. Claude, for instance, the Sonnet 3.5 was very much fire and forget. It would, it would write code in a, in a kind of Pythonic way, just like let it fail, d-don't be careful about it.

Whereas GPT-4 would use defensive programming, use self-assertions, so basically asserting facts about its internal state.

And what else was it doing? Is that fair?

Yeah. Yeah, I think G-GPT-4 had that. And the same, it's the same with Mini as well, if I remember correctly. They had a much more, more defensive check before we-

Check before we do anything

... check before we do anything. But funnily enough, often those checks were also incorrectly set up. So it tripped itself up with its kind of overly defensive nature sometimes.

Yes. So, you know, we have, we have all these capabilities in the environment. Um, agents can define skills. They can incorporate the skills. This gives agents a lot of reach as the factories get bigger. You could imagine they could create, you know, quite sophisticate-- Like for, for example, they could create hypothetically an NSAT solver in the kind of, uh, namespace and then use that for, uh, correct item and entity positioning in the game if they were able to do this.

As I say, models are, are fairly rudimentary at this at the moment, but part of the motivation for this environment was to enable us to evaluate superhuman models and the, the evolving state-of-the-art and not become irrelevant in six months as happens to a lot of existing benchmarks when they kind of max out.

It is, it is also a bit of-- a bit different compared to Minecraft because in, in Factorio, a skill is very-- because it's very dependent on the environment. Where do you have your resources? What, what is the current state of the factory?

It's a lot more difficult to create generalizable skills

In-- That, that would be useful actually in multiple, multiple factory settings apart from the easy, you know, connect X to Y, which we, which we exposed as an API already. So we did also give it a go at the early stages, but we found that often the skills weren't that reusable that you would hope at least because of exactly the reason that as soon as your environments change, if you make a skill over creating a factory, then that skill must be really general and like cover all the edge cases to actually correctly be usable later on.

Maybe it's a little bit more like, you know, Minecraft, the rules don't change that much or the, the environment doesn't change that much and, and with Factorio, you are literally changing environment every time you build.

Yeah. Yes. I mean, what you kind of hope to see, I guess... Who said, uh, there was, there was a, a s- kind of a principle, not the Peter's principle, but it basically says that in an organization, the technology you develop tends to mimic the communication structures that you find in the organization.

And I was wondering-- I was kind of hoping to see a similar pattern emerging here, where the, the abstractions and the code structures that the model invented would align quite closely with the t-topology of the factory it had created.

Let's say it had one section which was just about mining and then one section which was just about manufacturing. I was kind of hoping to see that it was able to bring that out and abstract that away to enable it to generalize better in future, but we haven't yet seen this.

I mean, this is something we hope to see when we start training, but, you know, that's an open question.

Did you ever try adding web access, like search?

No, we haven't. I mean, we, we kind of considered this. I mean, f- there's a couple of sources of data, I think inspired by MineDojo? Maybe MineDojo, where we, we pulled in a lot of data about Factorio on one hand, and then we pulled in a lot of factory blueprints that people created on the other hand and tried to use, uh-

Right. Yeah

... RAG, basically. Sorry?

Yeah. Yeah, yeah. That's right. That's right. There's a, there's a, there's a huge community of-

Yes. Yeah

... these pre-made things that, that we know the properties of. Yeah.

Yes. So, so I mean, the-- we asked-- I guess w- we haven't really... We still wanna try this, but it became a bit of an adjunct to the core work, um, doing this extra stuff. Um, one of the issues we found with blueprints is that first of all, they're specified declaratively, much like you get a Terraform, you know, or, or a cloud formation, infrastructure as code kind of blueprint, which said where everything had to be, but it didn't provide the imperative steps necessary to get there.

So on one hand, you know, we have our Python code that the agent writes, which are the steps one after the other to get to the factory, and on the other hand, we have blueprints which just say the out- the final state.

Um, so we, we spent a fair amount of time trying to ma-- uh, trying to bridge this gap. And, uh, we came up with some, I guess, procedural factory generation techniques, which could take a, a declarative blueprint about what this factory needs to look like and then

deduce the steps necessary to get there. Um, the downside of this is that it's very low level. You know, you're not getting the abstractions that you'd like to see. But on the plus side, we have a lot of that data.

We have tens of thousands of these factories. So I'm thinking that it-- this is the kind of thing that would be really useful just to train the model to do from scratch and just give it, you know, ten thousand examples of factory units that people have made, learn on it.

But when you try and move that into a benchmark setting with already pre-trained models just using in-context learning, it's just not that helpful. A, a thousand lines of Python telling you how to make this kind of factory unit, which it may not be directly applicable to the agent's immediate objective, it didn't really give us any uptick in scores.

So I think we kind of put that in the bucket of what we're gonna do when we, when we start really throwing compute at it. So f-for example, we-- when we did all these experiments, we kind of accumulated about a million programs that agents have written with scores attached.

So we kind of got this big bucket of programs and, and traces, trajectories through the game, and we wanna take these and just kind of throw it at a model and just see what happens if we really try and get it to learn how to maximize score directly, like a paperclip maximizer.

Is there any mechanical advantage to being a AI in Factorio? I remember when DeepMind was building AlphaStar, the StarCraft thing. One of the big advantages that it had is that you can, it can like, you know, so precisely click on units and manage them versus a human kinda has some margin of error.

If you were to put like, you know, the top Factorio players versus your FLE, is there like a big skill gap still, or do you feel like at some point it's more about just the long-term planning is really what, what comes into it?

Uh, so yeah, I, I-- there is a big skill gap. Um, as, as mentioned, because they execute Python programs, they can kind of come up with quite complicated policies which a human player would have to kind of develop on the fly.

So in that sense, the agent has an advantage. But when we kind of track the number of ticks it took to execute these policies against the number of ticks a human player would spend to get somewhere, the gulf between models playing this and humans playing it is absolutely enormous.

Um, so for example, the, the current record for speedrunning is about an hou- an hour and twenty minutes to get to a rocket launch. Whereas when you measure the ticks, it took models something like eight hours or so to get to the point where they have a kind of working factory that could make a few things, a few, um, let's say, uh, iron gear wheels or electric circuits or maybe some science and maybe start progressing through the tree.

So, you know, there's, there's probably a hundred times difference in the competency of these models untrained than a, a human expert. But I mean, a-again, maybe a better experiment would be to sh- to take someone who'd never played the game before-

Yes

... put them in the game, and then compare it with the models. Because the models, you know, they have never been trained on this stuff. They only know a little bit about Factorio. They, they have the manual of how to interact with it, and they just, they just have to go and, and try their best.

So maybe that's what we should do. I should try and, uh, recruit my fiancée, uh, and, and kind of make her sit down for, for 10 hours in front of the game and see where we get.

That was me nine months ago with the tutorial.

That was-- Yeah, fair enough.

Yeah. First time I, first time I... Yeah, the, to, like, jump... I, I don't think there is a mechanical advantage per se. I think, I think the largest improvement, yeah, as, as Jack said, is, is purely what type of a structure are they able to come up with.

And if, if you do start to train against that objective, then probably we will-- we might see some wacky structures which maybe humans haven't thought of in the future. But that's, that's the, the, the, the gap towards there and the road towards there is, is long.

Awesome, guys. Well, th-this was great, and there's a lot of Factorio fans. As we wrap up, a couple things. Are there any games that you wish people would build similar experiences on just to see how the models perform?

Maybe it can be, you know, different, different types of games that are not more builder, uh, driven.

Um, I mean, personally, I would really like to see some, uh, agents made for playing Age of Empires II, uh, because it's such a well-designed game and the, the community's kind of come back over recent years. But the AI of the models in the game is, is really terrible, and I think it'd be really fun to see how good we can, we can train these models, uh, to play that game.

I was thinking about the exact same thing.

Age of Empires II?

Yeah, yeah.

Oh.

I was also thinking Age of Empires along, um, which I think is more like a StarCraft-y type of theory rather than, rather than Factorio, but that, that would be-

Before we, before we wrap up, I just want to quickly scroll down just to kind of bring out some of the results very briefly, just to kind of emphasize that we found that this is on a log-log graph.

So we see that Claude right here has almost got twice the score of the nearest best model. Um, and then you have, uh, your Geminis and your GPT-4s, and all the way down here you have GPT-4o mini, which was so bad, upon occasion, it actually begged to be turned off.

It, it gave up-

What?

... and said, "I can't do this anymore. Please reset me." And then it would, and then it would refuse to do anything else for like 100 steps.

It's really smart.

So I think maybe the kind of ambition or determination is, is some kind of latent factor in the competency of these models as expressed in the game.

That's like when Doctor Strange just sees all the futures and is like, "I just gotta"-

"I just need"-

"One percent chance, I'm out. I'm out, guys."

Yeah, he-

I think getting models to acknowledge that they cannot do something, actu-actually I think it's an interesting form of refusal or-

Right

... um, or knowing the limits of their competence, because I think often we're incentivizing these models to say anything.

Y-Yes.

And so to me, that's almost a positive result, and it's just saying, "We're not going anywhere." Like, give a positive reward for that.

Thanks for trying.

Yeah. Uh, I mean, uh, just to give you the floor, please highlight, uh, you know, anything else that you got a lot of responses on this, uh, when you, when you, when you launched it. Uh, any other popular debates or misconceptions that you want to address?

Misconceptions.

I don't know if we have any that we didn't directly already touch upon.

Well, I, I can say something then. So, um, our plan... Uh, th-this was basically phase-- This, this work was phase one of a phase two plan. Our goal now is to start training these models directly on unbounded objectives.

We think that there is a pretty strong application to this in terms of, uh, to, to the field of model alignment, specifically because in this instrumental convergence hypothesis, the paperclip maximizer hypothesis, where we think that certain behaviors are always going to emerge, regardless of whether it's a human doing it or whether it's a machine, one really stands out, which is goal content integrity, specifically the reluctance for people or maybe models to change their goal once that goal has been set.

So we really want to train these models, basically build a paperclip maximizer and see whether this behavior emerges. Because if it does, that I think might force us to change our attitude towards alignment, which is heavily based on this kind of incremental approach.

But if we get to a point where the models are very clever and they don't want to have their objectives changed, it kind of means we only have one really good shot to set their objectives ahead of time, because otherwise it's gonna become quite hard to do it down the road.

Uh, so this is a kind of an open project. Appreciate support from anyone who, uh, is interested in this work or has had experience in either alignment or in, uh, training these models. My email address is in, in the link next to my name.

So, you know, we're currently based on the GPU mode Discord in a kind of private channel. But, but send me an email, we'll get you in-involved, and if you want to help out, we've got, you know, literally dozens of, dozens of things that we think would be really exciting, uh, uh, directions to take this work.

Cool. Yeah. Thank you guys for joining.

Thank you very much for having us.

Thank you for having us.

Have a nice day.

Yeah.