Can a MUD evaluate LLMs? A $99 proof of concept
I'm the author of a paper my friends and I wrote after we were curious if a MUD, text games originating in the 1970s, could be used to evaluate LLMs. We've spent the last several months on nights and weekends running this experiment and writing the paper on just our personal computers with about $99 in API credits.Our experiment did have an interesting leaderboard but even more surprising was the measurements of each LLM. We scored each on four behavioral dimensions, two of which lean heavily on an LLM classifier. When we removed those two, one of the frontier models fell six positions. When we then checked the classifier against a second judge, the per-model agreement between them ranged from 85% to 22%. The aggregate kappa (0.04 on probe detection) indicated the instrument was noisy without saying which models the noise was hitting. The most affected model shared a model family with the classifier. This isn't proof of bias, just one observation we recorded.We realize LLM judges can be unreliable, and while it wasn't our original intent to test this, it ended up being the most interesting finding. The divergence between the two judges is the finding we think generalizes to other judge-based benchmarks.We emphasize this is just a proof of concept and not a validated benchmark. We prepared a thorough limitations section in the paper, including just 50 runs per model, overlapping CIs among the top models, no human raters, a tiny environment, etc.Everything we did is publicly available, the paper and data are CC BY 4.0, while the code is MIT. The paper, transcripts, code, and complete API billing export can be found at https://doi.org/10.5281/zenodo.21386663If you find issues, please let us know, that's why we're sharing this. We're currently designing Phase 2 and want it to be as robust as possible. We're looking at human baselines, multiple judges, more objectives, a larger environment, etc.
64 points by Davisb135 - 33 comments
I really wish MUDs were still a thing. The text-based roleplaying communities were largely swallowed by Discord, where as before they existed scattered over proboards, jcink, and a few others.
Discord, being the imperfect text platform, limits the text and leaves most interaction limited by awkward formatting. To boot, even when a better ecosystem is invented, players refuse to give it a shot even if directly invited, because they got comfortable with Discord's terribleness.
It's so frustrating, and I don't know how to voice it, or even what can be done.
One adjacent project we're exploring for the future is if you took all of the current understanding of game design and mechanics with modern AI functionality, knowledge bases, etc, and built a MUD, what would it look like and would it be enough to bring back players from other games?
Maybe limiting npc's to only a certain number of moves that aren't a response to other users per day...
It could be a lot of fun.
With modern technology, training in both games is very automatable. But developing and optimizing your training routines is (in my opinion) incredibly fun. Actual roleplaying is less common it was in the 90s, but there are still nightly events where dozens of players will get together to socialize and explore the worlds in-character.
I suspect MUDs may have a bit of a resurgence soon, I've seen more and more interest in them as of late simply because they seem like an excellent pairing with LLMs which easily consume text. Though I think for your needs someone would need to work on the barrier to entry to getting on a MUD as well as the modern UI/UX expected today.
I think this could be solved with bridge bots to make your MUD multiplatform. You could have a dedicated web app, an IRC bridge, a Discord bridge, etc.
Discord-brained users can still use your MUD and contribute to network effects, and you can provide an off-ramp ("I just want to play VarelionScape, maybe I'll just open the dedicated website on my phone instead of going on Discord and digging through the server list").
About 15 years ago, for a while I ran a MUD over AX.25, because MUDs aren't nerdy enough on their own, and amateur radio - even packet radio - isn't nerdy enough on its own.
All three of us that used it found it pretty entertaining.
I started playing on an LP MUD in the late '90s. Over the past 20 years, I tried a few times to implement a similar environment, only with everyone being a "wizard". LPC was also a prototype-based OO language with multiple inheritance, and it supported live development, with code for objects stored in files and dynamically reloadable. I would always hit some kind of blocker, no matter the language I chose for the implementation. It looks like MOOs did (and do) what I wanted, but I just wasn't aware of their existence. :( Maybe if I used MOOs as a model, I'd get better results... but I'm not sure if I want to go all in on DB. I tried looking up how the code is loaded into the DB in the book (https://timbran.org/book), but couldn't find it. Is there an explanation somewhere of how MOOs are bootstrapped from sources, how you can maintain sources in files on disk, and how new definitions are pulled into the DB at runtime?
For one, I think MOO is kind of shite in lots of ways. But I and others spent lots of time 20+ years ago building alternatives and nobody used them. There's intrinsic value to just picking up and then improving on something that has established history. It gathers an audience of users, has name recognition, and most importantly keeps you responsible from an engineering dev POV because you are forced to focus on delivering a thing that works and you have a way to prove it.
Secondly, existing "mainstream" languages are not appropriate, mostly. None of them or their runtimes are really based on a properly sandboxed persistent model. I poked at WASM, and I could write a screed about that, but wrong system. MOO (and successors to it) were built with the idea of lots of people jamming on one shared world, and it shows.
Finally, I have all sorts of ideas on what I think is "better" than MOO, and did eventually open the floodgate this spring and start building out the "dream system" here: https://github.com/timbran-project/mica -- but as you can see from relative star count of one vs the other.. my first point has relvance.
There should be an explanation in the book on how loading works. If not, I'll need to clarify. Documentation writing sucks. But the TLDR is that one of the arguments you pass to the system at start is basically a source directory of ("objdef") human readable / editable source files, and that gets compiled into a persistent binary database. Once that's done, restarts load directly from that DB/image, not source. (But the system can periodically export or "dump" objdef files as well).
Note this is a bit different from classic LambdaMOO, which had no such process. It had a primitive text database that was not human editable really, and you started from what people passed around of those.
Hope that helps. Come by the discord if you have questions: https://discord.com/invite/Ec94y5983z
https://news.ycombinator.com/item?id=49001127
I think you have the correct idea.
If you ask an LLM to solve a multiplication problem using reasoning without code tools, depending on the model, it will get 15 digit (15D) * 15 digit multiplication correct (123456789012345 x 998765432109876). It will take between 4000 and 8000 tokens if Sonnet. Eventually with enough digits, it will start only solving the problem 80% ... then 70% of the time until it has so many digits it will never solve. There will be a certain number of digits where it will not converge on a solution nor will it stop working thinking it can solve it. That is very, very expensive. It takes a lot of runs to determine the probability it will solve it.
What you can do now, this is likely the most important thing, is change the prompt and evaluate how many tokens and at what speed it takes to accomplish the task. Sure the measurements of each LLM are important! Nonetheless, if you can say to a company that you have a technique to tune prompts and evaluate them so instead of spending $1 X 100,000 times a day, they can instead spend $0.90 X 100,000 times a day, you will make a ton of money.
That brought back memories when we were trying to find free terminal in few departments at our uni to get access to one that has internet connectivity.
I vibe coded a few room proof of concept MUD. I added a tools called `oracle` (aka me) that it could ask me questions to help along its way; and the ability to interject in the loop with a hint. Just as I might want an llm to stop to ask me for help instead of plodding along...
Interesting to limit the number of turns and see where it gets stuck or how quickly it can finish.
[1]: https://www.bbc.co.uk/programmes/articles/1g84m0sXpnNCv84GpN...
We then decided to pivot and thought, what if this would be a good eval environment for LLMs from the simplistic standpoint of, text is their native environment.