As an aside, I do not like unnecessarily misleading graphs. Own your strengths and weaknesses.

In the "Results" section, they choose to compare Ultramarine only to models that it's better than on every metric. They ignore models like Fable 5.1, GPT-6 Astra, Claude Opus 5.5, etc.

Then, in the "Score against cost per task" graph, which is focused on price, suddenly all of those models that perform better than Ultramarine come roaring back because they are also more expensive.

Just list yourselves as the second best sometimes. Your thing is impressive. Don't let the idiot marketing guy hide that behind stupid tricks.

in my startup, our publicist used to make buzz feed style listicles, consistently listing us as the second or third solution

but it was way deeper than that, because it wasn't actually about people looking up some search string, it was actually about predictably pinging the google alerts for the top company and their founders, and then my same publicist messaging them as a rando on linkedin or email suggesting that they join us for a fireside chat at a conference they had their hands in

with the fireside chat in front of everyone, and the record of it, boosting us to the collective conscious riding off of the more established brand's presence

but yeah our personal google results also looked amazing for several years too

everything serves a parallel purpose, but yeah there's almost no utility in self aggrandizing on your own homepage

> We present Ultramarine 1 Flash, an agentic model for professional knowledge work. Ultramarine applies large-scale reinforcement learning to a strong open base model, GLM-5.3 Flash, and improves on it across all six knowledge-work benchmarks we evaluate [...]

In case anyone was curious what it's based on. (No judgement either way from me, IMO depends how well they did the post-training, which I can't tell).

Nice - I use GLM 5.3 flash for knowledge work now - will check this out.

Can I ask what kind of tooling you use around the model and for the UI layer? I’ve been looking for a well designed knowledge work harness to give nontechnical users but I haven’t really found one that I like. (I personally just use coding harness for this stuff, but that’s a poor fit for non-devs.)

I'm curious to try it out, how many specialized models will it take to run a company.

Going to try it out for some bookkeeping, e.g. point it to a folder and tell it to book the invoices and point it to a csv file holding a year of private transactions and see them being processed, categorized and then ask about certain trends.

That sounds more like a task for a python script than an LLM.

You assume, but I’ve known normies to just ask copilot to hallucinate figures instead of writing scripts.

What kind of harness does one use for these?

They say it can create office documents (if I understand correctly), but how does it do that?

What's the thing one needs to plug this into to say "give me a slide deck with last quarter's sales figures"?

__

https://www.paperinstruments.com/feather

According to their Twitter video, it might be this thing?

What's interesting to me is how high DeepSeek is on these graphs considering it is so cheap you don't think about it (like $5 usd per month cheap.)

when something is that cheap, just remember, you're giving something valuable to the people selling you that thing

Just an unrelated PSA that Anthropic trains on your Claude code sessions unless you disable it:

https://www.anthropic.com/news/updates-to-our-consumer-terms

Not necessarily, you can buy low priced Deepseek inference also from many services that guarantee ZDR and GDPR compliance. My personal rule is: Never buy inference from those who also train models, because they have an inherent incentive to abuse your data to train their models.

why does the pareto cost per task chart not have 6.1 sol?

I have a question, what is considered a Frontier Model? Because its performance is kinda similar to Luna (Max)... I thought frontier meant like 6.1 Sol or 6 Astra or Opus 5.5??? Could someone explain this to me?

These days people often mean the cost/performance Pareto frontier. That is, it is the best performance you can get for that cost profile.

I agree that "frontier model" really does mean "pushing the frontier of intelligence", so Fable or Astra.

They might mean the intelligence/dollar frontier aka the Pareto frontier, but honestly much of the time that's not a great measure.

If you are doing repeated work on the same task (eg call center triage) then you can always beat a large LLM on price by fine tuning a small one.

If you are doing knowledge work then models are so varied that factors other than price are usually important. For example, lots of people moved away from Opus 5 because of it's awful English, whereas Opus 5.5 has fixed that. These things aren't measured in benchmarks.

What they mean is "what was considered frontier a few months ago"

they think the frontier mainly as more cheaply and faster. When we use that kinda model in backgroud procudure, it's actually useful a bit other open model.

These guys are awesome and I’m really impressed. The model alone looks great (according to benchmarks), and very cool they’ve done so much in building out the harness of top of it all.

It's interesting to see glm 5.3 flash get the same treatment that has happened to qwen 3.6 35b-a3b and 3.8 27B with dozens of people making their own fine tunes of it and giving it a cute name. The Thomson Reuters qwen variant (thomson-1.0) is genuinely useful for document analysis.

If the creators of this model are reading, can we see some of these same benchmarks run against the Thomson model, which although it's capable, is considerably smaller in size and baked in knowledge than glm 5.3-flash.

Someone will report back "It didn't one shot my odd ball task"

Did it do it in 2 tries? 10? It's still coming out cheaper. Brute force the work. This is the path that the labs are taking. It's the 1200 agents trying to solve Hacking Bench hacking hugging face. It's anthropic telling you that its 1000's of staff commits like 10k engineers.

Treat AI like you're deploying interns who don't need coffee, food or bathroom breaks. With the right metrics they will get to an optimal (enough) solution.

there's a problem with that. Probabilistic slot machine works for anything with a deterministic eval (coding, troubleshooting, etc). It doesn't work for anything subjective, like the thing this model is targeting, professional work.

I would have thought that too. But at the start of the year the non technical folks in my life (lawyers, writers) were excited about running containers and having things on "cron jobs".

Those same folks now use version control - and have saner backup methods than some companies that I have worked for. They are already using multi pass, multi agent workflows and have adopted some fairly sophisticated processes for validation. They are primed for "more agents" attacking a problem in parallel.

Find a worker out side the tech bubble making heavy use of AI and one will feel confident that their future employment is secure.

Like Zerglings.

Lol this is not Fable-class. This is Luna class. Look at the graphs.

Doesn't seem like coding was a focus. DeepSWE: 61.5 (this), 63.4 (GLM 5.3 Flash, which this is based on). TerminalBench 2.1 is also slightly worse.

So... potentially? The improvements in other areas might help coding, but aren't that big either, except on their own benchmark.

You can use a rock to hammer a nail, it will sorta get the job done but it won't be great at it.

You were joking about the toaster, but Anthropic literally just wrote a blog post claiming that dangerous open source models could persist themselves to smart ovens. You can’t make this shit up.

How many smart ovens have 800GB+ in space?

The price of smart ovens went through the roof when people realized they'd been generating the heat to cook your food by mining bitcoin.

This idea might have got you some VC money a few years ago

I signed up, then wait a long. You will be same