[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927

NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.

An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.

So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.

LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.

I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.

If you want to make a ticket classification model, you don't need to train across many customers doing ticket classification, who might not want you to train on their data and whose data is impossible to obtain unless you already are a large company. Instead, you can train on a completely different task which forces the model to learn the syntactic and semantic structure of the English language — say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English — and then fine-tune that model on your specific task with much less data.

Because the initial task is so general, you can have one company do the expensive pretraining, share the base model, and then have everybody else just do the finetuning on their (small) proprietary dataset.

LLMs and models like Jev go one step further, letting you use them as classifiers with no training at all, with nothing but rules in the prompt. And with a few tricks, you can indeed use them to get pretty high-quality labels, which you can then distill into a fine-tuning dataset for a much smaller model.

  > say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English

Word2vec is a two layer perceptron.

I feel like people in the field forgot everything.

True, but could be taken the wrong way. LLMs were also not able to train on that private customer data. They trained on so much general public data that they became capable enough to operate generally even on private data they had not been trained on.

That was the big breakthrough. I know that’s what you were trying to say, but on sites like this there are a lot of people who will see a comment like that and assume that LLMs like Sonnet were fed all of the private customer data that was withheld from training other models.

[1] I went down the rabbit hole of using sequence kernels quite a bit back then, see https://arxiv.org/abs/1910.13800, but the timing was terrible. RNNs and variants like LSTMs were beginning to become popular, and everyone (including me) switched.

[2] https://pocket-text-lab.abhishek-ghose-82.workers.dev/

You can also do stuff like setup active learning but have the labeller be an llm as well to make things more efficient.

Definitely worth it for PoC level things if not full ones.

Edit - also reviewing, so it can run training, test, review failure cases and investigate itself. You can leave them to plough through a few iterations where the early part is the classic time you’d have a few obviously bad sets of results.

Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.

I think we're seeing the modelling equivalent of something like electron apps. Where they waste a tonne of resources, but make an easier developement experience.

Will be interesting to see how much or little this changes with LLM costs.

So between llm and pseudo experts, at this stage, I prefer to go with a LLM.

Just like, LLMs can build web back ends, but I'd want a security professional to give it some kind of thumbs up. Especially if a lot of money rides on it.

Your model only took minutes to train if you don't count the data collection and labelling work

Saying that the field got reduced to only LLMs is like saying that the field of combustion engines got reduced to cars. The trucks, combine harvesters and peaker plants are still here, they're just in the background, invisible to people who are not part of the industry.

Banks still use random forests and SVMs, medicine and image processing still use CNNs (and people who take photos with their phone don't even realize how much image processing is being done), TTS, ASR and predictive typing models are in wide use, and there's probably still plenty of BERT in spam classification and such.

In software engineering, we learned a long time ago that giant monoliths are not the answer. You want smaller, but specialized units that work together. They perform better at their specific task, are more maintainable, you can just upgrade one part of the system without having to impact the others, ....

I don't know where the focus on building a single model that can do everything comes from.

LLMs are the powerful general-purpose CPU of the ML world. Using an LLM as a classifier is woefully inefficient compared to using a dedicated classifier. On the other hand, that LLM exists right now and you can hook it up in ten seconds. You could ask: why use an LLM when a dedicated classifier would be so much more efficient? You could also ask: why build a dedicated classifier when a sufficiently capable LLM is available off the shelf? Neither one is the right answer every time.

That depends on the question. If the question is "How do I utilize 10,000 software engineers?" then yes, a monolith is not the answer. However, this isn't a problem most companies face. Starting out with microservices is rarely a good idea either as you'll have a hard time getting the boundaries right.

If you want a model that can do X but don't care about Y And Z, intuition says you only have to train the model on data relating to X. However, early LLM research showed that doing this results in a far less X-capable model than one trained on data for Y and Z as well.

I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements. This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.

Coding an exercise app will benefit from a model that can code and understands what exercise is. A car showroom website will benefit from a model that understands what relevant filters would be.

Nah, let’s stand firm here and not give in so easily. Parent suggests a coding model needs exercise info in case you want to code an exercise model? Absurd, that leads to the know everything monolith to do anything.

Language, reasoning, tool use, and domain knowledge.. in this case the domain is coding. Someone wants an exercise app? Great, so we look that shit up, we don’t bake it in. We don’t want to bake anything we don’t need in.

We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful. it’s the other side that needs to justify wtf exercise has to do with coding or why everything from twitter posts to a working knowledge of 15th century French poetry just might be helpful for disconnected domains, like say serious work in contemporary astrophysics. Back to basics, No one asked for that, No one should accept as given that this is the way anything needs to work

No, the point is having broader knowledge than just "can code" is important for building software for humans to use.

> We don’t even need to explain this completely reasonable and totally normal engineering preference that small is beautiful.

It is, of course it is, as long as that small thing is useful. My argument is not "let's put literally everything in this thing just in case" it's "broader world knowledge is important for building many things".

Also, we don't look at CVs or histories of people and say "hmm, this person could be good at coding for us but I really wish they didn't have such a broad and expansive history".

> Great, so we look that shit up

But what do you have it look up? What is exercise? What is a muscle and what is a human? How is it going to understand those references? Does it have the broad enough understanding to see where the gaps are?

At some level unless all you want is "loop over the array and sum the results into a variable called sales_total" converted into code, you have to have some broader knowledge about things.

And once you start considering that maybe it needs more than simply code, we're not disagreeing we're just discussing where the line should be. Which I've also not placed, because there's no simple statement to make on that.

Damn right we don't say that. And I don't look at a carpenter and wonder if his head is hard enough to drive nails like a hammer either. People aren't tools, and tools aren't people. You've conflated them because they are both capable of doing work. And that's exactly the type of confusion and category error that the perspective I'm arguing against will lead to.

Wanting a multipurpose tool or system is occasionally reasonable. Much more often it is very, very to close misunderstanding the nature of a problem, what a tool is, or what a system should be. The particulars of purpose and mechanism of action matter, which is why a blowtorch is sometimes a wrench but only if you're kind of desperate. It is always a very scandalous thing to speak ill of someone's best new electric banana-phone / dishwasher-piano that they want to sell you. But that just sounds messy to me, I'm not sure it will improve the sound quality or taste of anything involved.

The Bitter Lesson is that human-written rule models will always lose out to trained models in the end.

Does it? All of the latest frontier models that have made technical details publicly available are using MoE; smaller specialized units that work together.

"AGI"

With the LLM you don't even need a data scientist to train the model, you just need someone who can connect to APIs. And the LLM is fine with adding and removing categories. No retraining.

It makes the initial faces incredibly fast. Then it is slow and expensive to run, so once the system settles, it becomes the target for model optimization, e.g, a dedicated, or semi-dedicated model.

I see coworkers using AI for tasks like "visit an API endpoint, download a file, put it in S3". This is so stupid. Okay if you want to use AI, but use it to write python code to do that, why pay in tokens every time you want to run that task, it makes no sense

Probably under the hood Claude creates some temporary python code or something.

But it bothers me, I don't know why. It's like paying a subscription service for the most stupid tasks

That is the reason.

Once you then have a system that is used enough, it starts making sense to look to replace that with a leaner classifier, but LLMs have a place in vast number of places where we'd previously never use a classifier because it was too much effort.

Absolutely! But that doesn't rule out that there are AIbros that will tell you to use an LLM, even if the task could be done with a static analyzer with a few rules. Use an LLM because it's AI!

I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.

So, a lot of tickets generated would simply just be data reports with some info dumped in the ticket description, designed to be auto-closed but possibly useful later as an easily-accessible timestamped reference to some data.

Lots of other auto-generated tickets were "probably not an issue but it should pass over the eyes of someone knowledgable about this server/network" tickets. Like Windows Event Log error roundups, CPU usage spikes, or specific triggers that were set up in the aftermath of some catastrophe to help preemptively avoid a repeat incident.

Anyway, tickets are a great "first line of filtration" where you can take a massive ingest of all different kinds of data (hardware metrics, system logs, network activity), apply meaningful human context ("This pattern could be a sign of xyz"), and store it in a unified database. With that use case, it's easy to imagine a company with millions of customers and millions of servers producing millions of tickets.

I checked just for fun and there are actually a LOT LOT more than 92M tickets in the past year.

heck, there's even a ticket if the office toilet is clogged

I'd rather have tickets for everything and full transparency except for confidential info.

If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded. If your Boss belongs to that club oh man god say the division.

“submit a ticket!”

1. Some auto-generated ticket (e.g. a trigger from unusual hardware activity or a log entry) resembles a previous ticket that had meaningful action on it, addressed and solved by a specific team. That ticket could spend a week getting ignored or punted around when the right team can solve it in 5 minutes. ML could detect the similarity and reclassify it by that metric. 2. Human-submitted tickets are annoyingly fallible. For example, an "I can't log into my account" ticket could be auto-classified as "account recovery". Or maybe, e.g. the ticket system attaches a recent activity log from that IP address to be cross-checked by ML, which finds 5xx errors and reclassifies it for the technical team.

These are generic examples, but you can use your imagination and see how a company as huge as Amazon could benefit from this. I think it's for the purpose of minimizing the number of tickets that need to be manually reclassified because the wrong info was given to the wrong person.

It’s not that baffling Amazon has something like 70k+ engineers worldwide. Never worked there myself but in large orgs things sometimes get lost in the shuffle.

I don’t think timelines were shared but this reads like a pretty good case of building an MVP and then lowering the cost once it’s proven. One of the whole value props for LLMs from a ML perspective is being able to get something out the door a lot quicker than you would otherwise.

https://strandsagents.com/blog/introducing-strands-decider/

They lopped the head off an existing Qwen or Gemma model and used its torso with their new head.

Strands Decider is a decision model. maybe similar to Jev, that while based on an LLM is not one - it has a different "head" since it's not build to generate. The original prototype was based on (a modified) Qwen 3.5 2B, and the production version is based on Gemma 4 since that proved to be more accurate.

IMO the real news here is the big picture, not the details. Here's a high volume real world use case where a large expensive LLM (Claude Sonnet 4.6) has been replaced by a tiny 2B bespoke decision model.

Amazon is such a huge company that I'd guess they have internal business flow automation opportunities matching those of many other companies, so it's interesting to see this come out of AWS. Just as AWS itself was created by Amazon releasing something they has originally built for themselves, I wonder if we'll see more of these optimized real-world AI solutions coming out of Amazon/AWS/Strands?

After that, I'm guessing it was just fairly low priority to update a working system. Amazon's net income last quarter was >$20B. Saving millions is huge on the scale of one or two engineers spending a week tackling it, but it's too small for the organization to make it a big initiative. By the time engineers decided to update it, decision models like Jev were popular and are naturally a better fit for the task than an LLM.

If you think the former is a likely hypothesis, then why did you even click on TFA?

As for your question, seeing what others are doing helps one decide both what to do and what not to do.

Why would Amazon go to another company to let them provide an adjusted decider model?

Amazon already has all these tickets labeld, they could just have build it themselves?

The blog post was focused on the switch to a decider model, not the initial architecture.

I think I'd be foolish to 'why didn't they just?', on the basis of <blank> because no mention was made in the post.

Never heard of Strands before, felt complelty different when reading it more like a sales pitch from a consulting company

I do not believe Jev is a “decision tree”.

I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.

It’s shocking how effective this is.

Isn't it detrimental?

So there’s some LLM use to develop a new version of algo, but maybe 2% of what would have been used by putting LLMs in the path of every API call. That generally does have ROI.

Putting LLMs in the loop each time is generally a dumb move, unless you’re an AI bro trying to get everyone to use LLMs for all the things.

— "Let's build efficient classifiers"

— "Forget about the silly classifiers, LET'S BUILD A MACHINE GOD"

— "Ah, that machine god is often too expensive and underwhelming. Let's build efficient classifiers"

That's like... really basic LLM knowledge? Like how e.g. this article measures how often answers change based on order https://zachlandes.com/trolley-problem/#appendix-a:~:text=Ch...

Or is this the Claudish "One thing that suprised us" that just means "and here's another thing"?

Here's an interesting reference on this that gives a bit more background information:

-- 8< --

Because LLMs (and many other ML systems) are chaotic, it is possible to manipulate them into doing something unexpected through a small, apparently innocuous change to their input. These changes can be illegible to human observers, which makes them harder to detect and prevent.

For example, flipping a single pixel in an image can make computer vision systems misclassify images. You can replace words with synonyms to make LLMs give the wrong answer, or introduce misspellings or homoglyphs. You can provide strings that are tokenized differently, causing the LLM to do something malicious. You can publish poisoned web pages and wait for an LLM maker to use them for training. Or sneak invisible Unicode characters into open-source repositories or social media profiles.

-- >8 --

Source: https://aphyr.com/posts/412-the-future-of-everything-is-lies...

A modern model like Haiku 5.5 is still going to be overkill for many of these decision model use cases, and is probably at least 10x the number of parameters (Haiku 20-50B ??? vs Strand Decsions 2B Gemma 4).

"except that on long tickets Strands Decider drops the oldest comments to fit."

"except that on long tickets Strands Decider drops the oldest comments to fit."

Of course you can try to dig a hole with a hammer instead of a shovel...

But just think of the street cred!

Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.

Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.

The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.

They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".

These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.

I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.

I lived in Luxembourg for 4 years, which you have to order through amazon.de and is fulfilled by amazon.de, and I've never had any of the issues you describe. And that's with an average of 120 orders per year.

Use some logic before attacking: Luxembourg is only the size of Saarland (German state). Fewer orders and fewer deliveries naturally mean fewer problems.

How about you read before attacking? I said it's pooled with all german orders.