> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.

I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.

That 200M figure needs much more justification as well, humans recieve huge amounts of data through their senses, particularly vision and touch.

A nice source for this might be "The unbearable slowness of being: Why do we live at 10 bits/s?" [1].

A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year [2], according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.

[1] https://www.sciencedirect.com/science/article/pii/S089662732...

[2] Assuming 16 waking hours, and 10^9 bits/s of input that the paper mentions. 16*3600*365*10^9 = 21.024*10^15

Nah. The "huge amounts of data" humans receive are incredibly self-redundant.

Incorporating non-text data into AI training runs has repeatedly demonstrated that for general reasoning capabilities, no data type beats text in value per byte.

This is only true on the surface. Yes, making a model a good debator can be acchieved like this, but every current frontier model exhibits the same semantc confusion when it comes to actual mechanics of physical objects. Understanding how not to pass the ball through the glass is not a property of textual reasoning. This is also very visible in model generated code where the ball and the glass plane very often intersect. Easy to identify in syntax (compilers will do the job for you), easy to miss on the concept level.

It is very redundant, but there is an incredible amount of it and we can "train" continuously and much more efficiently.

I agree with you. Also I really doubt we can so easily tokenize the input to the human brain.

If those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they?

This is such a beautiful question. It looks like a miracle that a brain, let alone a whole body, can grow from such little data. Michael Levin has been doing a lot of research on this, you should check him out.

The data is massively compressed and pruned. Where are the input tokens in an LLM?

DNA is probably the seed for the random initialization.

This is actually a genius idea. Imagine being able to compress a 1T parameter model to 10M params by creating a lossy model-scaffolding, like DNA

Something that more or less guarantees you’ll get about the same capabilities as the original model, but with a different final expression of certain characteristics

I wonder if this could be a good distribution model too, selling “dried” models (almost like raisins - or the beings from the 3 body problem), and then you can buy them to fully train (raise?) with your own data

It's nice to throw around claims like that, or like "microtubules connect the brain the quantum realm" (Penrose). Fortunately that's not how science works.

> If DNA has everything encoded

It is not. There are many biological examples of random processes generating patterns and weights, such tiger stripes, fingerprints, etc. DNA only encodes the reaction, but the pattern itself is seeded by initial position of random molecules, not the DNA.

> that it influences the initial state of the brain.

You've gone from "DNA is probably the seed for the random initialization", as in the deterministic seed for a PRNG, to "influences". Those are 2 different claims.

Most scientists will agree that DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc.

Most scientists will disagree that it is "probable" that DNA encodes/seeds the weights of each and every synapse.

I think you are mostly debating against a strawman of your own construction here. The person you're responding to never claimed that "seeding the random initialization" means literally the same thing as seeding a pseudo-random number generator, and the second message was likely intended as a clarification after being misunderstood. So it's not "2 different claims" contradicting; it's a single underlying claim being expressed through a lossy medium in an increasing level of detail.

My own reading of the poster's intent aligns with this: "DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc."

Tokens being passed on would be what, you close your eyes and experience your ancestors lives?

No. More like, your ancestors that have a fight or flight response survive more often and have passed that down.

Or that infants with a rooting response survive much more often.

Basically, weights encoded biologically from hereditary behavior.

Instincts.

Now, the real conundrum is that if our DNA can really only encode ~12gb then how are we passing this information down?

You sound just like Noam Chomsky when it comes to this :) He always talked about the LAD!

> Our new paper (Baron et al., 2026) finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget.

The dataset is not held fixed in practice. It would be interesting to see an ablation study of where best to allocate compute between more synthetic data, more RL rollouts, and this paper's suggestion of cranking up the parameter count. Unfortunately that's an experiment only a frontier lap could accurately do, and they are unlikely to provide such expensive and valuable information to their competitors.

The planet has 20 quadrillions of ants

And several thousand quadrillion grains of sand.

Edit: "several thousand" could be in the order of millions! (i.e. billions of quadrillions of grains of sand.) Apparently the number of grains of sand per this planet is not so clear. At least to me.

There are more ways to shuffle a deck of cards than there are atoms in the visible universe.

A typical grain of sand contains in the order of hundreds of quadrillions of atoms. A deck of cards can be encoded in 226 bits per deck, so one could encode in the order of a quadrillion decks of cards in a single grain of sand!

Paper: https://arxiv.org/abs/2610.37842

Yes synapses are weights, you know science, you smart