Today this is the second post I see where the 'traditional' testing methods (unit, e2e, integration, perf, pen, etc) are re-categorized, are put in new perspectives.
I'm betting on something adjacent but flipped. As agents are writing the code, humans are stopping (and getting worse at) reviewing, so the oracle needs to be mechanical. My personal bet: https://github.com/NMitchem/charpente/blob/main/MANIFESTO.md
[3 hidden]5 mins ago
Not sure I’m following, are you implying that deterministic simulation testing is a just a new name for a “traditional” test method? I’m not clear because 1. Deterministic simulation testing is not a new concept. (Foundation DB is one of the poster boys for it) 2. All the test methodologies mentioned are a name for a specific pattern and practice of testing, and DST is the same thing.
[3 hidden]5 mins ago
This is the future, I'm convinced.
I’ve been building a framework of my own and it’s already found dozens of important and critical bugs on my team at work.
The next problem to solve after this, though, is formalized intent.
A bug is often a bug if the app is not behaving as intended. What we intend is not always so clear, and it will remain an area of the engineer's and product designer's responsibility, because software is complex, and what we want cannot be computed so simply. Defining and understanding intent will remain a "human bottleneck" for many companies.
[3 hidden]5 mins ago
Imagine complaining about intent when the new paradigm's opening gambit is to solicit your intent in plain English on a silver platter.
You can't formalize intent if you are so sloppy as to miss the irony here.
You have measures you want to move and safeguards which are inviolate. What does intent have to do with it? Humans rarely know their own intent robustly, which is why it's usually always iterative anyway. The machine will find it well enough if you do not contrive to put a bottleneck in its way. I deeply suspect whatever you are trying to do will reveal itself to be far less complex once it escapes your own head. This is often the case. This does not mean it is necessary to formalize your mental phenomena. What we already know it means is you have to do more work to map your mind onto already existing formalisms. That's what we used to call thinking. You're simply being redundant. Most software is far less innovative than the marketing people would lead you to believe, through something akin enough to recursion the LLM will find it. The manifestation of your intent already exists out in latent space. Let the machine handle it. Most product designers are mercenaries for professional enshittifiers anyway. At least with ubiquitous LLMs any software "design" can become a runtime fact. Replace products and their overwrought designers with APIs and personal LLMs on device. Most people won't know the difference. Those who actually care will be infinitely happier.
[3 hidden]5 mins ago
Deterministic testing is one thing and it's badly needed.
But I'm totally convinced we're moving to deterministic execution: n various implementations, in different languages, running on different stacks, and a minimal (maybe even hardware) voter taking care of the n-modular redundancy to pick the correct "answer".
0-day Postgres CVE? No problem: one of the implementation gives bogus answers and is detected as broken.
0-day Java? Same thing.
This shall make 0-day in anything but the voters a thing of the past (and the voters shall be minimal and have a tiny attack surface).
The answer can be an API query but we can go much lower: network packets, deterministically created, by n systems and all going to the voter(s). Whatever the "answer" is, it's going to be deterministic and verified from the lowest possible level.
The implementations shall be written from specs, formally verified, etc. But the one thing that'll guarantee any fuck up is caught is going to be n independant implementations of the same specs.
And as soon as one of the implementation/stack return even just one network packet that doesn't match what the other systems do produce: something is fucked up and has to be fixed.
This shall require throwing a huge lot of the non-deterministic bullshit away but we'll see systems doing just that: n-modular redundancy and voters, at the lowest level possible.
I'm 100% sure we're going there (and AI is going to write most of that for us).
[3 hidden]5 mins ago
I agree 100%, and I’m definitely experimenting in that direction with my hobby projects.
[3 hidden]5 mins ago
interesting idea. any references (long-form) on this?
The other one is https://alperenkeles.com/posts/vocab-for-testing/
I truly understand our (only) job is to become the oracles of the machine: https://www.osequi.com/manifesto
I’ve been building a framework of my own and it’s already found dozens of important and critical bugs on my team at work.
The next problem to solve after this, though, is formalized intent.
A bug is often a bug if the app is not behaving as intended. What we intend is not always so clear, and it will remain an area of the engineer's and product designer's responsibility, because software is complex, and what we want cannot be computed so simply. Defining and understanding intent will remain a "human bottleneck" for many companies.
You can't formalize intent if you are so sloppy as to miss the irony here.
You have measures you want to move and safeguards which are inviolate. What does intent have to do with it? Humans rarely know their own intent robustly, which is why it's usually always iterative anyway. The machine will find it well enough if you do not contrive to put a bottleneck in its way. I deeply suspect whatever you are trying to do will reveal itself to be far less complex once it escapes your own head. This is often the case. This does not mean it is necessary to formalize your mental phenomena. What we already know it means is you have to do more work to map your mind onto already existing formalisms. That's what we used to call thinking. You're simply being redundant. Most software is far less innovative than the marketing people would lead you to believe, through something akin enough to recursion the LLM will find it. The manifestation of your intent already exists out in latent space. Let the machine handle it. Most product designers are mercenaries for professional enshittifiers anyway. At least with ubiquitous LLMs any software "design" can become a runtime fact. Replace products and their overwrought designers with APIs and personal LLMs on device. Most people won't know the difference. Those who actually care will be infinitely happier.
But I'm totally convinced we're moving to deterministic execution: n various implementations, in different languages, running on different stacks, and a minimal (maybe even hardware) voter taking care of the n-modular redundancy to pick the correct "answer".
0-day Postgres CVE? No problem: one of the implementation gives bogus answers and is detected as broken.
0-day Java? Same thing.
This shall make 0-day in anything but the voters a thing of the past (and the voters shall be minimal and have a tiny attack surface).
The answer can be an API query but we can go much lower: network packets, deterministically created, by n systems and all going to the voter(s). Whatever the "answer" is, it's going to be deterministic and verified from the lowest possible level.
The implementations shall be written from specs, formally verified, etc. But the one thing that'll guarantee any fuck up is caught is going to be n independant implementations of the same specs.
And as soon as one of the implementation/stack return even just one network packet that doesn't match what the other systems do produce: something is fucked up and has to be fixed.
This shall require throwing a huge lot of the non-deterministic bullshit away but we'll see systems doing just that: n-modular redundancy and voters, at the lowest level possible.
I'm 100% sure we're going there (and AI is going to write most of that for us).