We have been building GFQL for open source vectorized CPU/GPU cypher on dataframes (think duckdb for graphs, no DB needed) using agentic assistance... and our thinking is this is a stepping stone to something even more aggressive
Today, we embrace optimizing hotpath specializations for some patterns that would have been too expensive to maintain before. LLMs keep that cost bearable, and being able to do that for important users (like users of our analyst product Graphistry) is a superpower. Our scores on graph community benchmarks are beating the big guys - and importantly, on anything we care about.
Where we really want that to go is runtime / loadtime specialization. Our customers would absolutely want their top queries autotuned. DB research is already heavily tilting here, akin to tracing compiler research a decade ago. I'm quite bullish on what is now possible.
[3 hidden]5 mins ago
How would you find out which optimizations to apply to which queries?
One decision that brings a 10x improvement, might slow down a very slightly different query by 100x.
Thats the core issue of database query optimization, and thats why (good) optimizer target the 99% percentile, not the peak, or the average.
Not blowing up is the most important criteria to be able to rely on a system.
[3 hidden]5 mins ago
> Which brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
[3 hidden]5 mins ago
I think you're both right, but Neumann much more so.
It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.
What is the alternative? Test all the queries on all datasets?
I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.
Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.
A system is what it does, unfortunately.
[3 hidden]5 mins ago
All benchmarks are vulnerable to manipulation.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
[3 hidden]5 mins ago
The author is correct in pointing out that this is a continuation and amplification of an existing trend.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.
HN.zip
Today, we embrace optimizing hotpath specializations for some patterns that would have been too expensive to maintain before. LLMs keep that cost bearable, and being able to do that for important users (like users of our analyst product Graphistry) is a superpower. Our scores on graph community benchmarks are beating the big guys - and importantly, on anything we care about.
Where we really want that to go is runtime / loadtime specialization. Our customers would absolutely want their top queries autotuned. DB research is already heavily tilting here, akin to tracing compiler research a decade ago. I'm quite bullish on what is now possible.
Thats the core issue of database query optimization, and thats why (good) optimizer target the 99% percentile, not the peak, or the average.
Not blowing up is the most important criteria to be able to rely on a system.
Good! That means the benchmark was always prone to be gamed and misrepresented, and deserves to be obsoleted and ignored, or at the very least heavily disclaimed.
It's frustrating that people are betraying common trust by making misleading claims and engaging in misleading arguments, but ultimately this will improve rigor and decision making.
At least when it's a bounded topic like this, it's not the same as it is with everyday subjects, where misinformation ends in a death by a thousand cuts situation (for now anyways) and is generally endless.
It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.
What is the alternative? Test all the queries on all datasets?
I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.
Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.
A system is what it does, unfortunately.
At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
I'm in the data space fighting the good fight, spreading the gospel of Just Using Postgres, and I'm seeing more and more benchmarks that look extremely dubious. Query performance is like choosing a data structure in ordinary programming: it depends almost entirely of what access pattern you expect at runtime; it's entirely pointless to go search for the world's fastest linked list if you needed a hash table. Compare with the situation in the modern data engineering landscape. Are you sure the solution measured by the benchmark is actually a solution to your problem, let alone a significant improvement over Just Using Postgres? Have you measured or are you going on marketing and vibes?
Again, I'm sure solutions like this have a place, but at least right now it's not the same place as traditional engines.