A natural evolution of engineers losing touch with the customers and users.

I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.

I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.

I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.

I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.

I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.

I think LLMs have some of the same risks and benefits of stimulant drugs. They can make you more productive if used effectively as a tool, but they can also delude you into thinking you are better than you are and create a dependence such that you aren't just less productive without the LLM/drug, you fail to be productive at all because you don't know how to function without it.

Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.

Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.

If an RPA team ever gets direct access to a production database in your company, look for a new job.

> Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces.

For anyone else wondering what RPA means. Never heard that abbreviation before.

If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.

I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.

If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.

This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.

There is only so much need for taste and judgement, before even that has been incorporated into the models.

> If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.

I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator.

I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.

Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)

We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc.

I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes.

We recently had a situation, where in with our internal platform and claude we recreated everything in minutes.

Management in the end cares about the outcome and not how the meat is made.

The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).

Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.

I find the use of AI like quicksand.

The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.

Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.

Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?

That’s the big question for sure

Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM.

Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.

There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.

Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.

"THIS IS NOT A DRILL"