AI - the gift that keeps on taking
8 min read
2 seconds. This is how long it took LinkedIn to suggest the first post about LLM-assisted software development. Other platforms are not much better. It seems that all managers have collectively drunk the Kool-Aid and treat LLMs as the Second Coming of Christ. Their prophets are called Sam Altman and Dario Amodei. I guess every developer has been told by now that AI will replace them within the next six months. For the past few years. Or perhaps you’ve heard a more differentiated view, from someone whose synapses haven’t atrophied due to token limits yet: “AI won’t replace you. You will be replaced by someone who knows how to use it.”
Putting aside the immense cognitive effort of typing “plz make this feature. make no mistakes”, it is undeniable that LLMs can be useful. To some degree, it is truly magic to see a machine take your vague input and work its way through to some result.
And perhaps that’s exactly what makes them so appealing. They make producing code easier than ever before, even though writing code was never the whole job.
Software engineering is about more than making code work. It’s about understanding the problem, making trade-offs, and building systems that we can trust.
The process of building software also gives us an understanding of the product itself. We develop a mental model of how it works, one that helps us maintain it years down the line, long after we’ve moved on to new features.
We seem to be getting increasingly good at producing code, without getting equally good at deciding whether it’s any good in the first place, let alone whether it holds up to scrutiny.
So, will AI eventually replace developers for good? Well, I can already hear an MBA climax over this idea. But perhaps that’s the wrong question. The question isn’t whether AI can write software, either. It’s this: what do we give up in return?
At first glance, this may sound like a philosophical, artsy question; perhaps even a romanticization of the past. And perhaps there is some truth to that. I rather like a quote from Christopher Paolini’s Eragon:
When you can do anything with a word, the only things that possess any worth are those which you create with your own two hands.
But I believe we stand to lose much more than meaning and the satisfaction of creating something ourselves.
We lose:
Our skills. Our independence. And our agency.
We give up skill
ChatGPT, fix this bug.
I am sure that, at some point, everyone has done this in one form or another. I certainly have. As we sit in front of the screen, trying to find a bug, it becomes increasingly tempting to ask an LLM for help. It seems insane to “waste” an hour or two when you have access to a magic box that promises salvation.
The next time you encounter a bug, the threshold for reaching for AI is a little lower. And the time after that, lower still. Eventually, you end up with an OpenAI mug and alarms set for the five-hour limit reset. As our capacity-hostile industry keeps pushing us to ship more, faster, we start relying on generative AI for more and more of our work.
And that’s where I worry about the invisible loss. Software development is a craft. And like any craft, skills we stop practicing are skills we risk losing.
But there is another issue creeping up. To reliably evaluate and correct a system’s output we need to be capable of recognizing when it is wrong. In a sense, we need to be a stronger model than the one producing the output. For years, software engineers filled that role. We weren’t just writing code; we were also the ones expected to understand it, spot its flaws, and decide whether it was fit for purpose.
In MBA terms, we’re paying to erode our own human capital.
This isn’t a mere philosophical consideration. Some research is beginning to suggest that this effect is measurable.
The irony is almost too good. Anthropic itself, one of the companies building these tools, ran a randomized trial on exactly this question. The researchers, Judy Hanwen Shen and Alex Tamkin, had 52 developers learn an unfamiliar Python library, with and without an AI assistant, followed by a quiz on the concepts they had just used. The AI group scored 17% lower, which the authors call a “statistically significant decrease in mastery”. And in exchange? They weren’t even meaningfully faster.
To be fair, the study is small, its participants were mostly junior, and it only measured what stuck right after the task. It also found that how people used the assistant mattered: those who asked it to explain concepts held up fine; those who let it write the code fell behind. The tool alone isn’t the problem. The problem is how we use it when the deadline is on Friday.
But one detail stands out. The biggest gap was in debugging, which is precisely the skill you need to notice when the machine is wrong. Hold that thought.
We give up our independence
Claude, add this feature. Make no mistakes.
Convenience isn’t the only reason our reliance on LLMs is creeping up on us.
With the advent of coding agents, the sky token context is the limit.
And remember that study? It used a simple chat assistant in a sidebar. The authors themselves expect the effect to be more pronounced with agentic tools like Claude Code.
Features that were once deemed too expensive or too trivial to implement are now getting shipped one prompt at a time. Code has become cheaper to create than ever. Not necessarily better code, but more code.
Coming back to Paolini, one grasps the true nature of a creation, well, by creating it. A few years ago, we’d sit down and think through the problem we were trying to solve. And sure, there are probably some AI subscribers whose synapses haven’t fully atrophied yet.
However, when managers keep pushing us to deliver faster, the amount of code we need to review grows while the time available to review it doesn’t. Reviews become more superficial. And when we spend less time thinking through the problem ourselves, we understand less of what we’re actually shipping.
And this is where losing skill turns into losing independence. The less capable we become of evaluating AI-generated code ourselves, the more we have to rely on the very tool whose output we’re trying to evaluate.
If we let those skills atrophy, we undermine our ability to evaluate the very output we increasingly depend on. The less capable we become of catching its mistakes, the more we have to rely on the tool itself to fix them.
In the past, we often had a much more direct relationship with the code we wrote. If something went wrong, we had at least some idea where to look and why things behaved the way they did. Now, we may have to painstakingly wade through the token puke, or simply concede that we’ll need an LLM to clean up after itself.
This is a feedback loop. We ship more. We understand less. We’re more dependent on the tool that got us into this situation in the first place.
And then there’s the pricing. Token costs are heavily subsidized, and we have yet to see how far the enshittification of these services will go. It feels a bit like a drug dealer handing out free samples: the LLM is the drug, and today’s cheap access is what gets us hooked.
We’re prompting ourselves into a situation where maintaining our own software without LLMs may soon become untenable. Either because of the sheer amount of unvetted code we’ve produced, or because we’ve lost the skills to maintain it ourselves.
We “wrote” it. We own it. And yet we can’t maintain it without the machine that wrote it.
We give up our agency
You are an expert, please…
Even before AI, the standards of software engineering were eroding. If it worked, it shipped. The dogma was “Fail fast, fail often”. You will refactor later (never).
Software has become increasingly complex, and AI will only accelerate that trend. As we prop up the looming IPOs of Anthropic and OpenAI with our payments, in a desperate attempt to avoid thinking, we increasingly outsource reasoning to spicy role plays with a machine.
Concepts or issues whose domain may require some research are now being outsourced as well. We relinquished control over code; now we hand over control of the architecture, simply because we learned to depend on the LLM to reason through unknown domains for us.
Even if the domain is known to us, the micro-decisions we no longer make still
matter. An if statement here, an edge case there. Summa summarum, they form a
system over which we no longer have full agency, simply because we never really
thought through the decisions that produced it. And making these decisions is
often how we discover what we don’t know yet.
Agency doesn’t only disappear when we hand over one big decision to AI. It disappears when we stop making or even understanding all the small decisions that shape the system.
So. Where does it leave us?
In my view, we may be heading towards another Software Crisis. The previous Software Crisis helped give rise to software engineering as a discipline. Not because we couldn’t produce enough software, but because software was becoming increasingly difficult to develop and maintain.
And this is where “good enough” becomes dangerous. Software engineering has always been about trade-offs. Not every problem needs a perfect solution, and not every line of code deserves hours of scrutiny. But there’s a difference between deciding that something is good enough and deciding that it looks good enough.
I think this AI-generated music video captures it rather well:
LLMs are very good at producing code that looks plausible. And when the pressure is to ship faster, plausible can quickly become indistinguishable from correct. We risk normalizing a culture where code gets shipped not because we’ve established that it’s good enough, but because nobody has the time or the incentive to find out.
We pretended that writing code was the bottleneck and solved it. But understanding and maintaining software was always the real bottleneck, and we are now making it more expensive.
Anyways, my 5-hour limit has just reset. See you!