It’s become almost impossible to avoid the “AI” evangelists spreading the gospel of how “AI” tools are helping them work faster and get more stuff done in less time, but do any of those claims have any basis in reality? Should we really be firing countless people and replace them with “AI” tools? Should we spend god knows how much money on “AI” tools and force employees to use them?
Well…
When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.
↫ Joel Becker, Nate Rush, Beth Barnes, and David Rein
We’re very much in the early days of proper research into the actual effectiveness and real-world benefits of “AI” tools for all kinds of professions, so a study like this definitely isn’t a smoking gun, but it does fly in the face of the tech companies and their evangelists shoving “AI” down our collective throat. With how much these tools get even the most basic stuff wrong, with how often they lie and make stuff up, I just can’t imagine them speeding up as many tasks as people claim they do.
At the same time, “AI” tools do definitely have a place for very specific tasks, and I think that studies like these will look different for every single profession and even every single task within a profession. It’s going to be incredibly hard or even impossible to come to a “theory of everything” on the effectiveness and usefulness of “AI” tools. It won’t be until this idiotic hype dies down before we can have a grounded, honest, fact-based discussion about which “AI” tools make sense where.
> experienced open-source developers working on their own repositories.
This is the important part ^. My interpretation is that when you know what you’re doing, AI slows you down.
So it makes sense that it started being used at the C-level, I guess…
I know someone who knows close to nothing about coding, and is “successfully employed” as as coder, getting by with AI. Basically throw shit against the wall until it seems to do what is required. Credit where it is due: he is learning, not just blindly parroting.
He showed me some of the results, I wouldn’t want to inherit that…
As a person with little coding knowledge, this has been a huge help to get simple scripts working. It helped me update some of my old code to do some things I could not figure out myself and our in house developers didn’t have time or I should say interest to help me figure out. To me, this is a great tool for people with a small amount of understanding and knows how to use it to build simple code. To a real developers, this is akin to breaking his leg and giving him a crutch (A.I tool) to walk. Developers in the near future may be unable to work without this tool much like the man needing the crutch to walk. Lord help us when coders aren’t able to improve the A.I when they start to faiil or do nonsense.
Yes. Conversely if you’re trying to wade through Microsoft’s abstruse API docs to implement a new feature or do something in a different language than usual, using an AI-enabled editor that can show you how to use the APIs right inside your code is an enormous time saver. It’s like Clippy on steroids.
It’s interesting that how people hand-pick facts from papers to support their POV.
Yes, it’s absolutely true that study was using one narrow scope which may not be yet suitable for “improvement” with AI, but the most damning part of that study is not even the fact that developers did thing slower with AI than without AI… but the fact that THEY WERE SURE AI HELPED THEM TO MOVE FASTER – even when objectively it was slowing them down!
That is the most damning part of the study… yet somehow it’s ignored in almost all articles than mention that report…
Many reporters have come at AI with an agenda (both for and against). I agree with Thom that we need to assess the merits on a case by case basis. Too many critics imply AI is no good because it can’t do everything, but doesn’t have to do the whole job to legitimately save time. Like any other job, programming involves a lot of menial busywork, which slows down the process even for competent programmers. To me this is where AI has the most potential in the short/medium term. Creating a code template or framework around a new library is a good example of this. The AI doesn’t need to finish the whole project to answer questions, provide examples, and begin prototyping. Code refactoring and applying API changes are tedious regardless of skill level. AI that specialized at this could be a huge productivity boost. It might even be a solution to the unstable linux ABI problem. We’re also in the early days of AI itself and things are still evolving.
There is a lot of hype to be sure, the bad AI will fade away in due time, but whether we like it or not AI as a whole is not going away and over the next 10-15 years I expect we’ll see more of it rather than less.
I do not think any of this is so cut and dry. The devil is in the details.
In my own experience, LLMs are quite useful for automating grunt work. However, you have to know what GOOD looks like. Because though it often works well the first time, you have to quickly reject everything that comes back wrong and try again.
LLMs can also be useful when you have no idea what you are doing. Not because they do it for you but because they can give you a place to start. In this case, you have to know when to get off and take-over and not get stuck on a ride to nowhere.
Which is related to the biggest time waster of all. “Chat bots” consider the entire conversation, including their previous contribution. So, when they go down the wrong road, it is hard to shake them off. They self-anchor. Arguing with a chat bot or trying to get it to offer a solution totally unrelated to its current approach is a waste of time. And you can waste A LOT of time doing it. If you find yourself trying to convince the bot to do something else, just start over from scratch. It is MUCH faster.
Perhaps where I have saved the most time with AI / LLMs is having it read for me. Throw a lot of information at it and then ask questions. It can tell you which documents you need to read and which you don’t. And if there is a term or topic that I am not familiar with, I can skip quite far ahead in the learning curve by asking it to summarize. It can provide CONTEXT, even if I do not want to blindly trust the information.
As for programming, I can get insight by asking an LLM to summarize code I have never seen before. What does this code do? What external code is it leveraging? Can you give me a one-pager of the API calls, function signatures, and custom types? All massive time savers. The same is true of build errors. Push the error to AI and see what it suggests. I may not understand the build error, but I may understand the proposed fix. And now I can quickly see how I would fix the problem. Not how I would ask the AI to, How I would. It amazes me how often an LLM will 100% correctly state what the problem is and then proceed to propose things that do not address it. First part: gold. Second part: trash.
So, AI is really like everything else. If you know how to use the tools, they can save you a lot of time. If you do not know how to use the tools, they are at best useless and inefficient and at worst a great way of making a big mess in a hurry.
I agree with your comment. LLMs let me do things I need, but I don’t know how to achieve on my own.
However, they lead to misperceptions among experts. An expert can think AI makes him/her faster, but can waste a lot of time dealing with the crap gpt throws at him/her.
The ultimate question is, what are we gonna do in 10 years, when we no longer have experts, and every coder is a CS graduate who is taught only to review and revise crap produced by AI. When we no longer have people who developed their skills by doing the grunt work (because it is done by the AI), we’ll be in trouble.
cevvalkoala,
I believe AI is going to displace a lot of jobs, not all at once but incrementally, and society will face trouble for that reason. However as nobody being able to do the jobs anymore: why does that matter when the jobs are obsolete? Before commercial scale farming most workers would fall into agrarian farming. Hypothetically someone might have said of commercial scale farming “when we no longer have people who developed their skills by doing the grunt work (because it is done by big-farma), we’ll be in trouble.” In a way the logic is sound, but it clearly assumes we have a reason to go back to the way it was before.
While I can invent emergency scenarios where we have to go back and no one knows how to replace it, it may end up being a permanent shift such that the need for those skills just goes away. In another thought example replicators from star trek can replicate nearly anything. This would put so many manufacturing jobs at risk and the loss of skills could be very alarming. Yet, once you have efficient replicator technology, manual crafting skills and tools become largely obsolete. Maybe traditional artisans can keep making hand-made goods as a novelty, but most of those jobs will never return and you better find a job programming the replicators instead.
Alfman,
You may be right. Eventually it boils down to whether this will be considered progress a century later, or not.
Alienation of the plebs and their replacement with the slaves carrying out the work for ever richer patrici is one of the reasons for the collapse of the Roman Empire, after all.
Still, you are probably right.
cevvalkoala,
I would argue they are not really experts. Or at least experts in using the AI tools.
Look at what happened when spreadsheets and word processors were first introduced. They were also “slowing people down” until people learned how to use the basics (even including keyboard and mouse).
The same will happen here. People will learn how to use the AI tools masterfully, and the tools themselves will also improve.
(They now integrate well into some IDEs taking over functions like major refactors, or code generation for example)
In my experience code prompts generated by AI have been mostly out of context. Prompt quality will probably improve overtime, but for now these prompts are more of annoyance.
As seen on other news, that study is nonsense.
AI is a tool. And the developer in the study are great developer, but have not much experience with the tool AI.
If you using a tool wrong, it slows you down. If you using it right, it makes you faster.
Most people who are faster with AI using the AI like a helping developer, which have much lesser experince then they theyself, but which are faster.
So the developer wants to create a program. They devide the planned program in little easy to create modules. And then letting the AI creating that modules. With different instances of the AI, so that every instance create a different module.
So that the developer is the maintainer of the AIs.
Or creating a team of AIs. One AI instance creating a part of a program, which is not good. But an other instance of the AI analysis the code. The next one fixing and improving it.
And instead of writing the command in the text field, using for example ChatGPT to speak over the Microsphone with the AI. To have the hands free for coding.
Then the AI males devoloping A LOT faster.
But you can not give the AI the source code of Linux or LibreOffice and say, that the AI should improving it.
>”And instead of writing the command in the text field, using for example ChatGPT to speak over the Microsphone with the AI. To have the hands free for coding.”
So while you are coding by hand, you are also giving prompts to AI verbally? You really expect that to speed you up somehow?
> You really expect that to speed you up somehow?
Oh, yes. Absolutely. This way people may implement the exact same features that yesterday were achievable in 100 lines of code in 10000 lines of code and do that only 3 or 5 times slower than before!
I’ve seen an apt comparison of what LLMs do to coding (and almost everything else): it allows one to produce more useless garbage.
As in: factory that produces bicycles (as in: features that people actually need) makes X bicycles per day and Y tons of garbage (lines of code are not something people may directly use, remember?)… with AI you may produce 10 x Y tons of garbage (but only X/2 bicycles)!
If you, then, can find some gullible guy who may PAY for that garbage – then your “productivity” can give you hefty pile of extra money… the only problem is that someones people actually ask why there are so few bicycles… but that’s where “social skills” become important…
Funny how studies that confirm certain biases get published here, but others do not.
I have found it has it’s uses – I don’t code much at work anymore, but I was a long time C developer and then Java (and a bunch of older languages we used over those years – TCL, PowerBuilder, ASPX (VB), etc.). We had a 25+ year old internal web app – ASP (the version that went EOL back in around 1999). I’m amazed it still runs on modern Windows Server with IIS, but it still does. One day soon I bet it won’t.
So I popped into VSCode with GitHub Copilot, had access to a few different models, and in a day I had a fully working React UI with a NodeJS backend. Added a ton of features the one didn’t have – for example in the old if the user wanted to see a log file, they clicked a link and it downloaded the file – I asked CoPilot to “make it so if the user clicks the log file link it comes up in a modal window using the Monaco editor. make it read-only, no minimap, but with line numbers. Add a button for them to download it as well”.. and in about 3 minutes that feature was done – it added the backend API for the UI to call to get the file, and made the UI, updated the README and documentation for me.
1 day and the app has every feature the existing one had, plus some new ones, and it’s in languages that I do not know.
Being a long time developer helped greatly though – I knew how to prompt it and could tell it was veering off course and cancel/undo the latest change and fix the prompt. I can’t see how in 5 years this doesn’t replace a LOT of developers.
It’s funny how EVERY SINGLE SUCCESS STORY about AI is “some kind of CRUD app” that is supposedly easy to make with AI.
Guys, CRUD apps were easy to make 30 years ago. Visual Basic and Delphi NAILED IT – and made it easy.
Then we regressed. As in: seriously regressed. Instead of nice visual editors we’ve got crude and hard-to-use XML, JSON and sometimes even raw JS-code things.
The fact that AI can return us back to where we were 30 years ago (even if in crazy inefficient way) is impressive… but that’s still well within the same framework: with AI it’s easy to do work that should have been easy without AI, but was made hard because of marketing.
The usability of AI outside of that niche is still unclear.
It is much more helpful than people realize.
Here is a sample I tried a while ago:
https://gemini.google.com/share/30c4952703e0
What is this? A simple assembler, and a “C lite” compiler for Action Replay opcodes.
The only thing I gave it was the opcode sheet (as an image!) and some prompts. Is it perfect? No, it has many issues (hex to decimal conversion for constants for example).
However it gives a very good baseline to start the project. The HTML single page app along side the main functionality (tabs, examples, compile button, etc) are wired correctly.
It has rudimentary parser and compiler ready.
The rest I can take over, and complete.
This is a massive boost to start projects especially when there is repetitive code (implementing all those opcodes).
I don’t know about you, but I have no trouble starting projects. It’s finishing them that’s hard.
But yes, it’s true that AI is not ENTIRELY useless… the big question is: what may it actually improve?
It looks to me that it can do something that YOU ARE NOT AN EXPERT IN better than you (and if you have nothing where you are an expert then it may replace you).
The end result: it makes the current situation on a job market (bazillion contenders for a few job positions yet almost impossible to hire anyone because no one knows anything well) EVEN WORSE!
zde,
This was just an example. It can also help iterate over it (but I would not give full responsibility). For example, AI is extremely good on larger refactors, finding subtle bugs, writing documentation, helping figure out a “foreign” code.
It is also very useful in code review processes. Not the “cheap” ones you see outside, but large companies have their own in house models that make sure code adhere to their own internal standards, and subtle bugs are discovered wrt. common design patterns.
Basically a coding assistant on steroids, replacing the older hand coded “intellisense” tools.
The same discussion basically we had when IDEs were first introduced (“we don’t need that I can do everything with vim and gdb”)
> The same discussion basically we had when IDEs were first introduced
Precisely. And I’m talking that as the most prolific member of my team at my $DAY_JOB who is NOT using IDEs.
So it’s more of the same nonthing burger, but MUCH more resource-hungry, this time.
zde,
More power to you, It’s nobody’s place to tell you what’s best for you. I often use a simple text editor over SSH as well. However I don’t think there’s any sense in denying that IDE code aware features can save a ton of time even for professional coders. Autochecking and autocomplete don’t provide anything I couldn’t have done otherwise, it just takes longer to do manually. Context aware rename and refactor tools save so much time and are much more reliable than using context unaware find/replace tools.
I really do respect your own choices for yourself. However where I disagree with you is when you try to sell the idea that IDEs and AIs are or should be a “nothing burger” to everyone else.
zde,
I believe that AI programming skills are going to keep improving over time. The limiting factor today may be training data. At best it mimicks the human input it gets trained on. Naturally this sets an upper limit on skill, however this is only the beginning and not the end. AI trained using adversarial methods becomes iteratively better at harder problems. It’s a time consuming process but the results have created artificial NNs that the skeptics could no longer deny the merits of when AI started beating the best human strategies at tasks like go, chess, skycraft, etc. I predict programming will go the same way and the skeptics will be proven wrong again. Maybe a decade of competition before even the best humans have to cede the “game”.
Even before this happens though, I don’t think we should be dismissing how much of the work is tedious repetition of existing patterns. Most programming isn’t innovative, leaving lots of opportunities for even a dumb copy-cat AI to save time without inventing anything new. This is what most employers will be looking for AI to do in the next few years – save developers from wasting too much time on low level tasks – and yes programming is filled with such tasks.
I can respect your opinion that maybe the world would be better without AI, however I do worry that things may not turn out so well for the Luddites who are dead set on rejecting AI because it is not the practitioners like us who will choose the fate of AI. It’s going to be corporations who decide where their money is spent.
Alfman,
Yes, I would say most programming is “copy-pasta engineering”.
Devs used to Google for their tasks find a solution on Stack Exchange (or ask one there), copy that into their codebase. And since obviously it won’t work, ask follow up questions until it gets into a working stage.
The AI can do this much more efficiently, and it can also recognize the context. In other words the code usually works on the first try, and it can even answer follow up questions.
For the “copy-pasta” engineers this is actually an improvement (Stack Overflow became hostile to “newbie” questions over time, and that is also a major reason their value dropped as soon as AI became viable)
Resumes, it is mazingly bad at it.