Code Goes First
A peek into the future
Usage of generative AI is uneven, and I still know lots of folks who don’t do anything much with it. Even a lot of coders are still skeptical or are only using it as an autocomplete tool, not really getting much done. But there are many more advanced coders who are getting really astonishing amounts of work done, using new mindsets and new tools. I want to argue that this isn’t an anomaly - it’s foreshadowing of how all knowledge work will look soon.
Why would code go first? Code is well suited to LLMs - it’s all text, it’s mostly online so there is a lot of it to train on, the structured nature of it makes it tokenize well, and coding tools are already very command line oriented, which means the plumbing to move code and code actions and artifacts around is already well developed. This last is possibly the most important part - code is just easier for an LLM to work with.
But that plumbing is being built out for other knowledge work. There are new protocols, like MCP, being implemented now that will allow any program to be connected in an LLM friendly way. As more work happens with them, there will be more training data, and more best practices. Work on code will inform this but there will also be new inventions in other fields. And finally “a group of people working in a code repository with tools and artifacts that are both primary and synthesized” is topologically identical to “a group of people working on a project in a directory with artifacts they author and generate”. The only real difference is the nature of the artifacts, and some nice things about code, like change management, that might cross over.
So if code goes first, what is going on with code right now?
With the best coders, the approach is something I think of as “recipes”: combinations of code, system prompt and normal prompting that are capable of targeted, long-term processes. Usually these recipes have “metacognitive” guidance - a fancy way for saying that these are strategies for the models to observe itself, iterate and correct (some) errors during these long processes. This means that very complex things are getting done regularly.
Here are some examples, actually which aren’t coding (showing that this is starting to bleed over). I have a researcher working with me who has now written two papers, each in less than 6 hours, each of which involved a software design, external research, running an experiment (one on multiple LLMs, one a survey using mechanical turk), creation of software as a result and the writing (and formatting in LaTEX) of a formal paper. This wasn’t one single prompt (“go do this for me”) but it was a series of fairly complex tasks given to the models in parallel, while other things were worked on.
Two other personal examples: I needed to write a new chapter for my book, and I wanted to update on the past year. So, I asked a long-running agent to read the book (9 seconds! That’ll boost the ol’ ego), read the last 50 of these letters, look for themes, pick 10 letters that represented those themes, and write a 1000-word header for the chapter, and then append the 10 letters. This ran successfully in the background in 20 minutes or so. Another (silly) example: I gave an agent a picture of a broken drawer in my refrigerator, and a picture of the serial number/model card and told it to go find a place I could buy a replacement. First try, no problem.
(There are many, many coding examples that I’m seeing as well, and often they are more impressive than those examples, but they’re harder to understand for a non-technical audience, and longer to explain anyway)
There is a quiet, radical revolution going on in the coding community. I believe it will start to spill over soon into other information work. The examples above use the same techniques the coders are using, but in the consumer realm. We are starting to understand best practices, and we are starting to have models that are now capable of performing them accurately. Right now, the tools are a bit too rough and obscure for non-coders to easily use, just like every new technical wave is at first. But that won’t last long.
Code goes first but it won’t be the last.
While very whizzy, and no doubt will get much better with time, it is important to remember that LLM-based AI does not understand computer languages and programming, but essentially extracts what has been produced in the past. These AIs do not understand the books on programming, nor the code they train on in GitHub, Stack Exchange, and numerous books, often obtained illegally. LLM AIs "stand on teh shoulders of others", but do not acknowledge them, nor pay them for their works.
In the US, copyrights have been extended for years. Copyright even prevented derivative works. Woe betide anyone making even a crude Mickey Mouse. How close to an original work is hotly determined in the courts. Even using some chords from a song can result in infringement. Academic authors who plagiarise even a line from another without attribution can feel the weight of academic dismissal.
Intellectual Property (IP) was so important to US corporations that it was used as a bludgeon in trade agreements. An employee moving from one tech company to another might not be able to fully use the knowledge gained inside another company.
And yet, somehow, this is all OK when it comes to behemoth technology companies that suck up all the content they can to train their models. So extensive has this been that concerns were raised about the lack of new training content and the need to fabricate it from prior works.
If LLMs were trained on freely available open source software or compensated creators of software by some model like Spotify's for playing musicians' songs, or streamers' of content, then that might be acceptable for commercial use. But this is not the case, as copyright lawsuits are still in the courts, and the AI corporations are squealing that they shouldn't have to pay for their use of content for training or usage in AI coding output, where the code source can be identified.
LLM-generated code is likely to be very vulnerable to malware from poisoned libraries to prompt injection. As code is increasingly created by LLMs, this problem will likely be increasingly visible. Whether LLM AIs will also play defence and spot vulnerable attack surfaces and injected malware remains to be seen. I hope the tools will eventually do this routinely, preventing widespread harm.
The bottom line is that LLMs exist by ignoring their theft of IP, the very IP that corporations insisted was important for their success and must not be infringed. Yes, it is impressive that mindless machines can quickly create code without understanding it, just making probabilistic copies of existing code, steered by prompts. It works so well with code because code is so structured. IDEs can easily detect code errors, something that cannot be done with creative writing much beyond spelling and grammar checking. It remains to be seen if our civilizational code base remains robust or whether it will result in a slow decline in the quality of our systems.
Interesting it is how coding is evolving, I still find, 'make an entire app with prompts' a bit too misleading, but make your app great with multiple thoughtful prompts & MCP connected knowledge sources lot more realistic & a great value addition for coding ecosystem!
We have been improving our systems at multiple ends leveraging LLM based deep reviews, & I still am surprised how quick it was to bring about these improvements into production the moment we realized the gaps existed!