New Paradigms Need New Tools

and it takes some time

There have been something like 4 major paradigms in programming in the history of the computer industry: mainframe, PC, Internet, Mobile. AI is almost certainly the next one. Each new paradigm presents new problems and needs new engineering approaches - not just new tools, but new systems and ways of thinking.

Mainframes were big enough that you could start to write programs that were too hard to keep in one person’s head, and they were big enough to have more than one user, so we invented things like compilers, multi-tasking operating systems, and all of the tools that went with them. PCs were smaller and lighter - we didn’t need the same kind of user management (mostly) but we did need to manage fleets of machines, secure them, build more complex (and eventually graphical) applications, which needed things like application frameworks, automated testing, SQL databases and local file systems, etc.

The internet brought challenges around scale - suddenly we had to manage applications spread across many machines, with different kinds of UI surfaces, like the browser. This meant we had to learn how to do things like orchestration on the server (distributed systems), larger databases (NoSQL), app frameworks that combined the server and client side, and programming disciplines like CI/CD and automated testing. Mobile might be thought of as an extension of this paradigm, but it brought new things forward, like deeper network integration, always on telemetry, etc.

So now we are in the age of AI. What tools do we need here? Well, as app developers (not folks who create AI models, but who consume them), we need to perform many of the same tasks as before: build pipelines, QA and regression automation, load optimization and scheduling, user data management and security, remote telemetry. These all have to be managed in the context of something that is both stochastic and semantic - often the things we need to do telemetry or testing or scheduling on are in the semantic realm. For example, we might want to know if one of our prompts still produces an answer that is “nice” in a new generation of a base model, and we can’t do byte-wise comparisons because the models are stochastic. We might think of this as something like “Semantic Telemetry” - instead of monitoring latency, we need to monitor and manage something that is more in the domain of meaning. This becomes very important as we build more independent agents.

There are also new problems that emerge. We need to be able to decide where to run a given inference - we might not always want to run it remotely, or we might want to run it on a cheaper model. So we need some way to do “scheduling” where we can predict the quality of an inference on a given model, without actually running it first.

There will be challenges and new approaches to all of these problems. Sometimes the old tools will work fine (like compilers), or evolve (like interpreters). Sometimes we will really have to rethink the overall approach (like the shift from waterfall to agile that the internet both enabled and encouraged). There are lots of new problems to think about too - how do you prevent and defend against prompt injection attacks, for example? (This reminds me of us all learning to deal with SQL injection and XSS attacks).

This process won’t likely happen overnight. Each of the large shifts mentioned at the beginning of this article took years to work out properly. There will be false starts, religious technical wars, ideas that sounds good but are impractical, etc. It’s quite possible that the approaches that work best aren’t obvious at all yet, because no one has built a complex or ambitious enough product to need them.

The right thing to do is to be open to the change. There will be new approaches that feel wrong at first. It’s valuable to ground ourselves in what works, to solve specific problems, and to do as much of this as possible in the open and together, so we can all learn and debug in parallel. Don’t be tempted to write off early attempts because they aren’t perfect - this is “why not” thinking. It’s much better to be pragmatic and thing “what if” about new tools and approaches, and to solve problems rather than throw rocks at the people working on them.

The developers and companies who were most successful in the prior transitions were universally the ones who had this approach to the new paradigm, both its problems and potential.