Where is the autonomy in AI?

We do we get to take our hands off the wheel?

I’m not a deep ML expert - I understand the science pretty well and I read a lot of papers, but I’m more of a builder of applications than of models (I like to say I consume them, not create them). I’ve been working with LLM-based systems for about a year now and closely watching both my own and everyone else’s work. And I’m struck by something: no matter how capable LLM-based solutions are, there always has to be a “human in the loop”. That is, they can do great things, but only if we hold their hands in certain ways.

Some of this is obvious, and some isn’t. Prompting strategies like “chain of thought” are a subtle way that we are doing some of the work for the model - we are doing the “metacognition” of picking a cognitive strategy that is likely to suit the problem, first. Then the model “bakes the recipe” very well - but only once we’ve picked it.

Or for longer tasks, the models can do each step, but need us to do error correction, goal maintenance, and completeness evaluation. I can do something fixed, like the “textbook factory” experiment I built last winter that takes a sentence of input and uses about 200 lines of python and 5 prompts, and 600 calls to the model, to build out a full-year text book. That works well enough (mod hallucinations) but it’s only good for that one task, and I had to, again, do the metacognitive organization of writing the program.

Or I can walk something like ChatGPT through a complex process like writing a book or a program - but I have to do the walking. I have to prompt at each stage, correct it when it’s wrong, curate the answers, and decide what the next task is and when we are done. You can’t just say “write me a book” and then “mmm hmm…keep going” over and over and get a good result.

Think of a task like “write a clone of the vi text editor in Python from scratch”. You could give that to an intern or CS grad with a simple, closed code environment, and expect that they would spend some substantial time on it and get it at least most of the way done. We can easily imagine that “metacognitive” planning we’d do, too if we were that intern: start by organizing requirements, think about modules (editor, event handler, macros, testing), iterate and evaluate the design against the spec, recurse down to smaller and smaller pieces of each task (write module 1 - first step name it, second step define classes…), until we’re done, etc.

It seems hard to build that strategy into a model or LLM-based solution. I might be unaware (and I’m really interested in hearing if I am) but I don’t think anyone has even done something like the above (where I can give a long, complex arbitrary programming task that doesn’t fit into a single token context (or even 100 of them), and some system can complete it without intervention in some simple environment), much less a more complex system that can successfully navigate the real world for a complex, long running task, like “here are my data sources and last quarter’s results, recreate this set of artifacts with the current quarter’s results” or “look through our org chart and give me a detailed salary analysis relative to local markets, highlighting any more than one sigma outliers”. Or, especially, open-ended or underdefined problems where there are some questions that need to be asked (“all of the features? which version of vi?”) along the way.

I don’t know why this is. Some would argue the models aren’t smart enough yet. Some might say that we don’t yet have the right architecture on top of them (I fairly strongly suspect there is some kind of generic “recursive hierarchical problem-solving strategy” that we haven’t found, like people had to figure out B-trees and such way back in the day). It’s hard to describe metacognition, but not impossible, but maybe the errors from stochasticity compound as you go up the levels (which might make sense as the leverage increases) and there’s some subtle error correction approach we haven’t found yet. It’s definitely the case that chat isn’t the best UI metaphor, and we need better ways to have messy, branched, asynchronous iteration more akin to human workflows and conversations.

Setting aside the question of whether it’s a good idea to do this or not, it’s very striking to me that, after a year of more or less everyone trying very hard, we haven’t gotten very far with autonomy, even in “toy” environments like pure coding. I don’t know what it means, but it’s hard not to notice.