Cranky Old Sam

Weirdness in Agent land

(a brief post - written before I left on vacation).

At some point, my engineering team at Microsoft decided that it would be cool if they had a “crusty old engineer” agent that reviewed their designs from a seasoned (if slightly jaded) systems engineering perspective. Not something that would find bugs per se, but something that would point out bad patterns and suggest better approaches. We built this for Amplifier, and it works quite well (and I encourage folks to find it in our repos and clone if they want).

Then one day I mentioned to one of the engineers, “you should build that, but for simplicity. I think y’all complicate things too much”. They said “you should do that”. So I did.

The process was, literally three prompts. One was “I want something like the COE but for simplicity. You can call it Cranky Old Sam”. Then it went off, did a bunch of research and came back with a design. Prompt 2 was me approving and answering a question about structure with “Make it just like COE, do whatever we did there”. Third prompt was “cool, make a private repo and share <another engineer on the team> to it”.

Then I forgot about it. Actually, forgot whether I had even finished it. A week later, a bunch of the team start to tell me that the COS is really great - partially from amusement value (I gather it’s a bit salty) but also that it’s useful. Just encoding that perspective, that simple is better, and to look through that particular lens, had value.

This was a really surprising bit of low hanging fruit for me. Particularly the absolutely minimal degree of effort. I can’t tell if that’s the model getting better, the harness getting better, or both (I suspect a bit of both), but it’s somewhat stunning that what would have been at best a lot of work for me to personally track down and review designs, think deeply about them, and then persuade engineers of my perspective…is just an automatic part of the environment now.

I don’t know what that means. I suspect more aspects of team and company culture will wind up getting baked into our agentic tools, of course. I think it will be hard to do this well - and the layers of metacognition, strategies for having strategies for thinking, and so on, will make it very complex. This might well be the kind of thing that turns into a secret sauce - teams with a particularly good set of interacting agents, principles and tools will just be more effective, and it might not always be obvious why.

Fascinating to see, though. So much of what we try to do is hard and fails at first. Having something like this work with so little effort that I forgot about it, is quite remarkable. Perhaps just a lucky bounce, or perhaps there are more of these out there…