We all know that it’s notoriously hard to measure developer productivity. Almost any concrete metric - lines of code, bugs fixed, pull request counts, etc - at best will tell an incomplete story, and at worst (usual case) will be gamed.
In fact, giving an organization a metric like this is a little bit like overfitting an ML model. It’s important to be very careful what you are “training” on, or you’ll get strange side-effects.
Usually when managers look at this kind of measurement, they ask what seems to be a reasonable question: “what’s the best metric for measuring some aspect of performance, that I can think of and implement quickly?” This is actually a very dangerous question, and likely the wrong one.
The first question you should ask about any metric you want to apply in public to your teams is: what’s the worst possible misuse of this that I can think of? Because, sadly, it’s highly likely that something approximating that misuse will at least be a side-effect. Count PRs? Expect to get a lot of them, and be utterly unable to distinguish signal from noise forever. Bugs fixed? Expect your bug db to explode with trivial bugs disguised to look as real issues, that will completely remove your ability to get a good read on quality. And so on.
The analogy to overfitting isn’t a bad one. Organizations do learn, and they do respond to inputs. So, it’s valuable to have check metrics, which function as test data, for example. It’s valuable to make sure the inputs are really reflective of what you want to train on (input “count PRs, train on ‘PR count is important’”).
In AI, individual nodes aren’t smart but a bunch of them together can be. In organizations it’s almost the opposite - individuals are smart, but the collective emergent behavior can be quite “dumb” in that it will take instructions very literally sometimes.
Developer productivity is something that will never fully resolve as a topic. But as you approach it in your teams, be careful to think about side-effects of the metrics you choose.
(I’ll be taking the rest of the year off - see you next year!)