One of my first jobs at DEC was writing “design verification tests” for the Alpha (née EVAX) processor. These tests were assembly language programs that tested for correct CPU operation, and management had budgeted 6-8 weeks to develop a particularly complex set of tests. I developed them in 3 days, by writing a C program to automatically generate the range of tests.
I don’t tell that story to brag, but to talk about the “10x developer”. If you write code at all, you’ve surely heard of the idea there are programmers ten times as productive as others. The original notion came from “The Mythical Man-Month” by Fred Brooks, and the topic has been hotly debated ever since.
Occasionally, academics wade into the debate with data, including researchers at CMU’s Sofware Engineering Institute, who recently noted:
[…] while some programmers are better or faster than others, the scale and usefulness of this [10x] difference has been greatly exaggerated. Experience alone clearly is important, but its value is limited.https://insights.sei.cmu.edu/sei_blog/2020/01/programmer-moneyball-challenging-the-myth-of-individual-programmer-productivity.html
Their study measured “effort” (time to complete), lines of code, and defect count for ten programming assignments:
[..] each solution involved simple input, output, modularization, and the use of control and loop logic. Each was the size of one or two small Agile user stories requiring two or more hours to implement. The student recorded time on the major activities needed to complete a solution that passed all required tests, including planning, design, coding, testing, and personal reviews.
My anecdotal experience is consistent with this research: I don’t know anyone 10x faster than their peers in solving well-defined programming problems.
But this data highlights why the debate continues: highly productive developers (10x or otherwise) are problem-solving at a much higher level. Their productivity won’t be apparent from a narrow test, any more than the overall skill of a world-class football player would be apparent from (say) how fast they could run the 100m dash.
The most productive developers are solving big problems, and are bringing judgment and experience to bear on essential design, architecture, and “build vs use existing library” decisions. They’re designing key abstractions that will pay dividends through the life of the project. They’re finding massive shortcuts that aren’t in the CS textbook, such as Infocom’s use of a virtual machine to port their text adventure games to a range of early PC platforms. (Or, writing code to write CPU tests).
I think 10x developers, like world-class athletes, musicians, and authors, absolutely do exist. You’re just not going to find them with a coding test.
That’s a very reasonable comment. I wrote a wrote and maintained a lot of industrial code on the Alpha, so I appreciate your effort! And I don’t disagree at all with the notion that the real separation among programmers is their ability to solve progressively difficult problems.
People writing compiler optimization, for example, are different, in a good way. Most programmers couldn’t get that working at all. (infinite productivity variation?) The problem is a large body of work purports to show 10x production on ordinary tasks. I think we both agree, that just doesn’t hold up.
The PSP study addresses a narrow question that has been the subject of replicated studies about a specific (and in my opinion flawed) definition of productivity. The contribution of the PSP quasi-experiment is to show that the replicated experiments were wrong because they suffered from a common methodological weakness, lack of repeated measures.
Sackman and others showed a range of 10x productivity. The PSP data shows an even wider range for each program measured, (replicating old results) but also that individual programmers finished in very different orders on the different programs. Individual differences remain, but for ordinary tasks, the differences were not as wide as reported.
An important lesson is that measuring programmers by their speed completing tasks on an ordinary problem will not tell you as much as had been thought.
I prefer not to call the super programmer who solves the very difficult problem 10x because that just slaps a number associated with a very different measure, leading to some confusion over what was meant.
If I understand your meaning of 10x correctly, it is used much as biblical stories used “40 days and 40 nights” or the medieval collection of stories (1001 tales) was meant in the sense of “a really big number”, rather than as a precise quantitative assessment.
If you look at the original papers where the data for a 10x difference was first found, there are a ton of… issues.
I’ve covered this in detail here: https://www.youtube.com/watch?v=Gho89X72Xas
Could you write something to find them?