The Skeptical Methodologist

Software, Rants and Philosophy

The 10x Myth

There’s a myth abound in software development circles, and it needs some deconstructing. It’s probably one of the best indicators of how much further the industry has to go in regards to sexism, since it’s a patently masculine myth, evoking images of great Greek Heroes slaughtering thousands of men as they move forward.

The Myth of the 10x Developer

Now, this isn’t a myth because it hasn’t been researched. There is ample amounts of research on programmer productivity, at least from the 80’s, and if it is still to be believed we should assume that there is at least some difference in  programming ability between developers.

The real myth comes from the interpretation of these results, and that’s where the testosterone-fueled neck-bearded bias comes in. There’s the results of these studies, and then there are how the studies are understood by so-called “Rock Star” developers who always assume they are one of the 10x’ers and that’s their justification for why they shouldn’t be forced to get along with anyone.


The results are thus:

There is roughly an order of magnitude difference in productivity (measured as time to get code working for a toy problem) between the best and the worst programmers, with causes unknown.

However, here’s how it’s commonly repeated. See if you can spot the difference:

There is roughly an order of magnitude difference in general productivity between the best and average programmers, and it’s due entirely to innate talent.

So, let’s take this apart one by one.

We’ve solved this problem before…

The first flaw is somewhat methodological, however I don’t think the researchers ever claimed that their toy problem measured generalized productivity, so it’s also a flaw in how the general population of brogrammers have read the result. Think of it this way, if I took a random sample of programmers and gave them a test, even if their skills were all roughly the same, what kind of result would I see? I’d see some programmers doing better than others, because they’ve solved that or a similar problem before. I can compare a person who’s never written an SMTP server to someone who has, ask them to do so, and witness a miraculous 10x or more productivity benefit to the programmer who has built it before. Imagine that!

This is similar to what you might call the halo effect. Rock stars are identified by their ability to solve their specialized problem very well – perhaps they’ve built a few Rails apps from start to finish. They’re going to be great at that. But throw them at writing a compiler and watch them flounder.

Distribution of wealth…

The second issue is the confusion of the average for the worst programmer. Let me give an example of why this might be an issue. If I reported to you that the best programmers are 10x better than the worst, that’s one bit of information, but not enough to really say anything about the average (i.e., the majority of programmers). If I then said that average programmers are 9x better than worst, now we know something about the average, and the distribution. Unfortunately, we have no idea which way the distribution of these results is skewed, at least as it’s commonly reported.

First, it’s outright false to say that the best developers are 10x better than the average, even though that’s often what’s reported. Second, we don’t know if the productivity difference at hand is due to the best being that much better than everyone else or the worst being that much worse than everyone else. The issue here is that due to all the manliness in our industry, we of course all assumed it must be the former, and not the latter. Because we’re all magically that 10x’er, and that is why everyone else is jerks.

We point to this myth over and over again to justify why we don’t get along with others. It’s nearly always used to justify mistreatment of our colleagues – they’re all idiots, I’m brilliant, and that’s why I shouldn’t change and they should. I’m a 10x programmer. But clearly, the best you can claim if the distribution has a fat tail to the right (i.e., the worst are much worse than the average) is that you’re only a little bit more productive than the average, and dogonit, you ought to pay attention more to what your colleagues say because they’re all nice people and would it kill you to shower?

Cause and Effect…

The last issue that the studies say nothing about, at least as they’re repeated in their mythological form, is why? There’s an implied why, an implied cause for this difference, but it’s hardly ever stated. These genius gods-among-men programmers are so productive because they wield the magic of Zeus. What they do cannot be replicated, repeated, or taught to anyone. They’re entirely packaged up, not able to be distributed or copied. If you want the 10x programmer, you have to accept his ego, his arrogance, his complete lack of communication or emotional skills, and his tendency to shit all over everyone else. And yes, it’s almost always a he.

The issue here is that we have no idea what makes the 10x best programmers more productive than the worst programmers. Sheer numbers of years of experience don’t seem to play a role, but is that because there’s far too many enterprises where we can disappear and never have to code again? A year at a large enterprise curating UML documents is not the same as a year getting your own Rails site up for customers to use. What tools or techniques did these 10x programmers use that the worst ones didn’t? Were they more skilled with the debugger? Did they adopt more structured coding conventions (this was in the era of structured code)? Did they test their code any different?

There are many questions we can ask, and non of the productivity myths answer. It is almost always supposed to be magic, always supposed to be something innate to the neck beard itself that grants the brogrammer who dons it the powers to develop and deploy only the best code, and disregard everyone else’s opinion who may have something to add (or learn!).


There are 10x programmers – or at least there were, in the 80’s – who were ten times, roughly, more productive than the worst programmers. But given how bad some programmers can be, it’s probably safer to say that you should ensure you don’t hire (or at least train) the worst of your crew rather than try to always hire the best. Average programmers are, on average, pretty good in my experience. That informs me that the tail is fat to the right, not to the left. Moreover, invest in methods and tools that are shown to increase productivity: iterative methods, testing and peer review, static analysis tools, and training in your methods of source control and deployment. It isn’t magic – there is a way to turn average to great, and we can figure out what that way is if we use the methods of SCIENCE!

Finally, don’t fall for this machoismo myth that there are the great men and then there is everyone else. It keeps far too many ‘good’ average developers who don’t fit our implied mold of the great programmer – apparently an asshole white guy – out of organizations that sorely need them.

August 26, 2015 Posted by | Uncategorized | Leave a comment

When Counting from 100 to 1, Interview Candidates will do Precisely as Well as You Think They Should

Can you write a program that prints 100 to 1? Apparently, some are claiming such a program can be as valuable as Fizz Buzz in determining the value of interview candidates. Some people can’t solve this incredibly simple problem…

Wait, bait and switch time. I only told you about the easy part of the problem, not the hard part – now that you’ve already clicked on my article, I’ll go ahead and fill you in the ‘tricky’ constraint that any solution you have must start with:

for(int i = 0; …

This isn’t a programming challenge, it is now a brain teaser. Why? Because you’ve taken away the obvious answer and for no really good reason added an additional constraint. Brain teasers aren’t bad, they’re just tests of insight, not expertise. And insight is notoriously difficult to reach when you’re under pressure in an interview.

The main issue I have with this line of thinking isn’t the reemergence of brain teasers, it’s the author’s implication that programmers need to knuckle down on the hard practice of programming and put their egos aside. It seems far more likely the case that the author needs to knuckle down on the hard practice of Industrial Psychology and put his or her ego (I couldn’t gender check since the page was failing to load due to traffic) aside.

Despite the warnings that 22 is not a large enough sample size to get any significant result out of, the author goes ahead and does it anyway. If the rest of their book is written with such rigor and you’re interested, I advise you to buy my own book I put together in a few weeks after learning the graph function in Excel.

But the ‘hard’ statistics isn’t even the worst part of drawing conclusions from this ‘study’ – the ‘soft’ part is where the author utterly failed.

One data point that I obtained for the book (but didn’t quite include in the book because it was too programmer centric) was based on 22 job interviews for programming positions I conducted for one of my clients over a period of two months.

The author claims two questions were asked to test the hypothesis of whether or not what they very scientifically call ‘whining’ can predict what they’re claiming to be programming ability. Did you see the flaw?

Unless I’m reading the blog wrong – and I could be – the author him or herself asked both questions, with hypothesis in mind, most likely in the order implied: whining then programming ability. This removes what, you know, experts in statistics and survey design would call ‘blinding’. It means the author’s own implicit bias going into each interview could possibly skew the result. To sum up, the author could very well have badgered every candidate during the programming test that whined more than a few minutes or they could have stayed silent. With the study designed as it was, we wouldn’t know the difference.

What’s a much better conclusion from this statistically insignificant result? Candidates are going to do precisely as well as you think they ought to. Specifically, they’ll do exactly as well as you want them to on brain teaser type problems that require insight. This is why you need structured, repeatable tests that measure insofar as possible expertise, not insight. Insight is important, but practically impossible to measure under the pressure of an interview when the candidate is going to be analyzing your every subconscious twitch to see whether they’re getting the job or not.

April 10, 2015 Posted by | Uncategorized | Leave a comment

Engagement Versus Alignment

Engagement, or how interested employees are in their company, and alignment, or how well the efforts of disparate employees and teams are aligned with a single goal, are both important cultural metrics to track.

An unengaged work force is less productive, has higher turnover, and is a much less fun place to work. A misaligned work force can quickly become disengaged as it is at higher risk of infighting, wasted efforts, and missed opportunities for team synergies.

“Don’t work on that, work on this.”

An issue arises, though, when decisions can increase engagement at the cost of alignment, or visa versa. For instance, a micro managing leader may second guess their subordinates ideas or work. This is an attempt to improve alignment in some cases, as the lead does not necessarily agree that the work is in alignment with the teams goals.

Obviously an employee is going to be more engaged when she’s working on her idea and she believes her idea will help the company. So this micromanagement may have increased alignment but at the cost of engagement. In some cases towards overall organizational productivity, this may end up being a wash, as the rise in team productivity due to increased alignment may come at the cost of individual productivity due to lowered engagement.

But if we take a step back, there’s a third idea we’re not taking into consideration, and that is correctness. That is to say, alignment is the measure of how well the team is focused on the same goal. Correctness is some measure of how well that tactical goal achieves the overall strategic goal of sustainable profits for the company.

In many cases of micromanagement, the lead believes he better understands what’s wrong and what needs to be fixed. He believes he better understands the correct course of action, and thus the job becomes getting his subordinates to focus on that course of action (alignment) with all their potential (engagement). But what if this assumption is false?

Emergent Correctness

In the creative economy, knowledge often is much more highly distributed among the company than in more top-down organizations. At the assembly line, the foremen often has much more experience and often more education than the front line worker, thus the foreman supervising the front line worker in terms of correctness makes sense. But in the software startup, the front line worker often knows more about how any particular piece is architected, what new technologies solve lingering problems, and what problems they actually face. The foreman or tech leads role is to focus instead on the two remaining metrics, engagement and alignment. The correct goal emerges from a organization that is highly engageged and highly designed.

What are birds? We just don’t know

For example, take a flock of birds. No one bird, or set of birds, is in control of the flock. The flock itself, though, looks incredibly organized – both aligned and engaged. The flock behaves correctly, in this case, it flies south for the winter or moves towards food, due to the shared burdens on all birds in the flock.

Management’s duty in this emergently correct culture then becomes ensuring that lines of communication between each front line worker are open, to help ideas become shared and implicitly voted on by what interests people more. This might include removing organizational barriers such as one lower level employee not feeling comfortable talking to a higher level one, or emotional barriers if employees don’t naturally get along, or political barriers if employees start removing lines of communication to protect their own feifdoms.

Their duties also are to increase forms of engagement important in these emergently correct cultures – engagement in the company as a team, identification with the company as a team, and excitement about the future. This includes letting ideas that might just be more interesting than immidiately applicable fly for awhile, since the costs of shutting them down early are just too great.

Emergence isn’t perfect

The main argument against emergent correctness of decisions is that it is rarely perfectly correct. Often, indeed, in hindsight we can see exactly where the company made mistakes. We think this becomes, in turn, an argument for stronger hierarchical control. Indeed, this appears to be why over time and with size, companies become more and more hierarchical. Turning over control to the experts on the front always sounds good in theory, but we know they will make mistakes. We just don’t know, in advance, what those mistakes will be yet. It is said that it is often better to fail traditionally than to succeed nontraditionally. This tendency drives control freaks to argue – not only with each failure, but with each success – that they again be given more hierarchical control to ‘prevent the mistakes’ we just made.

This is an organizational fallacy. As we never seem to consider the opportunity cost of increased hierarchical control. In emergent organizations, the trade off between alignment and engagement never occurs. The addition of hierarchical control is the de facto addition of this trade off. It is, quite literally, the argument (using perfect hindsight information as evidence) that if we had given up engagement in some key areas and gained alignment, we would have been done faster or with higher quality. The core fallacy here is that you can never see those opportunities in front of you, only behind you. So it is never worth looking at those trade offs.

Engagement and alignment are both important for an organization. In many cases, it appears there is a trade off to improve overall correctness of our actions. But this is almost always a fools errand – we can only identify these actions in hindsight. Emergent control isn’t perfect, but this can’t be an argument for inferior forms of control in an organization where the front line, on average, really does know what is best.

January 10, 2015 Posted by | Uncategorized | Leave a comment

Ambiguity and Leadership

Today’s leaders are expected to deal with ambiguous situations.  We’ve all heard that, right? In a way, it’s trivially obvious but sounds like something you should jot down from your 10 day MBA book. It’s not like a CEO or Entrepreneur ever has their job responsibilities really nailed down for them. They are expected to define the position. Moreover, what’s the bit about ‘today’? Was it ever the case that leadership wasn’t ambiguous? Since when did we ever have things really nailed down for us?

This tautological line, however, is being used increasingly not so much as unhelpful advice to actual leaders, but instead as a cliche response from middle managers explaining away their inadequacy. Think about it – since when does a subordinate ever respond to their lead with “I understand the work I’ve done for you is ambiguous in value, but today’s leaders are expected to deal with ambiguous situations.” That is career suicide. Instead, the statement is more often used in the reverse – when a leader fails his subordinate, he might punt to the ambiguity of the situation. Indeed, some leaders may bath themselves in ambiguity, using ambiguous milestones such that they can claim progress where none has been made, or ambiguous job requirements so they can criticize a subordinate for not living up to ‘expectations’ when they’re having a bad day.

I claim we should be taking a different tact with this statement. Indeed, when a leader punts to a subordinate justifying herself with ambiguity, shouldn’t we ask the question – who’s responsibility is it to deal with that ambiguity?

Why, today’s leaders, of course.

When a leader cloaks their milestones, expectations or other requirements in ambiguity, they are literally not doing their jobs. They are not dealing with ambiguity, and instead having their subordinates deal with ambiguity. They have left their subordinates holding the bag when it comes to who to blame when ambiguous situations get out of hand, even though it’s their job to resolve ambiguity insofar as possible. This often seems like a political move – ambiguity’s metaphorical cloak becomes a bit more literal when it comes to masking intentions and actions to potential political rivals. Some smoke filled room horse trading is always to be expected – we can’t be naive. But allowing this ambiguity to leak through to your team is failing to do your job.

Misaligned strategies, the 8 different bosses situation, or even allowing the innate ambiguity of the actual markets to leak through are all examples of the sources of ambiguity. But they are all examples of horribly failed organizations or leadership, and it’d be unjust to instead claim that they are just what ‘we have to deal with today’.

Good leaders should seek out ambiguity and banish it insofar as possible. Are you not sure whether we need to satisfy customer A or B? Do as best analysis as you can in the due amount of time and just go with it. Don’t punt to someone else. Make a fucking decision, for Christ’s sake. It’s what you are paid to do. Good leaders unite their team behind a single vision even though the real world is very ambiguous. This is how you show up on game day and dominate because every other team still doesn’t even have a plan. Good leaders inspire their teams by removing the uncomfortable ambiguity of business – “I know both customer A and B are important. But this team is going to focus on A, you’re bonus will be tied to A, and your promotion will be tied to A.” This is how you motivate people, by giving them clear requirements even though you may yourself have not received any. If you give a subordinate any inkling that they should somehow do both customer A and B even though there’s only time for one, you’re going to get half assed work and a lot of hours worked for nothing. You will have failed to do your job to properly direct your subordinates, and you may be able to horse trade awhile long clouded in the mystique of ambiguity, but you will forever be an imposter and eventually, they will find out.

September 21, 2014 Posted by | Uncategorized | Leave a comment

This Lifehacker Diet Article is a Wolf in Sheep’s Clothing

“Hey there, did you know everyone else is lying to you about diet? It’s all due to their corrupt corporate interests! I have no agenda, and will explain to you why everyone else is lying. Also, avoid processed food, as everyone knows that’s bad for you.”

Before we get started, there are a lot of great gems in the article. It does go into why diet is so confusing. But unfortunately, it breaks its own rule. It fails to cite evidence for its claims, and even misuses the word ‘natural’ in the middle of an article dismissing the word as meaningless.

Let’s dive into a few assumptions here. First, they are right in stating that there’s a lot of mixed messages out in diet. I’d come away saying that we actually know little beyond people who eat a lot of fruits and vegetables seem to be healthier. But we don’t necessarily even know why since there’s so many confounding variables there.

But this ‘corporate interest’ bullshit that should be the first sign that it’s not one person selling you truth and another falsehood, its just two bullshit vendors providing different varieties. Corporations are not evil. Repeat that with me. Corporations are almost completely amoral, in fact. They, for the most part, exist to maximize profits assuming human management with bounded rationality is making the day to day decisions. What does that mean? Does it mean Corporations are trying to get you fat, to die early, to lie to you? No, it means that established corporations biggest motivator is to protect the status quo. The status quo makes them money.

Let’s think about that for a bit. Does that mean that General Mills might try to convince you Fruity Pebbles is completely healthy for you? Yes, it does. They currently sell Fruity Pebbles, its in their interest to try and convince you to keep buying it. But what if they sold Grandpa John’s brand dried vegetable slaw (now with more wholesome!)? Well then they’d try to convince you that vegetables are healthy for you too! They aren’t trying to sell you packaged shit, they just happen to already sell packaged shit, so they want you to keep buying.

This same logic works for so-called ‘natural’ foods, which appears to just be foods closer to their original state. Organic farmers can charge nice hefty margins because they’ve convinced you that organic is healthier, even though there’s little evidence for that. Basically, everyone who sells something is motivated to keep you buying. It’s really not that hard to figure out, and it doesn’t make them evil. If science finds the best diet, just wait twenty years and these same evil corporate interests will have built up supply chains, warehouses, and marketing campaigns to make sure you buy that best diet, whatever it is. But until then, expect them to come kicking and screaming. That’s just their nature.

The second assumption they cite without evidence is this ‘processed food is bad for you’ clap trap. First of all, what the hell is processed food? Apparently Chetoes. They really seem to hate Chetoes. But beer, cheese, bread, pickles, deli meats, milk and yogurt are all processed too. Fermentation, cooking, pickling, and the like. Those are all processes. Canned vegetables, frozen berries, these are processed too. Are all these things bad for you? Not at all. You can tell the author’s motivations when they compare so called ‘processing’ to cooking at home, which is just another process. Apparently, processing is okay if you do it yourself. Do you see the motivations and assumptions too?

I’ll spell it out: eat food rich people eat, and avoid food poor people eat. Cooking at home is a luxury someone who isn’t working two jobs can do, it must be healthier for you. This sort of unprocessed push is just more upper class privilege asking why the poor colored masses aren’t living as long. Clue: it’s because they are poor, not because they aren’t eating what you eat. Just like the obesity epidemic was more or less invented so we can continue to hate blacks and latinos, the rampage against processed food is another way we can all feel better about making fun of that lower class family picking up some more hamburgers from McDonalds and blame them for their own misfortune.

Does this mean Fruity Pebbles is healthy? I have no idea. Fortified cereals have been shown to lower a nationwide deficiency of folic acid, so they’ve got that going for them. But the more important thing to point out is that there are many processed foods that are bonafied healthy – tomato paste being my favorite example as having a more bioavailable form of lycopene than whole tomatoes. Using simple rules like “unprocessed” is a great way to eliminate more than half the population from your recommendation since they can’t afford to eat that way anyway and eliminate a whole lot of good pathways to nutrition.

I’d say this unprocessed fad is no better than all the others, and it really gets to me that someone would go through such trouble to point out why everyone else is lying to you while blatantly doing the same to you themselves. It’s the kind of moral superiority that upper class, out of touch, privilege gets them, I suppose.

May 7, 2014 Posted by | Uncategorized | Leave a comment


Get every new post delivered to your Inbox.