The Skeptical Methodologist

Software, Rants and Philosophy

When Counting from 100 to 1, Interview Candidates will do Precisely as Well as You Think They Should

Can you write a program that prints 100 to 1? Apparently, some are claiming such a program can be as valuable as Fizz Buzz in determining the value of interview candidates. Some people can’t solve this incredibly simple problem…

Wait, bait and switch time. I only told you about the easy part of the problem, not the hard part – now that you’ve already clicked on my article, I’ll go ahead and fill you in the ‘tricky’ constraint that any solution you have must start with:

for(int i = 0; …

This isn’t a programming challenge, it is now a brain teaser. Why? Because you’ve taken away the obvious answer and for no really good reason added an additional constraint. Brain teasers aren’t bad, they’re just tests of insight, not expertise. And insight is notoriously difficult to reach when you’re under pressure in an interview.

The main issue I have with this line of thinking isn’t the reemergence of brain teasers, it’s the author’s implication that programmers need to knuckle down on the hard practice of programming and put their egos aside. It seems far more likely the case that the author needs to knuckle down on the hard practice of Industrial Psychology and put his or her ego (I couldn’t gender check since the page was failing to load due to traffic) aside.

Despite the warnings that 22 is not a large enough sample size to get any significant result out of, the author goes ahead and does it anyway. If the rest of their book is written with such rigor and you’re interested, I advise you to buy my own book I put together in a few weeks after learning the graph function in Excel.

But the ‘hard’ statistics isn’t even the worst part of drawing conclusions from this ‘study’ – the ‘soft’ part is where the author utterly failed.

One data point that I obtained for the book (but didn’t quite include in the book because it was too programmer centric) was based on 22 job interviews for programming positions I conducted for one of my clients over a period of two months.

The author claims two questions were asked to test the hypothesis of whether or not what they very scientifically call ‘whining’ can predict what they’re claiming to be programming ability. Did you see the flaw?

Unless I’m reading the blog wrong – and I could be – the author him or herself asked both questions, with hypothesis in mind, most likely in the order implied: whining then programming ability. This removes what, you know, experts in statistics and survey design would call ‘blinding’. It means the author’s own implicit bias going into each interview could possibly skew the result. To sum up, the author could very well have badgered every candidate during the programming test that whined more than a few minutes or they could have stayed silent. With the study designed as it was, we wouldn’t know the difference.

What’s a much better conclusion from this statistically insignificant result? Candidates are going to do precisely as well as you think they ought to. Specifically, they’ll do exactly as well as you want them to on brain teaser type problems that require insight. This is why you need structured, repeatable tests that measure insofar as possible expertise, not insight. Insight is important, but practically impossible to measure under the pressure of an interview when the candidate is going to be analyzing your every subconscious twitch to see whether they’re getting the job or not.

April 10, 2015 Posted by | Uncategorized | Leave a comment

Engagement Versus Alignment

Engagement, or how interested employees are in their company, and alignment, or how well the efforts of disparate employees and teams are aligned with a single goal, are both important cultural metrics to track.

An unengaged work force is less productive, has higher turnover, and is a much less fun place to work. A misaligned work force can quickly become disengaged as it is at higher risk of infighting, wasted efforts, and missed opportunities for team synergies.

“Don’t work on that, work on this.”

An issue arises, though, when decisions can increase engagement at the cost of alignment, or visa versa. For instance, a micro managing leader may second guess their subordinates ideas or work. This is an attempt to improve alignment in some cases, as the lead does not necessarily agree that the work is in alignment with the teams goals.

Obviously an employee is going to be more engaged when she’s working on her idea and she believes her idea will help the company. So this micromanagement may have increased alignment but at the cost of engagement. In some cases towards overall organizational productivity, this may end up being a wash, as the rise in team productivity due to increased alignment may come at the cost of individual productivity due to lowered engagement.

But if we take a step back, there’s a third idea we’re not taking into consideration, and that is correctness. That is to say, alignment is the measure of how well the team is focused on the same goal. Correctness is some measure of how well that tactical goal achieves the overall strategic goal of sustainable profits for the company.

In many cases of micromanagement, the lead believes he better understands what’s wrong and what needs to be fixed. He believes he better understands the correct course of action, and thus the job becomes getting his subordinates to focus on that course of action (alignment) with all their potential (engagement). But what if this assumption is false?

Emergent Correctness

In the creative economy, knowledge often is much more highly distributed among the company than in more top-down organizations. At the assembly line, the foremen often has much more experience and often more education than the front line worker, thus the foreman supervising the front line worker in terms of correctness makes sense. But in the software startup, the front line worker often knows more about how any particular piece is architected, what new technologies solve lingering problems, and what problems they actually face. The foreman or tech leads role is to focus instead on the two remaining metrics, engagement and alignment. The correct goal emerges from a organization that is highly engageged and highly designed.

What are birds? We just don’t know

For example, take a flock of birds. No one bird, or set of birds, is in control of the flock. The flock itself, though, looks incredibly organized – both aligned and engaged. The flock behaves correctly, in this case, it flies south for the winter or moves towards food, due to the shared burdens on all birds in the flock.

Management’s duty in this emergently correct culture then becomes ensuring that lines of communication between each front line worker are open, to help ideas become shared and implicitly voted on by what interests people more. This might include removing organizational barriers such as one lower level employee not feeling comfortable talking to a higher level one, or emotional barriers if employees don’t naturally get along, or political barriers if employees start removing lines of communication to protect their own feifdoms.

Their duties also are to increase forms of engagement important in these emergently correct cultures – engagement in the company as a team, identification with the company as a team, and excitement about the future. This includes letting ideas that might just be more interesting than immidiately applicable fly for awhile, since the costs of shutting them down early are just too great.

Emergence isn’t perfect

The main argument against emergent correctness of decisions is that it is rarely perfectly correct. Often, indeed, in hindsight we can see exactly where the company made mistakes. We think this becomes, in turn, an argument for stronger hierarchical control. Indeed, this appears to be why over time and with size, companies become more and more hierarchical. Turning over control to the experts on the front always sounds good in theory, but we know they will make mistakes. We just don’t know, in advance, what those mistakes will be yet. It is said that it is often better to fail traditionally than to succeed nontraditionally. This tendency drives control freaks to argue – not only with each failure, but with each success – that they again be given more hierarchical control to ‘prevent the mistakes’ we just made.

This is an organizational fallacy. As we never seem to consider the opportunity cost of increased hierarchical control. In emergent organizations, the trade off between alignment and engagement never occurs. The addition of hierarchical control is the de facto addition of this trade off. It is, quite literally, the argument (using perfect hindsight information as evidence) that if we had given up engagement in some key areas and gained alignment, we would have been done faster or with higher quality. The core fallacy here is that you can never see those opportunities in front of you, only behind you. So it is never worth looking at those trade offs.

Engagement and alignment are both important for an organization. In many cases, it appears there is a trade off to improve overall correctness of our actions. But this is almost always a fools errand – we can only identify these actions in hindsight. Emergent control isn’t perfect, but this can’t be an argument for inferior forms of control in an organization where the front line, on average, really does know what is best.

January 10, 2015 Posted by | Uncategorized | Leave a comment

Ambiguity and Leadership

Today’s leaders are expected to deal with ambiguous situations.  We’ve all heard that, right? In a way, it’s trivially obvious but sounds like something you should jot down from your 10 day MBA book. It’s not like a CEO or Entrepreneur ever has their job responsibilities really nailed down for them. They are expected to define the position. Moreover, what’s the bit about ‘today’? Was it ever the case that leadership wasn’t ambiguous? Since when did we ever have things really nailed down for us?

This tautological line, however, is being used increasingly not so much as unhelpful advice to actual leaders, but instead as a cliche response from middle managers explaining away their inadequacy. Think about it – since when does a subordinate ever respond to their lead with “I understand the work I’ve done for you is ambiguous in value, but today’s leaders are expected to deal with ambiguous situations.” That is career suicide. Instead, the statement is more often used in the reverse – when a leader fails his subordinate, he might punt to the ambiguity of the situation. Indeed, some leaders may bath themselves in ambiguity, using ambiguous milestones such that they can claim progress where none has been made, or ambiguous job requirements so they can criticize a subordinate for not living up to ‘expectations’ when they’re having a bad day.

I claim we should be taking a different tact with this statement. Indeed, when a leader punts to a subordinate justifying herself with ambiguity, shouldn’t we ask the question – who’s responsibility is it to deal with that ambiguity?

Why, today’s leaders, of course.

When a leader cloaks their milestones, expectations or other requirements in ambiguity, they are literally not doing their jobs. They are not dealing with ambiguity, and instead having their subordinates deal with ambiguity. They have left their subordinates holding the bag when it comes to who to blame when ambiguous situations get out of hand, even though it’s their job to resolve ambiguity insofar as possible. This often seems like a political move – ambiguity’s metaphorical cloak becomes a bit more literal when it comes to masking intentions and actions to potential political rivals. Some smoke filled room horse trading is always to be expected – we can’t be naive. But allowing this ambiguity to leak through to your team is failing to do your job.

Misaligned strategies, the 8 different bosses situation, or even allowing the innate ambiguity of the actual markets to leak through are all examples of the sources of ambiguity. But they are all examples of horribly failed organizations or leadership, and it’d be unjust to instead claim that they are just what ‘we have to deal with today’.

Good leaders should seek out ambiguity and banish it insofar as possible. Are you not sure whether we need to satisfy customer A or B? Do as best analysis as you can in the due amount of time and just go with it. Don’t punt to someone else. Make a fucking decision, for Christ’s sake. It’s what you are paid to do. Good leaders unite their team behind a single vision even though the real world is very ambiguous. This is how you show up on game day and dominate because every other team still doesn’t even have a plan. Good leaders inspire their teams by removing the uncomfortable ambiguity of business – “I know both customer A and B are important. But this team is going to focus on A, you’re bonus will be tied to A, and your promotion will be tied to A.” This is how you motivate people, by giving them clear requirements even though you may yourself have not received any. If you give a subordinate any inkling that they should somehow do both customer A and B even though there’s only time for one, you’re going to get half assed work and a lot of hours worked for nothing. You will have failed to do your job to properly direct your subordinates, and you may be able to horse trade awhile long clouded in the mystique of ambiguity, but you will forever be an imposter and eventually, they will find out.

September 21, 2014 Posted by | Uncategorized | Leave a comment

This Lifehacker Diet Article is a Wolf in Sheep’s Clothing

“Hey there, did you know everyone else is lying to you about diet? It’s all due to their corrupt corporate interests! I have no agenda, and will explain to you why everyone else is lying. Also, avoid processed food, as everyone knows that’s bad for you.”

Before we get started, there are a lot of great gems in the article. It does go into why diet is so confusing. But unfortunately, it breaks its own rule. It fails to cite evidence for its claims, and even misuses the word ‘natural’ in the middle of an article dismissing the word as meaningless.

Let’s dive into a few assumptions here. First, they are right in stating that there’s a lot of mixed messages out in diet. I’d come away saying that we actually know little beyond people who eat a lot of fruits and vegetables seem to be healthier. But we don’t necessarily even know why since there’s so many confounding variables there.

But this ‘corporate interest’ bullshit that should be the first sign that it’s not one person selling you truth and another falsehood, its just two bullshit vendors providing different varieties. Corporations are not evil. Repeat that with me. Corporations are almost completely amoral, in fact. They, for the most part, exist to maximize profits assuming human management with bounded rationality is making the day to day decisions. What does that mean? Does it mean Corporations are trying to get you fat, to die early, to lie to you? No, it means that established corporations biggest motivator is to protect the status quo. The status quo makes them money.

Let’s think about that for a bit. Does that mean that General Mills might try to convince you Fruity Pebbles is completely healthy for you? Yes, it does. They currently sell Fruity Pebbles, its in their interest to try and convince you to keep buying it. But what if they sold Grandpa John’s brand dried vegetable slaw (now with more wholesome!)? Well then they’d try to convince you that vegetables are healthy for you too! They aren’t trying to sell you packaged shit, they just happen to already sell packaged shit, so they want you to keep buying.

This same logic works for so-called ‘natural’ foods, which appears to just be foods closer to their original state. Organic farmers can charge nice hefty margins because they’ve convinced you that organic is healthier, even though there’s little evidence for that. Basically, everyone who sells something is motivated to keep you buying. It’s really not that hard to figure out, and it doesn’t make them evil. If science finds the best diet, just wait twenty years and these same evil corporate interests will have built up supply chains, warehouses, and marketing campaigns to make sure you buy that best diet, whatever it is. But until then, expect them to come kicking and screaming. That’s just their nature.

The second assumption they cite without evidence is this ‘processed food is bad for you’ clap trap. First of all, what the hell is processed food? Apparently Chetoes. They really seem to hate Chetoes. But beer, cheese, bread, pickles, deli meats, milk and yogurt are all processed too. Fermentation, cooking, pickling, and the like. Those are all processes. Canned vegetables, frozen berries, these are processed too. Are all these things bad for you? Not at all. You can tell the author’s motivations when they compare so called ‘processing’ to cooking at home, which is just another process. Apparently, processing is okay if you do it yourself. Do you see the motivations and assumptions too?

I’ll spell it out: eat food rich people eat, and avoid food poor people eat. Cooking at home is a luxury someone who isn’t working two jobs can do, it must be healthier for you. This sort of unprocessed push is just more upper class privilege asking why the poor colored masses aren’t living as long. Clue: it’s because they are poor, not because they aren’t eating what you eat. Just like the obesity epidemic was more or less invented so we can continue to hate blacks and latinos, the rampage against processed food is another way we can all feel better about making fun of that lower class family picking up some more hamburgers from McDonalds and blame them for their own misfortune.

Does this mean Fruity Pebbles is healthy? I have no idea. Fortified cereals have been shown to lower a nationwide deficiency of folic acid, so they’ve got that going for them. But the more important thing to point out is that there are many processed foods that are bonafied healthy – tomato paste being my favorite example as having a more bioavailable form of lycopene than whole tomatoes. Using simple rules like “unprocessed” is a great way to eliminate more than half the population from your recommendation since they can’t afford to eat that way anyway and eliminate a whole lot of good pathways to nutrition.

I’d say this unprocessed fad is no better than all the others, and it really gets to me that someone would go through such trouble to point out why everyone else is lying to you while blatantly doing the same to you themselves. It’s the kind of moral superiority that upper class, out of touch, privilege gets them, I suppose.

May 7, 2014 Posted by | Uncategorized | Leave a comment

Theory is Dead: Long Live Theory

In this recent article, Bob Cringely argues that Big Data and the resultant progress in what has traditionally been called AI will inevitably automate everything – include science itself.

This “death of theory” has been argued before, and periodically comes up from various empiricist leaning circles arguing that all you really need is data. The problem is that, as Kant showed when he reconciled rationalism with empricism, theory and data are intractably linked. Contrary to what some die hard empiricists might argue, we do not believe “1 + 1 = 2″ simply because we’ve seen it enough. We invent notions and concepts to think symbolically, and in that realm, we set up definitions which we use to prove “1 + 1 = 2″ inside the system we’re discussing. We then tie our theories to fact on the ground to make predictions.

Indeed, I’d say the reason AI originally failed in the 1980’s is due to its over-reliance on theory to the diminishment of actual data. Purely symbolic systems can only go so far, and only go so fast, especially in discerning very noisy rules like those of spoken language. And now it appears we’re swinging too far the other way – we’ll never have need of symbolic or formal reasoning systems again, it may be argued. We’ll just throw enough data at it. This obviously rubs many of those actually in the machine learning field the wrong way, as its precisely those formal methods that allow them to code (type checking, compilers) and prove (mathematics, theorem provers) their models work within certain parameters.

The argument Cringely seems to be implying is that theory has just been a crutch for we mere humans, and that if we only had enough data, we wouldn’t need it. However, theory does many things, and even in the era of big data, it will continue to do these things:

1. It is our only path to actual truth.

One can prove theorems based on assumptions. All empiricism ultimately falls to Hume’s problem of induction.

2. Theory can succinctly describe in a single equation many terrabytes of data.

Big data means lots of data – but the above fact is still true no matter how cheap data gets. We can always do more when data is married with theory, each unit of processing power will always be more useful when mixing the two rather than simply trying to neural network the whole problem.

3. Theory can communicate ideas.

Linked to the above in terms of compression of data into a single equation, our symbolic language also makes it easy to communicate ideas to one another. This will be true of big data as well – moving around simple equations will always be cheaper and faster than moving around the entirety of data sets on which they are based.

4. Theory can make predictions out of sample.

Big data’s predictions out of sample are always, at best, guesses. Educated guesses, but guesses. And they are, in turn, guesses based on a few fundamental theoretical assumptions of attempting to minimize error. If we ever run into inputs that are not in our data set, or alternatively, if we want to backsolve for our inputs given required outputs, this is always easier when we have theories to supplement our data. Regression analysis, for instance, allows for a lot more theoretical interpretation of results than a cackle of random forests. When we have regression coefficients, we can make many more predictions about our data set using far fewer facts.

Big data is already changing things, and many of Cringely’s predictions are true. But to say that Big data is going to ‘automate science’ away is a large misunderstanding of what the theoretical side of science does, and how theory serves us.


April 29, 2014 Posted by | Uncategorized | Leave a comment


Get every new post delivered to your Inbox.