Zen and the Art of Software Development

“What follows is based on actual occurrences. Although much has been changed for rhetorical purposes, it must be regarded in its essence as fact. However, it should in no way be associated with that great body of factual information relating to orthodox Zen Buddhist practice. It’s not very factual on [software], either.

And what is good, Phædrus,
And what is not good…

Need we ask anyone to tell us these things?”

-Paraphrased R.M. Pirsig

This, ironically, is a post I’ve been looking forward to. “Zen and the Art of Motorcycle Maintenance” was one of my favorite books growing up, and while it’s no perfect form of philosophy, the subject it chooses to use as an example of ‘Romantic’ versus ‘Classic’ reasoning, technology, fits perfectly with software. Many of the things Pirsig says about technology, and this was back in the 60’s and 70’s, still fits today in software culture. This is not that surprising, as it’s well known that the classic “The Mythical Man Month” was written decades ago but it’s mistakes keep getting made.

The specific passages I’d like to discuss in this “Chautauqua” of sorts, is the discussion of Gumption traps to how software is developed. This is going to take more than a single post, so I’ll use this one to introduce some of the concepts.

First off is Pirsig’s concept of Quality. Pirsig writes quite a bit about Quality, even though he decides its undefinable. But the jist of what I understand, and what I’ve taken away from the book, is quality is what you get when someone competent cares about a project. In our work-ethic driven society, we so rarely remember that passion is not something you can just ‘try’ for, it’s just something that either exists or doesn’t in what you do. You cannot, in other words, try really hard to be passionate about something. You are either passionate about it, or you aren’t, and no ammount of nose-to-the-grindstone work is going to change that.

Quality in software, like quality in everything, remains undefined. But we can say some of the things that are closely correlated with Quality. Namely, a lack of software defects(bugs). Readability, maintainability, elegance.

When one sees an elegant solution to a problem, you cannot deny that the author of the solution didn’t care about the problem. It’s evident from the code! Indeed, I’d surmise that most of our bugs, most of our worst output comes from things we don’t care about. It’s cyclical, in that way. We loose passion in a thing, and then simply try and drive through it. But the code we write, and the designs we produce, during this non-caring state is absolutely pitiful, and we begin to build up technical debt. Because of this, we loose even more passion, and the process gets worse.

Waterfall, and other ‘processes’ are sort of a post-modern response to this. They implicitly acknowledge that designers will loose passion for what they work on, how do we best deal with that fact? Agile, however, takes a different approach in how to maximize passion for work. Honestly, they both have things to teach. But this post is not about software development processes.

Pirsig describes what he calls ‘gumption’ traps. Gumption is a folksy word, synonymous with ‘care’, and basically is the same thing as what we’ve been talking about. A person who cares about what they do is said to have gumption. Gumption traps, then, are common anti-patterns people run into in working on things they care about. Things that sap that care. Learning how to recognize and avoid them is key to fulfilling what one is cares about in the first place.

I’ll be discussing these gumption traps in future posts. But to surmise this gentle introduction to a folksy, modern beatnik philosophy of software development, let’s turn to another aspect of REAL software development we’re more familiar with.

The ‘zone’. You’ve experienced it playing video games and, if you’re lucky(and your employer is very lucky) you’ve experienced it on projects you work on. It’s when the mind becomes focused like a laser on the job at hand, and your productivity goes through the roof. Gumption is similar to the zone, but in an affective way. Gumption is what makes you want to be in the zone in the first place. Gumption is what drives you to finish a project just to see and use the results of your work in the real world, rather than artificial schedules and milestones. In fact, contrary to what some have said, when we are irritated that silly distractions pull us out of the ‘zone’ when we are working, we are not irritated for rational reasons. It is not the ‘zone’ mindset itself that is irritated. It is actually our gumption waning. Being pulled out of the zone is a major gumption trap in itself.

A good example of this is realizing that you really don’t mind being interrupted on a project you don’t care about. And when you realize you don’t mind being interrupted, as I hope I’ll show you through these “Chautauquas”, you’d be better off quitting altogether. Gumption, and caring, is what drives good software, and there is absolutely no process, no method, no trick or tool that will give it to you. You’ve just got to recognize when you have it, and realize it’s best to walk away when you don’t.

Convergent and Divergent Thinking

I read this article in the Times today, and began thinking of how apt a metaphor it was for software.

Not to put words in the author’s mouth, but to sum up, human thinking seems to fall into two discrete camps, what they call divergent and convergent thinking.

Convergent thinking is basically attentioned, focused thinking, where the mind is applies reduction rules to a problem.  Doing a math problem is a good example of convergent thinking, the brain is applying a mechanized system to some problem, where the answer is basically just a number of steps from the problem, and we know to go to B from A, and C from B, and so on, to the answer, it’s just a matter of doing the work.

Divergent thinking is what we do when we don’t know the answer, when we don’t know the next step, when the problem is completely open ended.  Another common problem that you solve using divergent thinking would be a word puzzle like Wheel Of Fortune.  How do you solve those puzzles?  Do you methodically plug in every letter of the alphabet into the puzzle until one fits?  Obviously, even a trivial puzzle could have you calculating for days using that method as it explodes in complexity.  Instead, you stare at it, relax a little, and the answer occurs to you.  It’s a eureka moment.

We’ve all had stories of spending all day working tirelessly, and fruitlessly, on a project, only to have the answer occur to us right as we drift off to sleep.

As software developers, we’ve basically come to believe the false notion that what we do is convergent thinking.  We’re lead to believe that, because ultimately what we’re working on is algorithms and numbers and bits and bytes, we need to use the same strategies we’ve always used for those technical problems.  Indeed, the vain hope with different software methodologies is that the ‘one-true-process’ will emerge and allow anyone to produce what they want given some specified requirements.

We don’t just need experience to tell us the folly of this expedition, now we can look into human psychology.

I would posit that, if anything, the arts, like sculpture, painting and drawing are fundamentally divergent activities.  Certainly an artist must have a measure of ‘technical’ skill, she ought to be able to draw well.  But just drawing well does not a work of art make – she needs to also have a flash of insight, to be inspired.  Otherwise what she will produce will be drab and lack quality.  It will be unmotivated.  This is not specifically tied to divergent thinking, but quality is something I’d like to go on more about, but at a later time.

Anyways, to produce a work of art, one must have the technical skill to do so, but also the flash of insight to know what to produce in the first place.

Software, while having some ‘engineering’ aspects to it, is fundamentally a creative pursuit.  This is not any sort of wishy-washy statement, but a simple conclusion from logical deduction.  Yes, it’s ironic that we’re using logic to show that straight out logic won’t suffice in the end, but that’s beside the point.

The fundamental problem solving technique we developers use is abstraction.  Upon finding a solution to a problem, we identify what can vary, abstract out the solution, and reuse it.  This is the basis for libraries and all other code ‘reuse’.  The point being, if there is ever an ‘algorithm’, or actual pattern of work that is done, some sort of mechanical process, it is within our power to abstract that pattern out and reuse it as a primitive.  Despite what the GoF will tell you, patterns CAN be generalized and pushed into languages and libraries.  Hence, in the ideal development environment, you won’t be faced with reimplementing some already known technique or solution, but instead will have those solutions already at hand.  What you will be doing is using those already known solutions in a completely unique way that’s never been tried exactly like this before.  You now have an open ended problem, for the sheer sake that all the problems that convergent thinking could do have already been solved for you by people before you.  And once your solution is complete, it too will fall into the hall of convergent solutions, and be reusable.  Hence, the only real work in idealized software is creative work of taking the solutions we already have at hand and putting them together in a completely new and unique way.

That ‘proof’ required a lot of hand waving, but I think you get the point.  Common sayings in software like DRY and KISS exist as words of wisdom because we should not be doing mechanical work as software developers – that’s what the computer is for.  We should focus on creative work, and let the computer do the rote stuff.  Most problems in software, I suspect, come from either the false belief that creative work can be shifted into mechanical work (and our endless supply of methodologies is evidence of that) or that mechanical work should be redone as creative work (and reinventing the wheel, or not-invented-here syndromes are evidence of that).

As the article at the top implies, creative work is best done divergently.  And creative work is best solved by not trying to solve it at all.  Know the problem, understand it in detail and research it.  Investigate other attempts at solutions and have discussions, but never think you can just sit down and mechanically come up with some sort of solution.  Inevitably, the ‘ideal’ solution that ought to be a one-liner will blow up into thousands of lines of case statements as you learn more and more about the problem.  As we know, the best predictor of number of bugs is number of lines of code.   Less is definitely more.

Back to my post on new hires being driven by company culture to simply throw as many hours at a problem as possible, we can now see how mere hours won’t amount to much.  Psychology has shown that while our ability to solve problems requiring convergent thinking decreases as fatigue increases, it does so at a much slower pace than our ability to solve divergent problems.  Sleep deprivation saps creativity, gumption and quality in one’s work.

In fact, any sort of ‘drive’ to find solutions mechanically will cause work to suffer.  Many times, when trying to figure out how to design some sort of extensible framework, the BEST thing we can do for ourselves is to stop working on it.  Sure, you can do the ‘moral’ breaks like taking walks, or a corporate-approved power nap.  But ‘immoral’ things would work too – surfing the web, reading blogs, chatting, playing video games.  All of these things improve morale and give your divergent mind a chance to churn away at the real solution.

I know it’s against our protestant culture and mindset to ever believe there’s virtue in laziness, but I maintain that all great insights occur when we’re not even looking for them.  So to increase your chances of finding those insights, you best stop looking for them as soon as possible and let them creep up on you.  You may end up ‘working’ less, but getting a lot more done.