Like software patterns, there are many different software creative processes that tend to be re-invented over and over again.  Similar to the way that some software patterns are actually different versions of each other, or more importantly, weaknesses in the implementing language, I believe some processes only exist due to weaknesses in the chosen technology.  An example of one of these processes is the ’spec’ or specification.

Specifically, I’m talking about the technical specifications described in the above link, not functional specifications.  I believe technical specifications are a signal of weakness due to the following thought process – a technical spec can be deemed ‘good’ when it is unambiguous and has a relationship with actual implementation nearing 1-to-1.  If either of those two facts is untrue, then that means there is some element of interpretation in the spec, which further means that two coders may implement the spec in two different ways, making different assumptions, thus leading to defects.

So let’s assume we have a good spec, one in which we have a close to 1-to-1 mapping between specifications and implementation, and a lack of ambiguity.  There are two thoughts I have on this – one is, how did we get here?  The other is, how is this any different from code itself.  I’ll deal with the second first, just to spice things up.

A good spec really isn’t any different than good pseudocode, in fact you’ll find a lot of English descriptions of looping behavior, if statements, etc., primitives in most imperative languages.  These can be turned into implementation rather mechanically, so the question results, why didn’t you just implement the spec in the first place rather than writing the entire thing out in English?  What exactly does a spec get you that doing the exact same process except using actual implementation language doesn’t?

The other question is, how do you create a good spec?  Well, most likely you write a working draft, then send it out for reviews.  Various other designers look over it, read the spec, then send back some comments.  Maybe you missed a corner case, maybe you didn’t think of a certain interaction.  Over time, the spec is rewritten to be more and more specific and consistent.

Let me know if you disagree with any of the above logic so far, because now I’m going to add a twist.  Let’s say, instead of writing a spec, we actually wrote implementation code from the get go, except we used the same processes described above.  That is, we iteratively rewrote it, sought frequent feedback from humans or otherwise, and we stopped when there was a 1-to-1 mapping between the problem at hand and a complete lack of ambiguity (rather easier to do with a programming language).

We just re-invented rapid prototyping.  You see, spec writing is what people are forced to do when they work with languages or technologies that don’t allow rapid prototyping?  What exactly, when enumerated, would these technologies be?  I’d venture a guess that things like a REPL and high level ’scripting’ interface would be a good subset.  Languages like Ruby or Python, or heuristically, anything that doesn’t have a long ‘compilation’ cycle.  There are many compiled languages, for example, that turn around quickly allowing rapid prototyping, so I’m not JUST saying scripting languages.  Languages with more boilerplate, more line noise, and a longer turn around cycle lend to writing specs over rapid prototyping (i.e., C/C++).

Rapid prototyping has many other benefits over spec writing.  For instance, during spec writing, human reviewers are not only finding high level logic errors, but also syntax problems, type problems, and the like.  These are all problems that a high level language will automatically find.  The enumeration of ‘what’ could be passed into a specific function mentioned in the article linked to previously is a prime example of type checking that Haskell would let you know on the spot if you hadn’t fixed.

Moreover, spec writing and rapid prototyping aren’t competitng ideas.  In other words, a spec COULD be a rapid prototyped code base!  If we need to build an embedded system using a subset of C, for example, rapid prototyping would be a rather onerous process compared to our REPL and high level constructs.  But no one said the spec couldn’t be itself written in a high level language to be ported to low level C once it was deemed complete.  Therefore, rapid prototyping should be used even on projects where a spec is used, and the two should be practically substitutable.

Of course, all process patterns like these are just that, patterns, and there really should be no ‘make software using process x’ rules since x may not always apply.  Sometimes you need some exploratory programming (very similar to rapid prototyping), or sometimes you may know how to express something in English but not in code (such as in technical fields).  Patterns, even process patterns, ought to be seen more as a ‘bag of tricks’ similar to the skills mathematicians build up – tricks to simplify or restate problems in other domains.  Too often we’re convinced that our problems are like arithmetic, and there’s just one simple algorithm or process that needs to be exercised again and again, but instead they’re more like problems found in the higher echelons of math.  They require insight, tricks, squinting, hand waving, and eventually, a eureka.

Things like dependency injection and IoC are cool, and they do decouple your code and make it easier to extend at test.  But the problem is that the software community always tends to reinvent the same solutions to the same problems and then give them a totally different name.  I mean, what are the real differences between Haskell Type Classes and the now defunct C++1x ‘concept’… er… concept.  (Seriously guys, we’re running out of different synonyms for ‘thing’.  We’ve got object, class, type, type class, structure…)  Here’s what IoC and DI are, how they interrelate, without using silly words like ‘container’, which is just another word for ‘thing’ pretty much in this context.  We have a function, let’s say, and I’m going to use python notation since it’s short and sweet.

x = 0

def func(y):

return x+y

Everyone can figure out what that function does.  But how will it react in a large system?  Who the hell knows, it uses a global variable that can be modified at any time, from anywhere.  Our function ‘func’ is now coupled to a global variable, as is, by associativity (or is it commutativity?) anyone who uses it.  So we change it to this far more elegant version:

def func(x,y):

return x+y

Voila!  In DI terms, we’ve ‘injected’ the ‘dependency’ of func on the variable x.  Now func is only dependent on two locally defined parameter variables, a far more elegant and simple way to do things.  Let’s fast forward to OO for a second.  Assume we have a class foo…

class bar:

def __init__(self):

self.myFoo = foo()

def func(self, x):

return self.x + self.myFoo

This is OO, so it’s nice and decoupled right?  Not really.  If you unroll the object, thinking of it basically as a poor man’s closure, you realize that func is actually doing something like this (assuming no one changes self.myFoo)

def func(x):

return foo() + x

Global functions, when they’re pure, are nice to have.  But you can still see the problem of the coupling between func and foo, now.  If foo is not pure, then we have side effect issues, but moreover, we still have just plain maintainability/extensibility issues with the coupling between the two.  The number one use case pointed to by dependency-injection types is that of testing.  What if we want to test func, but we don’t want to test foo?  Maybe foo takes a long time.  Or maybe we haven’t even written foo yet! (Now you can see the connection between TDD, dependency injection, and why the two might change your style of design.  You can test top level objects without even implementing ‘low-level’ ones yet..  But that’s tangential.)  We can do the same thing we did in the first case though to break that coupling:

def func(x, foo_thing):

return x + foo_thing()

Oh hey, higher order programming.  Who’d have thought that would show up here?  Let’s roll our stuff back into familiar OO terms:

class bar:

def __init__(self, foo_thing):

self.myFoo = foo_thing()

def func(self, x):

return self.myFoo + x

Dependency, consider yourself injected.  Ok, fine, what the hell is IoC then?  Why does it have to have such a crazy name, and what are these containers, and wah wah wah!  Well, let’s go back to our familiar func and make it a little more complex:

x = 0

y = 0

z = 0

a = 0

b = 0

c = 0

def func(var):

return var + a + b + c + x + y + z

Gross!  Global dependencies on like, a bajillion different things!  Let’s inject those dependencies!

def func(var, x, y, z, a, b, c):

return var + a + b + c + x + y + z

Ah, much better.  Er, wait… now my buddy, who had code like this:

def func_2(var1, var2):

return func(var1) + func(var2)

Has had his interface explode to:

def func_2(var1, var2, a, b, c, x, y ,z):

return func(var1, a, b, c, x, y, z) + func(var2, a, b, c, x, y, z)

And his callers have their interfaces explode, and their callers have theirs, etc…  But wait, any of you OO gurus notice that a,b,c and x,y,z never seem to make any sense without each other?  They always seem passed with each other, they seem semantically related, who knows what else.  Why, I’ll bet our design could be more… what’s that nice word elegant designs are described as?  Oh yes, cohesive!

class point:

def __init__(self, x, y, z):

self.x = x; self.y = y; self.z = z  #just leave them as plain-old-data and public

Oh wait, and two points, well, we can call that a…

class distance: #er, maybe not the best name…

def __init__(self, pt1, pt2):

self.pt1 = pt1; self.pt2 = pt2

Now are interfaces shrink down, only needing one extra parameter, a ‘distance’, which I’m sure in our func, foo, bar whatever application we’re building, make good sense semantically and is drawn from our domain.  Likewise, let’s say that we’re using dependency injection to make our objects a lot less coupled.

class bar:

def __init__(self, obj1, obj2, obj3, obj4, obj5):

self.obj1 = obj1; self.obj2 = obj2; self.obj3 = obj3; self.obj4 = obj4; self.obj5 = obj5

Wow, those dependencies sure are injected all right.  But that class has been injected so much, it’s beginning to look like a heroine addict.  These objects all seem to have something to do with each other, why don’t we bundle them up in a cohesive object?

class bar:

def __init__(self, obj_builder_thingie): #TODO: think of a better name

self.obj1 = obj_builder_thingie.obj1(); …

Ah, now our design is decoupled and cohesive.  Hallmarks of good design.  (Of course, simply bundling arbitrary objects together isn’t necessarily cohesive.  We’re assuming in this case, whenever we’ve bundled something together, those objects have had something to do with each other semantically or in the domain.)  For some reason, whenever we bundle these objects together, our Java buddies (and C# buddies) want to call this ‘Inversion of Control’, mostly because it’s now the caller/creator’s problem of figuring out how it wants to ‘configure’ the innards of our bar class.  They sometimes will go even further and make obj_builder_thingie not just a normal object, but a framework for going out and reading XML files to dynamically figure out what obj1() and obj2() should return.  That way, instead of editing a nice clean python unit test file to configure and modify your tests, you can edit a huge ugly xml file with lots of angled brackets to configure and modify your tests.  Sounds much more enterprisey to me!  Job security here I come!

So, let’s recap.  Dependency injection is moving a ‘global’ to a ‘parameter’, which insofar as primitive types and values, has always seemed like a good idea.  But for OO guys, moving global functions (or an object representing those functions) seems like a brand spankin’ new idea, when really functional guys have been moving global functions to parameters for years.  C’est la vie, at least we’re learning something from each other.  The specific case bandied about is always how this makes testing easier because you can ‘inject’ which objects you’d like to collaborate with in the constructor (or a setter, etc… but setter’s are evil.)  But this solution can be generalized to ‘inject’ any configurable behavior you want in the form of variables, functions or classes themselves in a dynamic language like python.  The side effect is your parameter lists get huge, and while your code is decoupled, its decoupled kind of like a box of legos with no instructions on how to build your starship.  So you use your ole’ object-oriented smarts to start grouping some of those parameters together into objects themselves, and now  your ‘injected class’ begins looking through the objects that are passed into it to find what it needs, rather than everything being unrolled on the parameter list.  IoC is basically, then, dependency injection (or the voracious removal of anything global) on a large scale and must, then, solve the problems (explosive parameter lists) that come with that.

That’s it in a nutshell.  Hopefully I haven’t confused you further.  Dependency injection is just another step in the good practice of “make things as local as possible”, or decoupling.  Parameter lists are more local than global, even for functions like class constructors.  IoC is the counterbalance to massive decoupling, and that’s to make the program more ‘cohesive’ again by grouping up many of these new things being passed in parameter lists into objects and hierarchies themselves.  Any questions?

Edouard in his post claims that smart pointers, like shared_ptr and auto_ptr, are overused.  While he actually makes quite a few legitimate critiques, I think he misses the fundamental problem that hits nearly all of the computer scientist’s beautiful abstractions – they’re all dangerous if they’re overused.  C++’s smart pointers are no exception.  I do wish to come to their defense, though, and some what might seem like obvious guidelines for their use.

Smart pointers are the poor man’s garbage collection.  Or, in better terms, they are the average C++ developer’s garbage collection.  C++ is a language that is a little bit too ‘expert friendly’, and dynamic allocation, similar to multithreaded programming, is hard(tm).  Smart pointers in many cases allow your average developer to turn a horrible bug – memory leaks or null dereferences – into a less heinous inefficiency.  That’s a win.  They are no complete replacement for object lifecycle analysis though, which is Edouard’s main point.  Of course, raw pointers that are intelligently managed will always outperform smart pointers.  But what if you don’t have the smartest developers in the world?  Or what if you don’t have the time/money to throw your smart developers at making sure you squeeze every last FLOP out of that processor, but instead need them to focus on implementing new features, etc?  It’s all economics.

I’d like to offer my own taxonomy to budding developers.  Kind of a road map to memory usage in C++ to find a good middle ground between performance and safety.

Your first rule of thumb, and the first place you should go to build any object, is the stack.  The stack is well understood, putting objects on the stack is usually pretty cheap, and a good chunk of the time, most objects you need are transient for a few statements and then aren’t needed any  more.  For awhile there, it was simply considered more ‘OO’ to put things on the heap (thank you very much JAVA).  Thankfully, those days are gone.

What CAN’T be put on the stack?  Well, large objects run the risk of pushing the stack to its limits, since most of the time, the stack is smaller than the heap.  In addition, the stack requires a class to be concrete, which makes polymorphism a little more tricky.  Finally, again with large objects, is that passing these arguments to functions or methods can be expensive as their copy constructors are called.

For a large object that you want to put on the heap, enter the scoped pointer.  The scoped pointer acts like a stack variable in its construction and cleanup, but is created on the heap.  You should also question why you’re putting so many large objects on the heap.

To handle polymorphism, for the most part, you’re going to have to use pointers.  An abstract base class pointer can point to a derived class, so it’s a question of what kind of management you want wrapping that pointer.  Scoped pointers don’t really help any with polymorphism, but our two friends – one of which is disparaged by the Edouard – come to save us.  In the case of ‘factory’ like functions, or functions that create an object on the heap and ‘pass off’ ownership to the returnee, we’d like to use an auto_ptr, since this type of smart pointer enforces this pattern.  In the case where we’re getting a reference to an object, but the method called still wants to refer to a shared place in memory, then we usually want a shared_ptr.  The original poster is right, though, in many cases, through careful analysis, both of these pointer types can be done away with by simply studying the expected lifecycles of the object underneath.  But this is not always possible in the constraints of developing software.

Keep in mind the const reference trick, too.  Returning a reference to a stack variable is a bad idea, since the stack variable’s probably been deleted.  However, the standard states that getting a CONST REFERENCE extends the lifetime of whatever variable is being returned to the enclosing scope.  That actually opens up efficient usage of some uses of polymorphism, since the memory’s allocated on the stack yet we can still get a polymorphic reference to it.  I’d use this technique with care, though.

Ultimately, such low level control and the plethora of options available to the developer makes memory management in C++ tricky, yet powerful and abstract.  There’s different use cases for all the potential views of memory, and subtleties to each use.  Shared_ptrs can more or less cover almost all these uses, but there’s a cost associated with not doing any real analysis of your memory footprint.  For the average developer, though, if you want to make sure they don’t introduce any risks, giving them a poor man’s garbage collector is probably your best option.

(*Though you still need to peer review their code and check for cycles, the notorious corner case that shared_ptr will still leak with.)

“Metacognition” is a word that basically means, thinking about thinking.  You take part in metacognition when you actively try and spot patterns in your own behavior that are acting against your goals, and then trying to create solutions that work around these problems.  It’s self reflection, but also action that takes place due to that reflection.  It might be uniquely human.

There is an analogue in society, though, when any sort of social movement begins to use it’s own doctrine to look back on itself.  This is generally restricted, though, and does not include  self reflective studies using other means, for example, a political movement using scientific evidence to rethink its position on things.  I’d suppose we could call that ‘iterative’ progress.  But instead, its when you turn a social movement with an intrinsic ability for self reflection back on itself.  Say, let’s psychoanalyze Freud, for example.  Or, more importantly, let’s apply the scientific method to science itself.

In science, let’s say medical science, we have a hypothesis that X causes B.  Now, medical science is one of a few ’special’ branches of science who are motivated by more than just the search for truth itself.  Generally, if X causes B, B is a bad thing and we want to stop it.  Knowing that X causes it means that we can have more control over B and reduce suffering.  So when we do medical science, we more or less want to do it as ‘best’ we can, to get to the bottom of various diseases and halt human suffering.

So we hypothesize that the current way we do medical science is the best, i.e., that it yields the most truth and that these truths are the most effective to reduce human suffering.  If, through that hypothesis, a researcher were to be able to find a pattern in medical science itself that yielded weakening evidence, that researcher would be doing science on science itself.

But like medical science, who’s treatments and drugs improve (by some small degree) by every paper published, does science not itself recursively improve through it’s own self analysis?  By applying the results of science and mathematics, in the case of the above link, network theory, to the processes that produced those results, science becomes STRONGER due to it.  The scientific method as we know it from the enlightenment is a sound way to find truth about our world, we must always remember that the evidence we produce from any one experiment is only as good as the assumptions that went into it.  And there are many assumptions in the scientific method itself, of which, we learn about more only through self reflection.

Medical science, not to pick on it, is rife with these enlightening moments.  From discovering that taking a sugar pill makes people feel better (the Placebo effect), to realizing that people who follow doctor’s orders tend to be healthier than those who don’t, even if those doctor’s orders are non-sensical.  We have invented the double blind placebo controlled trial to control for the scientists’ own bias as well as human’s predilection for sugar pills.  We’ve invented meta-analysis to momentarily reflect on our own science to try and combine and better glean results from a group of studies.  We’ve invented the idea of conflict of interest, the open access journal as well as many other tools and techniques that were the result of turning the scientific method back on itself.

You can see the result of this recursive growth in transistors.  We’ve built computers, and then only months later, used those very computers to design the next generation.  This same recursive growth is possible in science too – the very results mentioned above shows that we CAN do science better.  Lives are at stake in many cases, and if not lives, then large sums of money.  We can only do so much good science, produce so many real results, in a year for a given dollar.  We should focus on making these dollars go as far as possible by applying the results of (and funding more of) these sorts of self reflective studies.  The more bias we remove from our scientific process, the cheaper science gets for any dollar put in.  The cheaper it gets, the more we can do.  The more we can do, the more we learn.  The more we learn, the more lives we save and the more wealth we create.

And the cycle continues towards singularity.  Kurzweil is right, but he never mentions culture.  Culture, and the products of human culture, are also capable of exponential growth, and I don’t think we’re anywhere near an inflection point.

Long live Software Engineering!

The exalted Father of many of the processes we know and lovehate, Tom Demarco, recently wrote an article describing his second thoughts on many of his prescriptions from his book, Controlling Software Projects: Management, Measurement and Estimation.

Much of what he says should ring true to most of us in the trenches.  Attempting to code directly to software metrics is a fools errand.  Not only do the current methods of collecting metrics frequently have relatively high costs to true value, but they also cause a game of metric cat and mouse where software increasingly fits what metrics say ‘looks good’ but loses all qualities associated with those metrics when developers begin coding ‘to’ the metric.  It’s basically teaching to the test!

It’s refreshing to see such an intellectual giant in our field so humbly admit his faults in the past – surely many more could learn from him.  We’ve had far too many methodologies come and go while their authors seem to continue to pretend that the few good ideas encased within those methodologies are worth all the cruft that’s built up over the years.  Just like in many cases what a software project needs is a complete gut-job or rewrite, some methodologies could use the same.

Let’s not throw out the baby with the bathwater though.  I’m convinced that nearly every ‘fad’ that’s appeared in the field of software HAS had some valuable things to teach us.  It’s all about keeping the good and dismissing the bad, separating diamonds from the rough.  The Waterfall method as a whole has shown to create projects drastically over schedule and over budget, but it has shown that many of our most expensive errors come from misunderstanding requirements/use cases.  Likewise, coding to metrics in and of themselves is going to get us no where.  It’s like writing until Word’s language analyzer gives me back a high ‘reading level’ for my paper.  Word can’t analyze the content or tell whether I used the English language correctly.  But, all other things being equal, these metrics – when combined with sound human judgement, can make our jobs easier.

If I’m brought on to a failing project as a firefighter, I want to know where the flames most likely are.  I can either slowly and methodically scan over the code with my own eyes – most likely spending weeks chasing false positives or style issues (even the most disciplined of ‘real’ developer is going to confuse style with content every now and then), or I can use a host of automated metric gathering tools to give me hints on where to start.

A few object oriented metrics, like various definitions of coupling and cohesion, probably would give me a good clue on where to start refactoring for maintainability.  Some good old basic metrics like line count per function or cyclomatic complexity may give me some good hints on where I ought to start tearing apart some unreadable or stagnant code.  Simple metrics like test coverage and test count would give me an eyeball figure of how brittle I should expect this code to be.

The point is, metrics in and of themselves say nothing about software quality.  When interpreted by a skillfull developer, they make that developer that much more useful and productive.  Since, as I’ve mentioned before, Amdahl’s law says as much about software projects as it does about software itself (namely, that any particular ’solution’ a software product is to solved has an optimal number of different ‘threads’ of production, and any developers working on the project in addition to that will actually slow it down due to communication overhead), assuming we have an optimal number of developers, the only way we can speed up projects is to actually increase the speed of each individual developer rather than adding more.

But this is the rub, isn’t it?  In fact, hasn’t this always been the rub?  No one ever said metrics were all you needed, but it was in fact a misinterpretation by PHB (pointy-haired bosses) that we could automate and more easily manage software development with these metrics.  A hammer will allow a craftsman to do a lot more work than his hands alone, but also allow someone unskilled to do a lot more damage.  While the point that software is a people problem, not an engineering problem, is beyond the scope of this blog post, perhaps DeMarco’s later focus on the people problem and now his disemphasis on the ‘engineering’ problem will help the community lurch (ever so slowly) towards really understanding how to build software.

“Premature Flexibilization is the Root of whatever evil is left” takes a look at what I’ve also heard called ‘premature abstraction’, and in today’s high level languages, probably is on par with premature optimization for which causes the most headaches in maintenance code.

It’s also a really funny title.

First, a note about culture.  I work mostly with EE’s (electrical engineers) who cut their teeth on after all these years still swear by Fortran.   I’ve mentioned the odd distrust between EE’s and computer science majors:  engineers think scientists don’t know how to get real work done, while the scientist don’t believe engineers understand any of the complexity of what they build.  Ever find code that attempts to match a string by going through each character one by one against a switch statement?  That’s an engineer at work.  Ever work with a AbstractBuilderFactoryObserver that’s templated on five types?  That’s a scientist at work.  So as not to confuse these ’scientists’ with real scientists, I’m going to call them designers.  Engineers thrive on seeing things work, designers thrive on building them.  A designer’s work is done when she’s figured out how it should work, there’s no thrill in actually building it.  An engineer’s work is done when it’s built and he can test it.  He wants to see the capabilities of the thing when he’s done.

You can see that each of the major sins discussed so far, premature optimization and premature abstraction, also tend to be committed more by one community than the other.  Code that’s prematurely optimized tends to come from the engineer’s world, while prematurely abstracted code tends to come from the designer.  They each find different reasons to defend what they should know to be wrong.  Frequently I’ll hear an engineer friend tell me that he has to code in such and such way because he’s targeting an embedded system.  I’ll ask him for profile evidence, or try and reason with him that he’s avoiding a temporary variable that doesn’t appear to be in a tight loop.  I’ll try and say that the compiler’s optimization will catch what he’s trying to do.  No avail.  For the most part, he doesn’t trust compiler optimizations.

The designer will claim that she’s building with maintainability in mind, and that the alternative is cowboy coding.  She’s being rigorous, planning for change.  You might ask, “What are the use cases for this abstraction?” which usually yield something like “well, maybe we want to target a completely different OS.”  That’s true, maybe we DO want to target a different OS, but that’s not a change most projects take lightly.  More often than not, some business rule is going to change.  It’s an abstraction that’s going to cost developer time and energy for everyone who touches it, and most likely never have any pay off.

We all know (or we should) when to optimize.  At the design phase, we should be considering and analyzing use cases, choosing efficient data structures and algorithms.  We should draw on our experience in the field to predict certain hot spots, but not rely on this experience too much (in our rapidly changing field, experience has a pretty short shelf life).  We should be aware of free optimizations, or those that don’t sacrifice readability, when coding.  We should design with performance in mind, but realize that performance mostly comes from reducing work, rather than working more cleverly than normal.  When we finally have something built, we should stress test it to find hot spots, and then we should begin working on our ‘clever’ approaches.

But it’s not chiefly taught when to abstract.  This is because, for our go-lucky computer scientists coming out of their Java schools, abstraction is still pretty fresh in everyone’s mind.  It took us about fifty years of being burned by prematurely optimized code to realize we should code first for readability and optimize later.  It is only now, with our gigantic frameworks of frameworks and systems of systems that we’re realizing we’re creating huge  spider web messes of code.  Just look at the Eclipse framework – quite a marvel of design.  Anyone can write a plug in and it WILL work, and you CAN extend or change virtually any part of it, but I’d claim that 90% of the parts of code that CAN change DON’T in any extension.  Yet each part of code that is modifiable exacts a cost on the maintenance of the system.

So when should we abstract?  When should we decide to use polymorphism or some other means to put a ‘point of inflection’ in our code, a point where we can extend it?  I suspect we can derive some simple rules (indeed, many already have such as KISS and YAGNI) based more or less on the lessons of premature optimization.

First, in the design phase, for optimization, we take two things into account – we pick good data structures and algorithms that fit the expected usage and we let experience be our guide (to an extent).  How would this apply to abstraction?  To me, it says that our first pass of abstraction, or that which takes place during the design phase, should attempt to model the points of inflection that exist in the domain model itself.  If the domain model actually contains the classic ‘animal’ base class, with cats and dogs being subsets, then that polymorphism certainly belongs in the code as well.  How can experience guide us?  This is where patterns come in, and a designer who’s solved similar problems before may be able to offer some insight as to what he or she expects to change in the future.  BUT, just as experience has a short shelf life in optimization, it does as well in abstraction.  Design pattern Hell is the result of searching a little too hard for these, and ultimately, experience tells us how to build the last system, not the next one.

The second point to learn from optimization is the “free optimizations”.  These include compiler optimizations, and little tips like “pass big structures by reference” and what not.  These are the idioms in the language, in addition to being the more efficient way to do things.  What “free abstractions” can we expect from our languages?  Well, first and foremost, are the abstractions the languages can get us – these are things like libraries and frameworks that are already built.  Prefer these to home-rolled solutions, as they’ve already been abstracted in many useful ways without any work on your part.  Secondly are the “best practices” and idioms that improve abstraction with very little cost to readability or understandability.  This would include popular techniques like dependency injection and preferring to pass by interface rather than concrete type.

The third point is that we should design with performance in mind, but the best performance boost is work avoided rather than work done cleverly.  In abstraction terms, this means that small, elegant solutions tend to, ironically, be more extensible than large abstract ones.  The best example of this might be the object hierarchy mentioned above, our ‘animal’ base class and ‘cat’ and ‘dog’ concrete classes.  Well, to be extensible, we might need to have a ‘multithreaded_cat’ and ‘multithreaded_dog’ class as well, or maybe even a ‘distributed_cat’ and…  But all of this avoids the original point.  We can avoid all this work in the first place by ensuring neither cat nor dog make any assumptions on the threading scheme.  That way, one hierarchy describes threading and one describes our domain and they never need meet.  We’ve avoided a whole lot of abstraction (an exponential explosion of classes) by simply designing elegantly in the first place.

Finally, to optimize, we measure and begin offering up some more fine tuned ‘clever’ code where we find our hot spots are.  Similarly, in abstract code, we must refactor at our points of repetition.  You can get some abstraction done at design time, but much of your abstraction is only going to become apparent as you code.  You should be on constant gaurd for repetitive code, as this is a sure sign that the code should be encapsulated elsewhere.  Similar to the optimizations that mostly sacrifice readability, the abstractions found purely by refactoring tend to be the ones that are most at risk for ‘leaking’, or, sacrificing understandability for a point of inflection.  But, so long as we have proof that the code is used in multiple places, it will save us work in the long run.  It is these sorts of abstractions that can only be ‘discovered’, rather than designed in, because we are so poor at predicting what code is going to repeat itself as we develop our system.

Optimization and abstraction both are important parts of performing, maintainable systems, but we poor humans have shown ourselves to be poor predictors of where the tools of optimization and abstraction best be leveraged.  But we needn’t take another 50 years to learn how to best abstract.

It’s the cool thing to do, after all.

First, Jesse pointed out (actually, a friend of his pointed out) that if you can’t name five things you don’t like about your favorite language, then you still don’t understand it.  I think this is a little on the liberal side, though, as most reasonable people can figure out five things they don’t like about C++ before they understand even half of it.  Of course, in that case, understanding even half of C++ is about as far as most people get

Then was Zed’s own contribution to the list.  Both of these guys are very active in their communities and contribute a lot of code via libraries and projects they work on.  Neither of them is a ‘dirty little schemer‘.  Yet now that they’ve been seen disagreeing with the politburo, the wagons have been circled and they must be torn apart with ad hominem attacks until everyone’s convinced Python’s perfect again.

As Jesse and Zed both tried to explain, I will as well.  I love the Python programming language.  It reads like pseudo-code, and is incredibly expressive.  It’s ease of incorporating C or C++ makes nearly any performance complaint moot, and the vast number of libraries out there gives it a gigantic code base to work from.

But, again, as Jesse said, if I thought Python was perfect, I’d be fooling myself.  In fact, I think it’d be an interesting challenge to hear Guido’s own list of five things he wish he could fix in Python.  GVR is hard to read, sometimes he’s incredibly reasonable, sometimes he’s a little irrational.

But this post is not about politics, on with the list!

1. Static Default Args

This gets brought up time and time again.  The arguments that you provide as defaults to functions are done at ‘compile-time’, and are not re-evaluated during runtime.  This means you get funny behavior like this:

  >> def func(arg=[]):
  ::   arg.append(1)
  ::   return arg
  >> func()
  [1]
  >> func()
  [1,1]

I’ve got two problems with this.  First of all, as is the classic way to shoot down ideas the community has already ‘decided’ upon (other than the ‘code it yourself’ comment), what exactly is the use case for this behavior?  What idioms does it allow that I’ve yet to see?  Indeed, ‘idiomatic’ python seems to be to work around it, by using ‘none’ as the only real default argument.  First you check for ‘none’, then if the arg is in fact ‘none’, you set it to its real default.  This is added boilerplate and, quite honestly, seems to be the default (forgive the pun) rather than the exception.

Another argument for the status quo is that it’s ‘more efficient’.  I don’t get this line of reasoning at all.  Virtually 99% of debates between ‘efficiency’ and ‘readability’ in Python always end up on the side of readability.  That’s why Python is Python!  Why efficiency gets a free ride in this most awkward of cases is beyond me.  Moreover, it’s not at all more efficient, as far as I can tell.  The replacement, checking for ‘none’ and then recomputing the value at runtime is exactly the same efficiency hit as the proposed dynamic default args!  Most people are already taking the efficiency hit.

Indeed, it looks like this is the rare case where Python has optimized for efficiency of a niche use of the language, which to be blunt, is not at all Pythonic.

2.  String Concatenation

Most people new to a programming language are going to get a feel for the way to do things.  When you want to put two lists together, they experiment in the Python shell, try simply +’ing them together, and hey, it works.  Doing the same thing to two strings also has the same effect.  So obviously, if you need to add many strings together, you add them together like you’d add many other things together.  Wrong.

Instead, the “Pythonic” way to do things is to put all of your strings in a list and do the following:

  >> "".join(myStringList)

Does that at all seem obvious at first?  The joining of many strings is a method on another string?  In its defense, the string being ‘joined’ on is the separator string.  But even that seems odd, since the separator really isn’t the ‘object’, in object-oriented terms, I’m worried about here.  The object is a list of strings!  The reason given by the Python community is that “”.join() is faster, and therefor preferable for large scale string processing.  While I might balk a bit at again optimizing for efficiency in Python, where everything else is optimized for consistency and readability, at least with string processing there’s a major use case for it.  Python is used for string crunching a lot, and the “”.join() construct has some specialized C routines to do it fast.

This brings me to a slightly tangential point, but one that really applies here.  Python is a notoriously ’smart language, dumb compiler’ compared to languages like C++, where it’s a ‘dumb language, smart compiler’.  That codifies, in other words, the constant strive for readability and intelligence in language design.  But it also implies that there’s no major drive to optimize the Python compiler.  Scripting languages are slow, and are going to continue to be slow for awhile.  Besides, Python is mostly used for things that aren’t processor bound anyway, and the FFI is good enough to drop into C whenever you need.  Why optimize?

But there is a particular kind of optimization that enhances readability as well as speeding up the execution time, called Strength Reduction.  The idea is, when the compiler detects a construct that can be stated in an equivalent yet more efficient way, it replaces it behind the scenes.  That way you can still divide by two, because that’s how a human would read your algorithm, yet the compiler will go ahead and turn that into a much faster bit shift.

Similarly, appending multiple strings together is a pretty common operation in Python.  I still believe that the most obvious way to do this is to just repeatedly add them together.  Strength Reduction optimizations would detect this repeated addition and change it to the more efficient C routine – or hell, even an assembly routine.  GVR has been skeptical of optimizations in the past because he’s afraid they will change behavior from what the user’s expect, but in this case, behavior is entirely conserved.  Only the runtime changes.  In fact, from an optimization standpoint, Strength Reduction is more Pythonic since there’s far too much anecdotal evidence that the user’s original expectations is to use repeated addition rather than some special method on an empty string.  The ‘optimized’ approach to use a library function to drop into C is not expected and is actually pretty difficult to find the first time.

3. Non-recursive Reload

Working with the interactive shell is awesome for someone coming from mostly compiled languages.  Getting immediate feedback allows for easy exploratory development.  At first, I was entranced by the idea of having a large code file, making incremental changes in an IDE, then reloading it in the shell and testing various procedures on it.  That is, until I realized reload does not do what one expects it do.  When you import a file, it reads the module, and imports all the things that file imports, as well as defining things in that module’s namespace, and so on.  But when you reload a file, the imports – since they’ve already taken place and are in the global namespace – are skipped.  You only reload the things defined in that file.  If I’ve imported Module X, and it’s imported Module Y, and I change Module Y, simply reloading X won’t do me a bit of good.  X will still point to the old version of Y.  Instead, I need to reload Y… oops, no, that doesn’t work either.  First I need to import Y, and then reload it.  That doesn’t make any sense at all.  How come reloading tells me it’s not loaded, yet import assumes it is.  Importing a module that has already been imported by some other module just copies the namespaces over, I’d imagine, and reload seems to absolutely depend on those namespaces.  Either way, it’s a lot more complicated than it seems at first.

Upon reloading a file, imports need to behave differently, and files really ought to be re-imported.  I’m sorry if this is a performance hit to you, but you really shouldn’t be using reload in a library anyway.  Reload should only be used in interactive development, in my opinion.  Python’s reload and import process is similar to other languages Make process.  Make is highly recursive, indeed, that’s often a problem since you have to nest things so oddly in esoteric Make files.  But Python’s reload doesn’t even have the functionality to go out and see whether or not it needs to re-import libraries that might have changed.  The quick fix is to always re-import libraries, the good fix would be to check their load times and their current file modification dates like a Makefile would to look for changes.

4. Love/Hate relationship with Functional Programming

GVR seems to maintain that Python is not a Functional Programming language, and that Functional Programming is not Pythonic.  This is despite the ‘Pythonic’ list and generator comprehensions (generators are a form of ‘lazy evaluation’), closures, first class functions and currying.  What, pray tell, would make Python a Functional Programming language if not these things?

But this odd schizophrenia doesn’t really bother me.  It’s the casualties that get to me, namely, and yes, this one is debated again and again, neutered lambdas.  Why aren’t lambdas allowed closures like normal functions?  Because Python isn’t a functional programming language.  This seems like an odd excuse given all the other support for FP Python has, and it seems altogether too convenient to use whenever the Python community wants to.  Closures are ok, because those are ‘Pythonic’.  Lambdas, no, because Python isn’t an FP language.  I feel like I’m stuck in 1984.

GVR has made some valuable, and some less valuable, critiques of FP.  One of the places where FP seems to muck up ‘Pythonicness’ was in ‘reduce’, which is still in the language, it’s just not built-in anymore.  Folds are a major part of many FP languages, but it appears for now they’re a little too terse for most people to understand.

One of the criticisms of lambdas, or anonymous functions in general, is that they get abused and grow to a size in which they should be a named function to add some level of self-documentation to them.  This is fine, and I agree with it.  Lambdas in Python probably ought to remain one-liners.  But why they don’t support closures doesn’t seem at all to be covered by this line of reasoning, and indeed, they really ought to support closures or just be taken out of the language altogether.  I’ve refrained from using them because I’ve been bitten too many times by odd errors.  Why would such and odd construct be allowed – in fact, lambdas DO capture variables, they just don’t close over them.  It’s one of the weirdest scoping errors I’ve seen in a modern language.

5. No Encapsulation

There are obviously many differences between the way Python does OO and other languages.  It’s incredibly unpythonic, for example, to write getters and setters.  Hell, that’s just bad OO altogether.  Python provides properties for those sorts of operations.  But one thing that Python could sorely use is some explicit encapsulation protection like private, public and protected.

I can hear the Politburo now harping away at how such things are not at all Pythonic, how a person should be able to modify and change whatever he or she wants whenever he or she wants it.  I agree.

But private, protected and public, as well as any other encapsulation protection mechanisms  you’d like to add, add a large level of self-documentation to an API.  It is one more layer of encoding designer intent.  The best case I can give you is using the dir() command on any sufficiently large object you get from a library, or even a module.  You get a few screen fulls of non-sense and implementation defined stuff that doesn’t concern  you, and then  you have to go hunting through this giant list of operations for the one you want.  How would Python incorporate encapsulation?  Could be nothing less than a keyword.  Pass in ‘SeeImplementation=True’ to the dir() command and you get everything, leave it off, and you get only that which the designer said was public.  When I’m doing exploratory programming, I really don’t want to explore implementation minutia which wasn’t meant for the user anyway.  I want to see what tools were explicitly given to me.  It helps with the information overload that becomes all too common when working with large API’s.

Much like FP, Python implicitly agrees with this statement, that a to define and communicate whether or not something is implementation or interface is important, via the _underscoreFirst coding convention.  But, in a language that strictly enforces a white space coding convention, at least compared to languages like C, it seems odd to suddenly fall back on just a ‘general and friendly agreement between the whole of Python programmers’ that putting a _inFront of your variable or function means its implementation details and not to be trifled with.  If it’s conventional enough, why not enforce it?  Why not allow some explicit introspection rather than hand coding some hack to look for a ‘_’ in the front of a variable to provide your own dir and other functions?

Things that are not broken, and I agree with. (In which I try and gain the forgiveness of the Politburo for my heresies before I’m to be shot)

Explicit self.  I actually love this, and it’s changed the way I code in C++ to scope everything with a this-> pointer to be clear.  It also makes methods and normal functions nearly the same thing, allowing you to more easily pass around methods just like you would functions (but this is NOT FP, remember, it’s Pythonic

Not allowing implicit TCO.  TCO, despite its name, is way more than an optimization.  As said before, it opens up an entirely new model of computation.  That’s hardly something I’d like to happen behind the scenes from a sufficiently smart compiler.  This is why I was an advocate of an explicit TCO via a keyword, although since I got branded a dirty little schemer (despite not knowing the language!), that idea got shot down by our good friends at the Kremlin.

Whitespacing.  You know how to get rid of all arguments about coding conventions?  By choosing one arbitrarily and enforcing it as apart of the language.  This and this alone has focused the Python community’s effort on much more useful things.  It also allows almost anyone to read any python code and be able to figure out it’s structure without getting a headache.  Readability does count.

No brackets.  Same as above – you don’t need explicit brackets if your white space shows the structure for you.

No doubt none of this will go anywhere.  But I thought, given the, er… ‘open-mindedness’ of my favorite programming language’s designer community, a little extra jabbing couldn’t hurt.  For my sake, let’s hope not

Just some notes for today.

Despite all the methodologies over the years, there are a few key topics that always re-present themselves.

• Iterative Development / Rapid Prototyping

The faster you can take a solution to a problem from just someone’s vague idea to actually testing it against requirements and customer expectations, the faster you can narrow in on a satisfactory system.

• Conceptual Integrity

The more clearly it is that a solution has come from one unified idea or mind, the easier it is to extend and maintain that solution so long as extensions and maintenence don’t interfere with the unified conceptual integrity that allowed their existance in the first place.

A post or two ago, I advocated a potential solution to adding explicit tail-call optimization(TCO) to Python.  In this forum, it seemed like a reasonably popular idea.  Hell, even reddit seemed to think it was somewhat useful, barring some grumpy old men who, to their credit, learned me a thing or two about TCO and continuations.  I proposed this to the Python ideas forum and it died.  I didn’t see why it could be so popular outside of Python, yet so dull inside.  Who was reading my blog, Schemers?

Really, the entire debate was a culture clash between Schemers and Pythonistas which I walked into blindly.

After reading the BDFL’s posts, and the counterposts, it seemed that it was a forgone solution that TCO was ‘useful’, just that it had problems being ‘pythonic’.  That is, easy to learn and consistent with the rest of the Python mental model (Ha!  Take that all of you who want a definition of Pythonic! (What’s that, you say, I’ve just moved the goal posts?  What’s the Python mental model?  Why, anything that’s Pythonic of course!)).  In a stroke of geniusluck, and idea occurred to me that many of Guido’s criticisms could be side stepped if we just made TCO not so much of an ‘optimization’ as an explicit part of the language.  I proposed an overloaded use of the ‘continue’ keyword, which, after having many debates on the color of the bike shed, would probably be better stated as ‘return from’ (to mimic yield from).  By making the syntax explicit, and checked, many of the perceived drawbacks would disappear.  The community offered solutions to the traceback problem and other details.

After submitting this idea to the Python ideas forum,  I was confronted with a new argument – that TCO is simply not a useful thing to have.  “There is absolutely no use case for it in Python.”  I mentioned the litany, that I had learned from others, of message passing, true unbounded recursion, state machines, even navigating websites.

First though, an admission.  In the practicality versus academic political leaning of languages,  I lean academic.  Whenever I see a new idea in programming, I want to try it and understand it even if I don’t have any use for it.  I have a love and passion for the stuff.  Thus, I came at the TCO debate assuming that if it could be put in the language cleanly, and there was at least SOME use for it, then the proposal should be taken seriously.  I mean, I still haven’t found a common use for coroutine generators, but that doesn’t mean there isn’t one.

In my opinion, there are three ways to solve problems in software.  There is breaking a problem down into smaller pieces (componentization), there is identifying similar patterns in the pieces and building them back up into a single construct (abstraction).  There is also a third way, one which is less practiced and closer to the ‘refactoring’ ideal, and is derived from how mathematicians solve problems.  You transform the problem you have into an equivalent one in which you already know the solution.  I felt like TCO was an example, at least in Python, of the final sort of approach.  The transform/refactor approach to problem solving requires you have a large ‘bag of tricks’ to work, since it is limited by the methods of transformation you know and the solutions you already have.  TCO was one of these solutions looking for a problem, I thought, and in time we’d forget how we got along without it all along.

I now want to talk about culture.  I gave a pretty bad defense of TCO because, to be quite honest, I don’t know much about it.  It’s a new idea to me, I thought I could add something to the community as an outsider to the debate via the simple explicit keyword solution.  I didn’t realize at the time that it wasn’t a debate over whether TCO should be in Python, rather it was, as frequently breaks out in our community, a religious war between programmers.  When I approached the Python community with my idea, some were skeptical, some were interested, but some were downright hostile.  To them, I was yet another dirty little Schemer in their midst trying to ruin their ‘perfect’ programming language.  I was to be told I didn’t understand the Python way of doing things, that in Python, you didn’t need the solutions TCO provided.  And after the BDFL advocated the use of the ‘elegant’ trampoline solution in his blog, that became dogma.  The trampoline was now to be the standard ‘Pythonic’ solution to the problems solved by TCO.

There was only one problem – I don’t know Scheme.  Not a bit of it.  I’m only loosely familiar with Lisp-like languages in the first place.  I understood the basics of TCO, and understood the arguments of those who argued for it, but they were not themselves Pythonistas, but Schemers.  I listened at the table of the Schemers, and brought back what I had learn to my home, Python, and was treated like a Leper.  There is a large underlying hostility here that I suppose I was too naive to believe existed.

Fast-forward (or rewind) to the recent Ruby on Rails uproar.  Two separate incidents that cause a bit of a question of just how ‘mature’ the Rails community is.  Take everything I say with a grain of salt (read the disclaimer on the right!) as I’m very likely talking about something I’m completely ignorant of.

But if we didn’t do that, very few people would say anything.

Anyway, first there was the porn thing, then Robert C. Martin using terminology like “masculine” to describe C++ languages (“a man’s language”) and “feminine” or, worse, “insipid” to describe languages like Java.  Ironically, and hilariously to his counterpoint, he was claiming to bring professionalism to software.  Here’s a hint, Uncle Bob, if you want developers to be a profession, perhaps we should stop insulting entire genders like a bunch of cigar smoking misogynists from the 40’s.  But that’s beside the point.  (I need another snifter of brandy, damnit!  And a sammich!)

After critiques of these sorts of talks took place, the Ruby community for the most part reacted like a ten year old child, claiming that it was really the people who were offended’s fault for complaining in the first place.  Good way to take constructive criticism.  This was all a microcosm and a side show for what really bugged me, and what has bugged me about many of the isolated programmer cultures that develop.   The rails community continued to more or less believe they had invented programming.  This was the underlying theme of the entire debate.  When outside criticism erupted, of anything, the knee jerk reaction was to distrust outsiders, circle the wagons, and shoot first.  After all, they don’t really know programming like we do.  There was no openness to criticism, which is indicative that there is little sharing of information at all.  When a new idea hits Ruby on Rails, it is assumed to have been invented there by those in the community (Read: Test Driven Development).  For the most part, they are the youngest members of our overarching programmer community, and should be expected to lash out like this.  The problem is, it isn’t just them and never was.  As I showed above, even more established communities like the Schemers and the Pythonistas can jealously guard their ideas and distrust anything new.

In the case of the Schemers and the Pythonistas, it was the old saw that Scheme is an ‘academic’ language and Python is a ‘pragmatic’ language.  Which would be untrue, if it even meant anything.  What the advocates of the language are actually saying is that ‘only academics’ use Scheme, but real programmers (pragmatic types, we) use Python.  Real programmers don’t have time to debate theory and angels on the heads of pins.  We’ve got deadlines and customers.  What could a “schemer” possibly teach us?  I believe this debate originally started, and is perpetuated primarily by the amateur programmer versus computer science major, basically the two ways to get into our field.

There is no ‘real’ split here, though.  For example, take Haskell, one of the most ‘academic’ languages and C++, never a more ‘pragmatic’ language (although, pragmatic in another sense, i.e., runtime efficiency).  Haskell has an idea called Type Classes that, oddly enough, do the exact same thing C++ ‘concepts’ do.  I doubt either of these two communities openly talk about how similar such ‘academic’ and ‘pragmatic’ languages are, because both secretly want to take credit for a structural typing idea that is sure to have been around since the 60’s (citation needed).  The point is, I can’t imagine a larger divide between communities, and yet there really is no large divide there.  The devil, quite literally as a force of hate pulling us apart, is in the details.

Let me finish by clearing up a few things, to anyone who will listen.  Just because you haven’t read SICP, you aren’t an idiot.  But just because you have read SICP, you aren’t necessarily a genius.  There are a whole bunch of good ideas and concepts in that book, like many of the books in our shared ‘cannon’ (I’m working through TAOCP, I swear!).  While I disagree with Uncle Bob that we might all one day be ‘professionals’ (or that we should be), we can certainly all be friends.  We are united by the fact that we all love and have a passion for this stuff, we should not let that passion for our particular solution, implementation or language divide us.  We have too much to learn from each other.  Python isn’t a child’s language (or, perhaps it is, but in a sense that even a child can learn it).  Haskell isn’t too dense (because, ironically, it has been taught pretty easily to children too!).  Scheme isn’t just academic, C++ isn’t THAT ugly.  And rubyists aren’t that immature.  Ok, well, some of them are, but we love them anyway   When it comes to programming, when it comes to what we do, what we’ve chosen to create, I’d rather have a friend by my side than a ‘professional’ any day, even if that friend is a dirty little schemer.

You’re reviewing source code. It is a relatively small ‘for’ loop in a procedural language, and it uses the ‘continue’ and ‘break’ statements. Suddenly one of your colleagues makes a critique – “Continue and break are tantamount to gotos, and therefore are hard to understand. You can simplify this loop by adding a few flags and only returning in once place.”

The author is notably flustered, “Adding in those variables will make this loop less efficient! Plus it will add needless lines of code and could introduce errors if we don’t check the flags right.” he responds.

“Your solution is too complex!” The original critic might say.

“But it’s elegant!” The author defends.

What is complexity? When is a solution complex, and when is it simple? When is it understandable? In the context of yesterday’s conversation, when is a language feature – continuations – make a language too complex?

By complex, we usually mean ‘hard to understand’. But as we’ll see, this is hardly an objective measure. We’ve seen one example of two different ideas of complexity in the argument above, but there are others. Some of these have been settled, so we’ll look at what Python has considered complex.

Nowadays, if you code up a huge for loop that operates on each item in a collection and builds a new collection, it’s generally recognized that a list comprehension is less complex. Why? A comprehension is also less complex if you’re building a new collection from an old one, but removing some members based on some predicate.

On the other hand, using another functional construct, ‘reduce’(fold), is considered MORE complex than the equivalent for loop that operates over the collection, except in certain cases.

Another example would be the general notion that simple logic and shallow nesting is almost always less complex than intricate logic and heavy nesting. Should you find these things in your code, you’re supposed to break them up into smaller chunks. Why would breaking something up somehow make it less complex – you still have to check all your flags and do all your loops? In fact, isn’t it even more complex now that everything is spread out all over the place?

So breaking things down, even pulling things away from their context is, counter-intuitively, a way to make a program LESS complex. What gives? You’re giving a person LESS information about the problem, yet they understand the whole problem better?

The fewer things we have to think about, the less complex they are! The less we have to keep in our head, the less complex it all is!

Think of your mind as having, like a computer, RAM and a hard drive. What happens when you fill up RAM? The OS steps in, carves out a place on the hard drive, and switches out some virtual memory. We, like computers, only have so much RAM we can use as our working memory. The more we have to hit our hard drive, the longer it takes us to do something. Plus, unlike computers, we’re not all that infallible when it comes to remembering things. We may do a look up of a certain problem on our hard drives today and come up with one answer, and then tomorrow come up with another. Uh oh, programming bug!

How do things like list comprehensions and loop control statements help us not hit our hard drives when we’re thinking about a problem? How does breaking a problem up keep things all in working memory?

The second statement should be obvious – the smaller our focus, the fewer things we have to keep in our heads. We divide one large problem up into multiple small ones, plus one more. The small ones are the modules we use to decompose the problem, while the plus one problem is the far simpler ‘big picture’ of how these small modules work together to solve our original problem.

The first statement, though, reduces complexity in another way. Think of breaking a problem up as ‘optimizing’ the problem for our mind’s cache. We think much quicker that way, and can keep everything in fast, working memory. There is another option we can take when we try and shrink a problem to fit in memory and that’s compression.

We like to zip up files because transmitting over the Internet is slow, so we want to transmit as little information as possible to the other side. Compression takes data and eliminates as much redundancy as possible to shrink it to its smallest ‘essential’ size. This identification of ‘essential’ also applies to our problem solving and programming. We can compress ideas by identifying them by their ‘essential’ structure, and ignoring all the extra details. This is the process of ‘abstraction’.

You see, ‘continue’ and ‘break’ abstract away a very common loop control structure. That means they allow a more complex loop to be stated in a simpler way. Instead of keeping the details of exactly how the loop will continue or break in our heads, we ‘compress’ that information away and simply mark those facts with the smaller, ‘essential’ nature of the problem.

List comprehensions do the same thing, they boil a big problem – a loop over a collection – into a smaller one that only consists of the ‘essential’ complexity of the problem.

But what about the critic in our example? Surely he knows that continues and breaks reduce complexity via abstraction. Er, don’t they?

Not necessarily. Compression in your mind, just like on a computer, takes time. It’s called ‘learning’. We see patterns, over and over again, and only after awhile do we begin to replace the pattern with it’s ‘essential’ nature (the parts that change). Only after we’ve seen something many times do we begin to think about that thing as an object in itself, and then begin to try and shrink that object into its most compact form.

The critic probably was not familiar with the use of continue and break, and thus saw the continue and break solution as inherently more complex. To him, continue and break didn’t boil down to small complex problems, but instead made the original problem even bigger. Now, in addition to understanding what the loop DOES, the reader has to understand these esoteric control flow keywords. That’s even more stuff to keep in your head!

And now comes the rub. I call this the Teacher’s Fallacy: You tend to project your level of competence onto other people. In other words, if you understand a subject, you frequently over-estimate how much others know. Indeed, we’ve all been in talking to a professor or local guru, asking a question and they start answering with jargon about three levels above the question we asked. They don’t realize they’re being incomprehensible. They don’t know what patterns you have or have not abstracted, and the problem is, once you’ve abstracted a pattern, you become blind to it. After all, how else is your mind going to keep all that information from clouding your RAM?

Similarly, if you do not understand something, you tend to project that level of misunderstanding on to others. The developer skeptical of ‘continue’ and ‘break’ sincerely believed that everyone else in the room also found their use as complex as he(or she) did. He( or she) most likely believed that the author of the code was simply trying to be ‘clever’, and obfuscate things in an effort to show how smart they were.

So, when is a potential language feature too complex to add to the language? People who aren’t familiar with the feature, but ARE familiar with the language, are likely to greatly over-estimate the complexity of the proposed feature. After all, we all get complacent with our level of mastery over things. Too often, we often see all knowledge as a single vast mountain, rather than a mountain range. In other words, we think “Well, I understand so much about programming, and this concept is complex to even me, therefor, it must be VERY complex to someone new to the language!” In actuality, you’re not looking at a new steep cliff near the top of your own mountain of knowledge, but rather a rather shallow grade at the bottom of someone else’s. You’re on equal footing with many beginners, but are so used to the thin air solving your own advanced problems that you don’t recognize where you are.

Indeed, if continuations and functional programming were, in fact, more difficult to understand than imperative or procedural programming, you’d see a trend of students taking far longer to master the former than they do the latter. But we don’t. There have been projects to teach Haskell, generally conceived to be one of the more obtuse academic languages out there, to high schoolers. And they were successful.

Functional programming seems hard because we’re already imperative programmers. Good ones too! We forgot how hard it originally was to wrap our minds around objects, around function calls, around variables and references versus values. When we approach functional programming, we generally do so at the level of an amateur (at least, those of us new to it), but we still believe ourselves to be experts. It will take your average smart imperative programmer about as much time as it would someone completely new to programming to ‘get’ Lisp, Scheme or Haskell. It’s starting all over again.

So, we’ve established that it’s very hard to judge the complexity of a new idea, because we’re so used to ‘getting’ it so quickly. We tend to over-estimate how hard it might be for someone else to grasp, since it’s so hard for us, and we’re, after all, the experts.

For someone arguing that continuations are too complex to put in a learning language like Python, really, the onus is on them to show that learning a stack-based function call mechanism is somehow intrinsically simpler for a person completely new to programming than a simple continuation based flow. Until then, I’d argue that it is simple enough for anyone to grasp. Even we expert imperative programmers

Another major question looms, though, and that is – whatever the complexity is, is a feature or solution useful enough to justify it. If, in our original story, we lived in an alternate dimension where loops were written so differently that really, ‘continue’ and ‘break’ only made sense in one or two cases throughout all code, then it would not be worth introducing those control flow mechanisms.

Learning is hard, recognizing new patterns takes time. I don’t want to learn a new language construct if I’m never going to use it. Then, even if it only adds a little complexity, it’s entirely needless complexity. Even easy things should be justified.

There are two kinds of abstractions, though. The first kind of abstraction is ‘evolved’. It comes from repeated exposure to a pattern until we shrink that pattern into its essential form. Once we have shrunk a few patterns, there emerge new patterns – patterns of patterns, and we continue to shrink entire systems of patterns down into their essential form. This process continues, and these abstractions build off of each other.

The second kind of abstraction is different. It comes out of the blue at us, from something completely unrelated. A flash of insight, maybe. These abstractions are ‘orthogonal’. They come at us from right angles, and are usually taught rather than learned through our environment. They are also used in a different way. As you recall, evolved abstractions come from obvious patterns – in other words, the use of them precedes their existence. We have a problem, and we find a solution, and now we can solve that problem again and again. Orthogonal abstractions go the other way, presenting us with a solution. We frequently see the problem they solve as contrived or unrelated to what we need.

This type of abstraction arises frequently in mathematics. We frequently prove theorems well before we find applications for those theorems. Math is in constant supply of solutions looking for problems, and the most frequent way to apply these new solutions is the transformation. Instead of seeing a pattern in our world and reducing it to its essential form, we see instead raw data. No patterns. But we can transform this raw data, this accidental complexity, into a form of which we already have a solution to. Mathematicians frequently make progress by stepping back an restating the problem.

We take a problem we don’t know how to solve, or one that has a messy solution, and we turn it into another problem that has an easy solution. But the only way we can do this is if we steadily expand our solutions, our ‘bag of tricks’, through other means than evolution alone. Evolution is never going to give you a solution you don’t already need, and such, your problems will always grow linearly with your solutions. Orthogonal abstractions are the only way to really grow your bag of tricks.

Continuations have a few uses that have been stated multiple times. But they still may seem like they solve a problem that’s almost as easily solved in some other way. Are we just tacking on something to the language that doesn’t NEED to be there? The Zen of Python is that there should be preferably one way, and only one way, to do things. This helps the learner of the language, because when they discover a new feature, they should immediately be able to find applications. If there is one way to do something, and a feature is in the language, then it’s abilities aren’t replicated by any other, and when you run into a problem it solves, then you’ll need it and only it.

But just as we said the imperative expert will misjudge the complexity of learning functional style for the newcomer, I think we’ve shown that we all are going to underestimate the usefulness of orthogonal abstractions. They’re solutions looking for problems – why should we tack that on to the language? In fact, such things may even be risky. A solution today with no problem, but tomorrow we may find it solves the same problem as something we already have. It could only be needless today, but tomorrow it could be downright redundant. That is not Pythonic.

There is a debate in economics over whether demand drives the economy or supply. Demand siders say we should give people what they want. They are clearly indicating what problems they’d like solved, and we evolve solutions to those problems from stuff we already have. Supply siders say that if you build it, they will come. People frequently don’t know what they’re missing, or what they need. Take the introduction of the personal computer, for example, the archetype for a completely orthogonal solution. No one demanded a computer, no one even knew what they were. Many were skeptical of their usefulness, yet today, obviously, they are ubiquitous.

In terms of continuations, an extended continue syntax, as an orthogonal abstraction, could very well solve problems we don’t even know we have. Stackless Python has shown the potential of a Python with no stack, while continuations may be able to work that power into a language with the safety of a stack. Native thread advocates constantly bemoan the GIL, yet continuations may have the potential to make efficient, expressive green threads a reality in Python, with very little legwork. We will learn to solve many problems today by transforming them into problems continuations solve far more simply. They are coming at us at right angles, and we never know where that will lead.