Conway’s Corollary

Conway’s Law states that a software’s architecture will inevitably resemble the organization that produced it.  An example being that if you have four groups of people building a compiler, you’ll get a four pass compiler.  Well, I posit that the opposite is true as well, that given any sort of software architecture, there is an optimal social organization to build it.  This seems trivially true, but at the same time, gives us some insight into the software ‘engineering’ process, or project management.

Right now, there is a strong inclination in software development organizations to have two chains of command, a ‘people’ chain, and a ‘technical’ chain.  The ‘people’ chain, commonly derided as ‘management’ in general, tends to deal with the contracts, the hiring, the firing, the personnel reviews and other ‘business’ stuff.  The technical chain decides how to architect a product, what technologies to use, how to implement it, and other more nitty gritty details.  This is not really because the separation of people management from technical management is a good idea, but because it is so incredibly hard to find talent in both fields.  Your most technical people tend to not get along with others, while your most social people tend to balk at technical work.

The problem is that work is broken down by the management side, not based on any sort of architectural guideline, but based on manpower and resources available.  The resulting organizational structure will be far more driven by office politics, historical relationships and other business partnerships than what is ideal for the product.  And due to Conway’s Law, we can suppose that the resulting organizational structure will most likely have more impact on the product and project than any other decision, technical or managerial.

Software development is, poetically, much like software itself.  The Mythical Man-Month states that one of the inherent falsehoods repeated in software engineering circles is the idea that effort equals productivity, and that measuring effort in man-months confusedly implies that men can be exchanged with months.  This is altogether too similar to another problem currently faced by the computer science world: concurrent programming.  Throwing another processor at a program will not necessarily double it’s speed.  In fact, attempting to make some things concurrent can slow you down when the problem is inherently sequential in nature.  The trick is decoupling processes that can be effectively completed in parallel.

The same truth can be said of software development.  There are some inherently sequential tasks, tasks that must be done step by step, and thus really can only be done by one person.  Splitting up these tasks results in less work done, as now two people can at best, go no faster than one person (if they were both to simply solve the problem independently) and at worst, slower due to communication overhead.  The trick to utilizing the most manpower for any project is to find the optimal number of ‘threads’ that run concurrently throughout the project – that require the least amount of communications overhead naturally.  Then that is your number of developers you can use.  If you try and find any more concurrency than this point, you’re going to be drowning your developers in meetings and other communications overhead.  The project will cost more, and take longer.

If we take Conway’s Corollary, that the best software architecture necessarily has a best organization to develop it, then we realize that project management cannot begin until primary architecture is in place.  This architecture must be done entirely from a technical point of view, since any manpower concerns will necessarily trick us into thinking we’re optimizing for our resources.  Conway’s Corollary says we are not.  The ‘best’ architecture can be split up into so many components, and so on subdivided until we reach a point where the sub-components are too coupled with each other to yield any more real concurrent development gain.  This, then, should be used to develop a project plan, estimate manpower and resource needs, rather than the other way around.

Conway’s law says that organization drives architecture.  If we turn it on its head, and try letting architecture drive organization, we might find out why what took the generation before us ten people and a time-share mainframe takes us now teams of dozens of developers with the best equipment.

Tying your hands behind your back and loving it

Small post today, but a thought.

Sometimes, for many problems, the best way to approach the programmatic solution is to code/specify the solution in the language you wish you had, and then implement that language.  This is the DSL approach to programming.  The best poster case for this sort of programming is Lisp and Forth, primarily because they are so easily amenable to the meta-programming required.

As a side note, it’s ironic that the DSL approach to programming isn’t more widely adopted in corporate and engineering culture.  It’s just about every other day some new company is promising a new way to codify system requirements to help in the software engineering process.  Seems like the solution they’re approaching asymptotically in the ‘enterprise’ field is to code up user requirements in a programming language itself.  But I digress.

DSL’s are incredibly useful from a usability perspective – frequently, a domain specific declarative language can specify something in far fewer lines than the equivalent fully fledged programming language can.  I recall a presentation I saw that, unfortunately, I don’t have the time to link to, that described Adobe Photoshop’s code being 90% publisher-subscriber patterns trying to run an overall MVC such that any new command you use is instantly reflected in the underlying picture.  That means, as a good estimate, 90% of their bugs exist in that code too.  The speaker then explained a small, declarative language he developed to capture most of the same usage of Photoshop.  The code was far smaller, maintainable and readable.

But, a frequently overlooked side of DSL’s is that they limit you.  In many cases, DSL’s are not Turing-Complete, meaning that there are tasks that any particular DSL simply can’t solve.  Believe it or not, this is the best thing they have going for them, though.  A language that’s Turing-Complete can express any computable algorithm, but it also suffers from some drawbacks, not the least of which is the Halting Problem.  The idea is that, for any Turing Complete language, one can never prove whether any arbitrary program halts in that language or not.  In layman’s terms, we can’t detect infinite loops.  That’s not all, though, as many problems can be reformulated in terms of the Halting Problem, meaning that they too are undecidable.

Do we need a Turing Complete language, though, to describe a GUI?  Or a MVC framework?  For most of our work, we can sacrifice a tiny bit of power to step back into weaker computation mechanisms.  In fact, when you really think about it, the only problem we really need a Turing Complete language to describe is the problem of what non-Turing Complete language we really ought to be using.  I’ll get back to this in a second.

If a language is not Turing Complete, that means we can prove far more useful things written in that language.  That means program proving mechanisms become a much more useful way to eliminate and track down bugs and defects.  None of this is to say that we can’t prove a lot , or at least, assume a lot about programs we write today – it’s very impressive just how far static analysis can go based mostly on heuristics.  But the fact of the matter is, in simpler constructs, we can prove far more.  And when I say prove, I mean prove. Unit tests, while very valuable for defect detection and development, can never prove bugs absence, just their existence.  Via proofs, in a language that is limited enough that can be constrained in this way, we can prove the absence of certain bugs.

What does this mean insofar as development is concerned?  It means that perhaps, once size should not fit all.  It means that we can tie our hands behind our back with DSL’s and love it, as long as we have a variety of DSL’s to do every small job for us.  We now have constrained where defects can occur to the Turing Complete languages we must use to implement our DSL’s and the glue code we must also write to have our DSL’s interact.  That’s what I was talking about before – perhaps the best role for our powerful, Turing complete languages is not to solve problems, but to describe less powerful languages that solve those problems and also prove things about those languages.

A language for GUI’s, a language for events, a language for this and that.  We already are used to this sort of thing, we just call these sorts of things libraries.  Learning a new DSL, since it does not have to be any where near as expressive as a fully fledged programming language, need not be any more difficult than learning any new API.  Using this approach to development, not only can we begin to reap the huge productivity benefits of using declarative, specific languages to solve specific problems, but also reap the huge quality benefits of relying more on automated proving mechanisms rather than just tests.  This quality benefit itself turns into a boost in productivity, since most of our time gets caught up in chasing down defects anyway.

This sort of separation of concerns can already be seen in a language like Haskell.  While Haskell has not demarcated a certain part of it’s core vocabulary as Turing Complete and another part as somehow less powerful, it has taken the important step to separate pure computation from more dangerous things like I/O via it’s monad mechanism.  As an example, then, I imagine it’d be far easier to prove something about pure functional Haskell code than even in another functional language.  This drastically reduces the amount of program testing and manual verification we humans have to do, and the language already lets us know where to focus our effort – state driven, I/O code.  This sort of separation of concerns can continue such that each time we delve into a specific way to solve a known problem, we already have a huge safety net ensuring quality, even before tests.  Combine this with the expressiveness of the same specific language on the specific problem, usually reducing the amount of code by as much as a factor of ten, and you can almost see, if not a silver bullet, a good, solid lead cannon ball for development and developers.

Convergent and Divergent Thinking

I read this article in the Times today, and began thinking of how apt a metaphor it was for software.

Not to put words in the author’s mouth, but to sum up, human thinking seems to fall into two discrete camps, what they call divergent and convergent thinking.

Convergent thinking is basically attentioned, focused thinking, where the mind is applies reduction rules to a problem.  Doing a math problem is a good example of convergent thinking, the brain is applying a mechanized system to some problem, where the answer is basically just a number of steps from the problem, and we know to go to B from A, and C from B, and so on, to the answer, it’s just a matter of doing the work.

Divergent thinking is what we do when we don’t know the answer, when we don’t know the next step, when the problem is completely open ended.  Another common problem that you solve using divergent thinking would be a word puzzle like Wheel Of Fortune.  How do you solve those puzzles?  Do you methodically plug in every letter of the alphabet into the puzzle until one fits?  Obviously, even a trivial puzzle could have you calculating for days using that method as it explodes in complexity.  Instead, you stare at it, relax a little, and the answer occurs to you.  It’s a eureka moment.

We’ve all had stories of spending all day working tirelessly, and fruitlessly, on a project, only to have the answer occur to us right as we drift off to sleep.

As software developers, we’ve basically come to believe the false notion that what we do is convergent thinking.  We’re lead to believe that, because ultimately what we’re working on is algorithms and numbers and bits and bytes, we need to use the same strategies we’ve always used for those technical problems.  Indeed, the vain hope with different software methodologies is that the ‘one-true-process’ will emerge and allow anyone to produce what they want given some specified requirements.

We don’t just need experience to tell us the folly of this expedition, now we can look into human psychology.

I would posit that, if anything, the arts, like sculpture, painting and drawing are fundamentally divergent activities.  Certainly an artist must have a measure of ‘technical’ skill, she ought to be able to draw well.  But just drawing well does not a work of art make – she needs to also have a flash of insight, to be inspired.  Otherwise what she will produce will be drab and lack quality.  It will be unmotivated.  This is not specifically tied to divergent thinking, but quality is something I’d like to go on more about, but at a later time.

Anyways, to produce a work of art, one must have the technical skill to do so, but also the flash of insight to know what to produce in the first place.

Software, while having some ‘engineering’ aspects to it, is fundamentally a creative pursuit.  This is not any sort of wishy-washy statement, but a simple conclusion from logical deduction.  Yes, it’s ironic that we’re using logic to show that straight out logic won’t suffice in the end, but that’s beside the point.

The fundamental problem solving technique we developers use is abstraction.  Upon finding a solution to a problem, we identify what can vary, abstract out the solution, and reuse it.  This is the basis for libraries and all other code ‘reuse’.  The point being, if there is ever an ‘algorithm’, or actual pattern of work that is done, some sort of mechanical process, it is within our power to abstract that pattern out and reuse it as a primitive.  Despite what the GoF will tell you, patterns CAN be generalized and pushed into languages and libraries.  Hence, in the ideal development environment, you won’t be faced with reimplementing some already known technique or solution, but instead will have those solutions already at hand.  What you will be doing is using those already known solutions in a completely unique way that’s never been tried exactly like this before.  You now have an open ended problem, for the sheer sake that all the problems that convergent thinking could do have already been solved for you by people before you.  And once your solution is complete, it too will fall into the hall of convergent solutions, and be reusable.  Hence, the only real work in idealized software is creative work of taking the solutions we already have at hand and putting them together in a completely new and unique way.

That ‘proof’ required a lot of hand waving, but I think you get the point.  Common sayings in software like DRY and KISS exist as words of wisdom because we should not be doing mechanical work as software developers – that’s what the computer is for.  We should focus on creative work, and let the computer do the rote stuff.  Most problems in software, I suspect, come from either the false belief that creative work can be shifted into mechanical work (and our endless supply of methodologies is evidence of that) or that mechanical work should be redone as creative work (and reinventing the wheel, or not-invented-here syndromes are evidence of that).

As the article at the top implies, creative work is best done divergently.  And creative work is best solved by not trying to solve it at all.  Know the problem, understand it in detail and research it.  Investigate other attempts at solutions and have discussions, but never think you can just sit down and mechanically come up with some sort of solution.  Inevitably, the ‘ideal’ solution that ought to be a one-liner will blow up into thousands of lines of case statements as you learn more and more about the problem.  As we know, the best predictor of number of bugs is number of lines of code.   Less is definitely more.

Back to my post on new hires being driven by company culture to simply throw as many hours at a problem as possible, we can now see how mere hours won’t amount to much.  Psychology has shown that while our ability to solve problems requiring convergent thinking decreases as fatigue increases, it does so at a much slower pace than our ability to solve divergent problems.  Sleep deprivation saps creativity, gumption and quality in one’s work.

In fact, any sort of ‘drive’ to find solutions mechanically will cause work to suffer.  Many times, when trying to figure out how to design some sort of extensible framework, the BEST thing we can do for ourselves is to stop working on it.  Sure, you can do the ‘moral’ breaks like taking walks, or a corporate-approved power nap.  But ‘immoral’ things would work too – surfing the web, reading blogs, chatting, playing video games.  All of these things improve morale and give your divergent mind a chance to churn away at the real solution.

I know it’s against our protestant culture and mindset to ever believe there’s virtue in laziness, but I maintain that all great insights occur when we’re not even looking for them.  So to increase your chances of finding those insights, you best stop looking for them as soon as possible and let them creep up on you.  You may end up ‘working’ less, but getting a lot more done.

The Most Important List Your Manager Will Never Read

There has been no silver bullet in software development, but there have been a few dozen good lead ones that I’m finding it more and more hilarious that go completely unheeded.  Whether it’s skepticism from management about new development practices, or skepticism from developers themselves because some new technique violates a dogma that they held in particularly high esteem.  This second type of hold up is particularly dangerous since it usually results in a developer shooting themselves in the foot just because they refuse to learn something new.

I’ve decided to put together a list of five of these lead bullets that if you aren’t familiar with them, you really ought to make yourself so.  These are things that really can aid in  your productivity, but not in any sort of slave driving way.  They’ll help you ENJOY programming again, and as such, should be important to be perused by managers as well as developers who want to institute practices to get the most out of their developer’s potential.

I would also like to hear from you as to what YOU believe has really helped in your productivity, since this list won’t be in any way exhaustive and will of course suffer from huge biases on my part 😉

And so, to begin…

1. The REPL Loop

Most languages people still use today have a familiar code-compile-run cycle born out of the roots of punch cards, but it need not be so.  REPL, standing for ‘read-evaluate-print-loop’, institutes a type of interactive programming that facilitates incredibly fast turn around compile times and ‘live’ coding and testing sessions.  REPL was instituted first by LISP, however, many other languages have some sort of REPL today.

Adding a REPL window to your task bar that normally has your IDE, text editors, etc., running will give you a quick place to build those ‘dirty’ programs to test out a new idea or concept, or to refresh exactly how some API works.  The best REPL capable languages assume this amount of interactivity in the built-in API and provide functions to interactively explore packages, like a Help() function that can take in a method, package or object and tell you what you can do with it, what it does, what traps you should avoid in it’s use, etc.

2. Dynamic Typing / Type Inference

These are two related technologies that really yield completely different type systems, but in the cases of productivity, tend to work in similar ways to the developer and provide the same development boost.  Compare this:

vector<int, myAlloc> x = make_vector(objects);

to

x = make_vector(objects);

It seems like a small difference, but allowing the developer to ‘forget’ type information drastically cleans up the code.  In dynamic typing systems, this can open the door to certain types of bugs that static typing prevents, so usually dynamically typed languages rely more on unit testing to catch these bugs.  But in a language with type inference, even light weight type inference that will be added to C++0x via the new use of the auto keyword, this takes a huge cognitive load off of the developer that the compiler should have been doing anyway.  Type inferred languages allow all the benefits of static typing but none of the insane verbosity, of which Java’s probably the worst offender.

Make no mistake in the dynamic versus static debate, dynamic typing is more productive, while static typing, on average is safer.  The point being, when you can, you should go dynamic where you want and static where you have to.  Few languages without extensions or some mixture can use this combination easily, however, it’s a maxim that’s pretty well understood.  Type slows down productivity as a trade off for finding more bugs.  Type inference is a happy medium.

3. Object Oriented Programming

This is no major surprise.  But let’s go ahead and call this a tie for third with something that’s less well known among mainstream developers and managers.

3. Functional Programming

Object orientation is useful because it breaks up the cognitive load on the developer.  Information hiding allows the author of an object to tell the user “this is what’s important to you”.  It drastically cuts down on the amount of mental modeling the user has to do to have an object.  It descends from modular programming, both of which are useful not only for program modeling and understandability of a program, but also for managing a program.  Having one team implement one module, meeting a specific interface, while another team implements another module, is a very nice way to break down tasks and features without the two teams stepping on each others toes.  Object orientation just takes this a bit further with some more bells and whistles.

What is not acknowledged though is that many of the more verbose ‘patterns’ that arise in Object Oriented languages, again Java’s a big contributor here, should actually be concerned more as work-arounds than true solutions.  Functional languages like Lisp and Scheme allow the passing around of functions as first class things, as opposed to the way C and C++ allows you to pass around function pointers.  You can achieve some level of functional programming in C and C++ via the use of the functor idiom, however, like I said above, this is just a workaround for a missing feature.

In many cases, what may take 10,000 LOC in a pure OO language can be put in a mere 1000 lines in a functional one.  Functional languages do better to capture certain features and patterns that are implemented over and over again, like different things one does to a collection, as well as clever ways to capture state without resorting to objects.  Then again, there are functional programs that are put much more clearly and succinctly in object oriented style.

The two are not competitors, but supplements, and having developers that are familiar with both methods will allow them to take advantage of times when ideas can be expressed more clearly in either of the two ‘paradigms’ styles.  This results in less work on the developers part as they are not fighting against the language, say by introducing about a dozen different objects to implement the ‘strategy pattern’ when really, all they need is a passable function.   Or alternatively, implementing a huge object hierarchy that holds certain bits of state when all that was really needed was currying and closures.

A third competitor for this tie might be considered metaprogramming techniques which allow for easily constructed domain specific languages, which I would believe to be the next ‘evolution’ in paradigms.  But for now this should probably be considered ‘too hard’ for average developers since the language support just isn’t there in most cases.  Yes, LISP has had this sort of thing since the beginnings, decades ago.  But let’s face it, for some reason or another, LISP has never caught on as a mainstream language that most managers and developers who cling to their Visual Studio will trust.  I don’t know why.  And perhaps that will change in the future, but until then, or until a metaprogramming is easier in the current batch of popular languages, just having a java developer think in terms of map, filter and reduce should make us happy enough.

4. IDE’s / Text Editors

Another obvious one… but wait, both?  No, obviously IDE’s are the modern approach to proper tool integration some might say.  But others still cling to their ‘old’ VI and Emacs like the dinosaurs they are.

Basically, the lesson here is do whatever you’re LESS familiar with.  Text editor aficionados laugh at IDE users because they require so much ‘framework’ to get going, while IDE users fear the spartan interfaces of the anciently inspired range of text editors.  Both of these are pretty useless stereotypes.  Text editors can easily be extended to support every feature even the most modern IDE has, from debugger integration to build cycles and more.  Likewise, IDE’s should be thought of by Emacs and VI users as a pre-configured text editor with tabbing, code folding, auto-completion, etc, already set up.  Admittedly, a heavy-duty text editor user won’t find as much going to a mixed environment as a heavy duty IDE user – from a developer perspective, text editors give you all the power of IDE’s with a ton more extensibility.  A good .vimrc fits a developer like a glove.  But, from a management perspective, providing a clean, intuitive and easy to use IDE is obviously cheaper than training everyone on the intricacies of maintaining their own text editor profiles, and as such, technical managers that themselves rely on text editors probably need to experience the world of Visual Studio and Eclipse from a cost performance perspective.

Managers and developers who do all their coding in VS or Eclipse would do better to become familiar with one of the more popular text editors.  A heavy IDE user probably won’t get much of a thrill out of setting up IDE conveniences like auto-completion and what not in VI, but will become more productive by not having to rely on an IDE that takes up half of their memory for every minor programming problem.  Like the REPL, you don’t always need a solution file and a 104 step wizard just to build a program that plays with some API to learn more about it.

Likewise, while text editors are nearly infinitely personally extensible, they are not targeted by tool manufacturers like IDE’s are.  There are dozens of high quality, very useful integrations and plugins for the major IDE’s that are hard if not impossible to fully replicate in a text editor.  On-line static analysis and integration with bug and issue trackers name a few.  Certainly, these things can be done in text editors if the tool providers have provided a scriptable command line interface, or, instead of figuring out your own Emacs Lisp scripts to make these things work in a one-off fashion, you can just bite the bullet and use their offered integrations to get these features up and running for the majority of your developers in about five minutes.

5. Frameworks

Again, this one should be an obvious productivity enhancer, but time and time again developers end up forgoing a library based approach and would rather roll their own.  This happens for a few reasons.  Some developers are simply ignorant that a standard library already provides what they are trying to do.  Others arrogantly believe that what they can accomplish in a few hours of hacking will naturally be more efficient or safer than what it’s taken 30 years of mature Fortran back end to do.  A good example of the second is why every single site seems to want to roll it’s own linear algebra code, even though there is freely available insanely fast libraries already available.

From a developer perspective, it is our job to educate ourselves on the libraries availed to us by our languages and environments.  This means whenever we run into a problem we don’t know how to solve, we should FIRST research the problem to make sure no one else has not already solved the problem either using built-in or liberally open source code that we can use.  Remember the maxim, “Good programmers write, Great programmers steal.”  It does you no service to reinvent the wheel – and even though libraries have been around since almost the first languages with punch cards in the form of common subroutines, we STILL manage to try and reinvent solutions to common problems almost every day.  Even if an API isn’t ideal, it doesn’t do things in exactly the way you’d like, you still need to weigh whether you can be more productive with a slight learning curve + a good library, or just writing the whole damn thing from scratch.  In most cases, learning a new library is better.

From a management perspective, managers need to stay on top of all the tools they can provide their developers.  I know some sites are very skeptical of even liberal, non-viral open source licenses.  While developers’ anti-pattern with regard to libraries is “reinventing the wheel”, managers play their role with the classic “not invented here” syndrome.  If you honestly don’t trust outside code, if it is open source then you can still use it within your current review and testing processes.  Simply work with your developers to figure out which parts they are using, and treat that code as if you have produced it – review the source for errors, test heavily used parts, etc., treat it like legacy code.  Only most open source code is far better written and with far fewer bugs than legacy code.

The first lead bullet ever developed to improve software productivity was the idea of reusability, and it’s a shame we still haven’t gotten the picture.  Generally the most elegant and efficient solutions to some of our most commonly run into problems are already captured in well tested, highly used libraries.  What we end up producing in their stead is ugly, slow, unreadable code that every time it’s reviewed, has another boundary case exposed as not dealt with.  In other words, whatever you’re working on, it will never have as low of a bug count as the C++ STL or Java API’s, so if you’re rolling your own based on skepticism of others code, you’re doing it completely ass backwards.

Like I said, this list is hardly exhaustive.  And most of these things listed here we’re all already familiar with, at least a few of them.  The point is that these are things we need to be intimately familiar with, and more importantly, those making high level technical decisions need to be intimately familiar with, if they hope to compete in producing code fast and as bug free as possible.  Our abstract factory builder strategy patterns use ten thousand lines to do what could be said in ten, while the code-compile-run cycle turns what ought to be a five second endeavor into a five minute one, which is a growth of over an order of magnitude slower.

There is no one language or environment to rule them all, so mixing and matching should not be discouraged.  Some might balk at dynamic typing, for example, as far slower than statically typed and compiled languages.  Ok, then dynamically type most of your code, then go back in and rewrite the slow stuff in C.  It’s a tried and true approach to getting 99% of the developer speed and processor speed of both worlds.  Similarly, a REPL in a language that’s specifically not being used can still be used to prototype approaches and algorithms for another language.  Prototyping how a system ought to work in Python or Lisp will catch most of the high level design flaws much earlier than waiting to have a full C++ system rolled out, and it will be more malleable too.

The bottom line is that while there is no silver bullet, that does not mean that we don’t have a responsibility to use all the lead ones we’ve accumulated over the years.  Reusable code is decades old, yet we are still rewriting matrix algebra packages.  Why write a set of scripts to integrate Emacs with your latest unit test tool when there’s a plugin for eclipse?  Likewise, never, EVER rely on cut-and-paste for jobs an Emacs macro can do more safely and extensibly (luckily, there are in fact some integrations of Emacs and VI key bindings with some popular IDE’s.  Combining the two approaches in this way might be interesting, although I personally have not tried it.)  It’s a simple principle that the less we have to type to implement a feature, the faster we’ll implement that feature, but it seems to be ignored, even today.

MDA is dead

MDA is dead, and IBM killed it.

Model Driven Architecture is dead, and the so-called champion of MDA is it’s killer. A craftsman is only as good as his tools, and due to the utter incompetency of the chief toolmaker, an entire way of developing and designing software is being relegated to the same dustbin that other such buzzwords as the “Rational Unified Process” are.

“As a great and wonderful agile developer”, you say, “how come I’ve never hard of this MDA? If it’s so great, why haven’t I used it, since I am always using the ‘newest thing™?’ Well, the answer is you probably have been exposed to it, and it’s exposure sent you into knee jerk convulsions of architecture astronomy. Find your nearest ‘Software Architect’, and ask him to explain to you the wonder that is UML. Apparently, if we draw our pretty pictures just right, the software design will just fall out. That’s MDA, using UML to create models of your software as apart of the design and development process. And despite your justified sense of nausea upon understanding it, it does have a great deal of promise which, unfortunately for you, oh great Agile developer, will never been shared with you.

If you want to know how MDA(or MDD depending on which Pointy-Haired-Boss(PHB) buzzword you want to use today) works, and be happy with it, then good God, never, EVER look at the UML standard. UML just happens to be MDA’s poor retarded younger brother who got a job with him because his mother made him feel guilty. No, instead, if you want to understand MDA, look at lisp. Or C++ template metaprogramming. Or any sort of ‘programs-that-build-programs’ approach. UML just happens to be the langua-franca of MDA, but the heart of any solid MDA approach is using the models you develop to auto-generate code. Some Architecture Astronomers out there might try and convince you that the models in and of themselves have worth, but I’m sure by now you know better. As a documentation tool alone, they are sub-par, but workable. What really makes models shine is their ability to generate code, and turn our pretty pictures into an exact replica in actual, fieldable code.

This could work amazingly well with many other modern ‘methodologies’, like agile or TDD. UML basically represents the code-base as structured text(and pictures!). It allows for comically easy major-refactorings, freeing up your coder’s time from skeleton/boilerplate work, and pretty pictures to speed up new guy’s familiarization with the system(as well as automatic documentation and something to impress your customer with…) Auto-generation alone won’t get you integrated with modern approaches like TDD, instead you need both forward and reverse engineering capabilities – transforms that take a model and turn it into code, and transforms that take code and turn it into a model. This allows your architects and coders to work in the same code-base, the coder looking at code and the architect looking at pretty pictures, and neither is none-the-wiser.

But therein lies the problem – hackers and architects do not want to be none-the-wiser. Even if you can show both camps how they can easily work together, respect eachother, and not be so damned arrogant about their differing approaches, they won’t do it. Instead we run again into one of the deepest and most ugly underlying human anti-patterns: People would rather be right than get things done. If a product doesn’t ship, but the designers can claim that their process was flawless, most people would be happier than if a product did ship, but those people’s feelings were hurt. Hense we get these huge rifts between the designing and developing communities, and just as huge amounts of rework as architects attempt to accomplish something in pictures that ought to be in code, and coders completely unaware(and tripping over themselves as a result) of good design practices.

For some reason, we coder’s let MDA get gobbled up by the architects, and since the architect’s try their best to never code, the code auto-generation(that is, the magic beans that make the whole endeavor worthwhile) are stuck between two divorced parents. And since Mom thinks we’re over at Dad’s and Dad thinks we’re over at Mom’s, and we(code auto-generation) are actually out smoking pot with our friends, we’re the one who suffers. Architects continue drawing pretty pictures that have virtually no worth, and hackers continue barely scraping by any major system.

When architects go into their ‘architecture conferences’, they hear these sales pitches for tools like IBM’s wares. RSD, RDA, R-something-something, basically are all the same package, and since IBM’s the biggest pusher behind MDA, you’d think they’d be a top notch shop. But they also know they don’t have to sell their tools to the coders, they just have to sell them to the architects. So as long as the pictures look prettier than everyone elses, it doesn’t matter WHAT the tools actually produce. Besides, isn’t it the coder’s job to manually and demeaningly translate the architect’s design into code? Who needs code auto-generation when you have cheap new-hires?