November 27, 2005

Down the upstream

Writing software, bright computer scientists will tell you, is an intensely creative process. Much like writing short stories or novels. If you're good, you will express your ideas clearly and logically, and that applies to software or stories. So well-written code (software) is often called "clear" or "innovative". Or, best of all, the adjective all software writers quietly aspire to: "elegant".

There's another side to that coin. Computer languages are so expressive and powerful that you always have several different ways to accomplish the same thing. It's easy to know a language inside out and still turn out what is derisively known as "baroque" code. (I have first hand experience).

But if you use all the nuances and power of the language, and produce code that's just like a-ringin' a bell -- take a bow, you're writing elegantly.

And like with other creative processes, attempts to regiment the writing of software tend to produce mundane code. Imagine following a list of guidelines while writing a novel. What would you get but a dull, boring tome? Do you think the inspired prose of a Vikram Seth or a Dom Moraes or an Amrita Pritam came from The Cookbook of Creative Writing (TM)?

Yet a whole slew of people work to stifle creativity in software. They seek to cloak software design in a fog of regulations and guidelines. These folks share two characteristics. One: they never actually write any software. Two: a god-given talent to use a hundred words to say what the rest of us would manage in ten. (Or, preferably, not say at all).

For two years in the mid-80s, I watched a group of these people first-hand. Their mandate was to consider software design and produce a computer-based model of it. Why was this desirable? I have no idea, but the people in this group -- they called themselves software engineers -- went at it like it was earth-shatteringly important.

The popular notion among them was that software design was like a waterfall. (Yes). Thus there were some who studied the "upstream" part of the process. Naturally, others studied the "downstream". Not content with that much, others decided that the "up-upstream" was where their interests lay. Naturally, this spawned the "down-upstream" people.

If you think it didn't stop there, you're right. If you're also asking "What does it all mean anyway?", the answer is "nothing."

But they were paid well. And they turned out documents by the gross. For software engineers -- and these were some of the best known -- are adept at turning out documents. At writing about writing software.

They didn't, however, write any software. In the eight years this project lasted, it produced one software product of note. That was the effort of a team of four who held a healthy, barely secret, contempt for their colleagues. Following no particular rules -- certainly none that the documents contained -- they took five months to turn out a robust, useful, even elegant piece of software.

A lesson for the "up-upstream"-ers? Hardly.

In the '90s and later, we've been plagued by a whole new field of play for these guys. It's called ISO 9001, and its faithful will tell you it aims to improve the quality of software. (All software is poor quality, didn't you know?) To persuade you, they produce lots and lots of -- you guessed it -- documents.

Yes, if ISO 9001 is characterized by just one thing, that would have to be the kilograms of documents it generates. They come complete with official looking numbers and designations and warnings, so you start thinking they say something worthwhile.

Like one that I have, "Guidelines for the application of ISO 9001 to the development, supply and maintenance of software." It begins with this vital announcement:
    Use of the masculine gender in this Guideline is not meant to exclude the feminine gender where applied to persons.

As any writer of software will tell you, he -- without meaning to exclude the feminine gender where applied to persons -- worries about exactly this while doing his -- without meaning to exclude the feminine gender where applied to persons -- job.

And what's inside this document? Endless lists of points about writing code. Then you read them.

What, for example, should you keep in mind while making copies of your software?

Paragraph 5.7.3.1 (a) has the answer: "the number of copies of each software item."

What, for another example, should a "maintenance plan" for your software include?

Paragraph 5.8.2 (d) has the answer: "Maintenance activities."

24 pages of this stuff. The waterfall guys would salivate. (Or maybe they wrote this).

But you, you would do well to wonder: why isn't there an ISO 9001 for creative writing?

5 comments:

Abi said...

Elegant code is all nice; but ask the guy sharing the cubicle to maintain vikram_Seth kind of code after Vikram Seth has left to write his magnum opus. You can soon start looking for another person, because this guy has either left you or committed suicide. ['Guy', when applied to persons, here refers to someone of either gender!].

Maintaining software is something that salman_Rushdie doesn't have to worry about. And you only need a few dom_Moreas's to keep the entire world happy.

Software needs people -- millions of them -- who can produce competent, maintainable code. This requires some skill (not a whole lot), and a lot of discipline. This is why companies have coding standards and such.

The responsibility for elegance -- in the sense of design -- has been moved up to people with greater skills and training: architects who can talk in terms of things like design_Patterns.

The ISO manuals are similar to the 'machining manual' of the brick_Mortar industry. It is able to produce a tvs_Star or bajaj_Boxer, but in skilled hands, it can also produce a harley_Davidson. It can produce a hello_World in 1000 lines, but it can also produce a linux or even a Mac in millions of lines.

Sorry for the long_Winded response. The shorter -- and dare I say, elegant -- version would be: "Yo Dilip, you are completely off_Base here".

Kartik said...

Good enough for you to speak about it - spare a thought for those of us that stand facing an academic and working life full of these :-)

Not my chance, by choice of course - wouldn't want to convey the impression that we were kids barely out of school that had no idea whatsoever what we wanted to do and thus got swept along by the popular tide - ah uh!

Dilip D'Souza said...

Abi, in some ways this is the argument between the techies and the marketing types; when I was a techie, the marketing types seemed full of hot air and that's where some of this (but only some) comes from.

You have a most interesting point about code maintenance after the original writer has upped and left. But that's part of it: elegant code will also, almost by definition, be easier to understand.

In all my years in software, I always felt the best software was the stuff produced with a deal of inspiration. Hard work there too, but inspiration for sure. I think software companies need to focus not so much on coding standards, but on giving their most creative software writers the kind of ambience and support where they can tap that inspiration.

I see this as analogous to support for pure research in the sciences, for example.

But nevertheless, this is one of those debates without end, as I'm sure you're aware!

I may be off base, but I wanted the perspective you bring, so thanks. And hey, I'm not out there calling you "prof congenital idiot".

Kraktik, the only thing I can say is, stay away from ISO whatever.

Abi said...

Dilip,

I agree that elegant code is also easy to maintain. However, there is no universal definition of elegance, right? Fact: there are so many Hemingway wannabees, that there are quite a few write-alike contests!

So, I would settle for competent, maintainable code, simply because we are talking about an industry that employs millions of interchangeable people. The truly elegant coder -- who emerges rather by consensus -- may get some sort of an exception (leader, mentor, designer, leader, architect!), and I think that's what is done in industry.

For all it's stratospheric status, software development is still an art; the number of projects that are abandoned midway is huge. Thus, the effort to convert that into a predictably reliable engineering discipline is worthwhile; perhaps a uniform enforcement of ISO type thingies for all -- including the elegant coders -- is a misguided effort, if it indeed has a stifling effect on the elegant coders. I don't believe it does, but it is quite clear that you do.

Endless debate, indeed!

Umesh Patil said...

It always happens. Some waste their time and resources in regimenting SW development process. I guess people have not heard of 'Extreme Programming' where the entire 'water fall' model is thrown to garbage. Extreme Programming is becoming quite popular in USA. Large number of Open Source practitioners use this model successfully. Many companies do as well. Start ups in Silicon Vally obviously like this nifty way of producing SW - it has the primacy for results rather than empty motions of process. Google as expected is one of major practitioner of this art.

The reason many Indian SW professionals and groups may be attracted to 'water fall model' is because of the preponderance of 'contract / consulting' work in Indian SW industry. It is in the nature of the business of outsourcing that you try to pin down the customer for the exact and precise requirements and then to implement exactly that part. Outsourcing companies do not get rewards much for going beyond and being proactive. Much to chagrin, Indian companies as well take the easy route of not getting involved. That is the reason there is such heavy emphasis of regimenting SW development in India. High turnover of SW professionals is another fundamental incentive to be regimented here. If you listen to Nilekani and other titans of Indian SW industry; you will get the boasting of how much process they have adopted and improvised. Nilekani has gone on record to say that Infosys has turned recruiting and retaining professionals into an art form. Needless to say Infosys boasts that they are at the top of the world as far as pursuing this art form. SW process and regimentation is not far away from the techniques of insulating SW from team member turnover. By and large product based SW industry would need to be more proactive and need to think out of box. May be after few years SW professionals in India would out grow the weaknesses of such water fall model as well as the obsession with process. Current business structure of SW industry in India is not conducive for such enlightenment.