A customer from Norway heard from a friend of a project with software dating back several decades (not as uncommon as you might think). He sent me a link to, and a request to blog about, an article on Software that Lasts 200 Years, written by Dan Bricklin. Since Dan is one of the creators of VisiCalc, he ought to know a thing or two about the topic. His main thesis is that software for certain tasks in society (libraries, public records etc.) needs to be built, maintained and indeed thought about in a different way from today's software: as infrastructure akin to roads or bridges.

Dan Bricklin is always interesting, I've found myself coming across his site several times. I don't really have any great insights on what he says in the article, so I'll just add some thoughts on a few bits. I like his comment that open source fans should concentrate on getting trains to run on time rather than persuading railways to run on Linux. The basis of open source was that it's nice if people who write software have access to the source code of other software, not so much about making software and giving it away for free to people who will never look at the source code. Both are fine aims, but sometimes these days the focus is only on the secondary aim and people forget about the primary aim.

Dan is refreshingly realistic in accepting that there will be multiple ways to achieve "software as infrastructure": companies, government, open source communities. Some of his views seem a little naïve, e.g. the idea that licensing professionals is just to improve quality: I think economists generally say an important function of licensing is to restrict entry, thus increasing scarcity and helping justify high prices for existing practitioners. Similarly his thoughts on standards bodies assume they only work for the common good, not to restrict competition or force competitors to compete on existing companies' terms. (I'm deliberately pointing out only the other side of the coin here; the truth includes both sides.) In contrast to what he says (admittedly back in 2004), major enquiries have indeed been held when software has failed, but since the parties involved are mostly private companies rather than government (unlike for bridges), they aren't often public.

Those are some of my opinions, but I'm sure Dan has better grounds for his: read the article to get the full picture. (Mind you, I suspect a good economist or analyst could do a lot better than either of us: if any readers have good links, add a comment!).

I still occasionally use the first piece of commercial software I wrote (a sound editor for Casio's CZ range of synthesizers) and the first programmer's text editor I used at work (Intel's aedit) -- both from around 1986. People born in that year have now graduated from university -- a sobering thought. Not just because of how much it dates me, but because of how little has changed. That university graduate has certainly changed and matured a lot more in 22 years than our industry or the tools we use.

Ever since we first started getting customers in Japan a few years ago, I've been consistently impressed by their professionalism, attitude and achievements. Juha-Pekka recently sent me a fascinating paper* comparing the productivity of developers across the world. The study covered over 100 projects in a variety of domains, with a balanced spread across Japan, the United States, India, and Europe. The most eye-opening result was a comparison of the output per developer per month (in KLOC units of source code), and of the defect rate in the year following release of the project (in defects/KLOC):

The myriad factors affecting individual projects mean the actual figures aren't really that important or trustworthy. Even so, the massive difference between Japan and the rest of the world is worth thinking about. Interestingly, the biggest difference in development practices between the regions was how many projects created "detailed designs": 85% in Japan, but only 32% in the US. In India, the figure was 100%, as might be expected if these were outsourced projects.

Some code generation was used in around 50% of projects in all regions, with Japan actually using it least (41%). Given the survey took place in 2001-2002, presumably the majority of generation was from fixed language tools and wizards, where the developer is expected to edit the results. It would be fun to see how DSM projects would look on the graph: obviously several times greater in terms of output, but would American DSM users be able to match Japanese hand-coders' defect rates? 20 times higher quality is a pretty tall order, even for DSM!

* Michael Cusumano, Alan MacCormack, Chris F. Kemerer, and William Crandall: Software Development Worldwide: The State of the Practice, IEEE Software Nov/Dec 2003.

Via Grady Booch:

Fellow IBMer Jim Conallen pointed me to this report which indicates that the linked list data structure has been patented. A quick search at the USPTO verifies that this claim is real: the abstract for this patent may be found here . The patent was filed on September 26, 2002, the inventor being Ming-Jen Wang of LSI Logic Corporation , and subsequently US Patent 7028023 was granted last year, on April 11th, 2006.

I wonder just how many times this patent application was processed by software using linked lists in its implementation?

Looking at the abstract, I'd actually say that this isn't a patent for linked lists, at least not in the traditional sense. I can't access the images, but it seems that the idea is that each data element in the list has at least two pointers stored with it. The first pointer is to the next element as in a standard linked list, and the second pointer is to another element. Following the first pointers gives the canonical order of the list, whereas following the second pointers gives a differing ordering. If the elements are people, the canonical ordering might be sorted by last name, and the second ordering sorted by first name. Other orderings, e.g. order by age, can be added by adding an extra pointer to each item.

Personally, I'm happy to let Ming Jen Wang have his patent. Hard-coding the number of possible orderings right in the data structure seems like a bad idea. Linked lists are great for insertion and deletion, but the usage here seems to be more that a single, unchanging list needs to be sorted in many ways. In that case it would make more sense to allocate a separate array of pointers for each ordering. Adding and deleting orderings is then easy, and using each is fast. If you also want to add and delete items to the main list, simply make each ordering array itself a linked list.

So, while I would say this patent probably doesn't transgress too badly in terms of prior art, I would still have rejected it The USPTO home page says a patent must be "new, useful [and] non-obvious", and this for me fails the "useful" test. Even if someone shows an area where it's useful, then it still fails the "non-obvious" test. Teach linked lists to a class, then ask "but what if you wanted a second ordering too", and I bet more than one will come up with this.

Ironically, the USPTO home page currently trumpets their new accelerated process, citing a case where the average review time of 2 years was cut to 6 months. The trick? Rather than the USPTO searching for prior art, they let the applicant do it. Right...

Interesting snippet from Usenet: according to these posts by Geoff, Smalltalk churns out programs several times faster than Java:

IBM keeps function point counts on their projects. My understanding is, for a wide range of projects, ST comes out way ahead. ... ST has gotten as high as 60 FP's per 100 man hours, Lotus Notes 30, Java 15 to 20, and PL/1 5.

Having programmed in all those languages, I'd say those figures are pretty close to the mark. Smalltalk is perhaps benefitting slightly from the fact that the best programmers tend to prefer it. Maybe for the same programmer, Smalltalk would be "only" 2.5 -- 3x as fast as Java.

In the end, though, it's always a question of horses for courses. Each of those languages has its own sweet spots: application areas where it shines. Even using PL/M didn't feel bad for building mainframe firmware, because we didn't have to stretch the language outside its comfort zone. Also, there won't be figures for using Lotus Notes to build firmware, but there will certainly be Notes-like applications built in Java and probably PL/1.

As Geoff says, the figures above are for a wide range of projects. One thing I find interesting in Smalltalk is its ability to build a wide range of applications well. The libraries and other goodies accompanying Cincom VisualWorks, for instance, cover everything from 3D CAD to Web Services. When you find a new area that doesn't have a library, creating one for your own needs is easy and feels natural. Smalltalk really does seem to be the best generic programming language for building domain-specific languages.

How much does this contribute to the productivity figures? Nobody is going to convince me that using a period rather than a semicolon to end a line of code is going to make a 3x difference in productivity. Even the superb collection classes and IDE can only account for so much -- the less so as bit by bit others manage to copy them. Being a dynamic language helps in and of itself, but most of all because it seems to enable or promote the creation of DSLs -- at least it seems so, given that the three "DSL friendly" languages that always get mentioned are LISP, Smalltalk and Ruby.

For any given project, measuring the productivity of several languages is hard: you can't build the same system twice with different languages, but with the same programmers with the same level of understanding. Measuring similar projects is possible but can still be dubious: maybe one language is just good at that type of problem. It's only when you see a big difference over a wide range or projects that you can really start to think that one approach might really be shown to be better than another. I don't endorse Smalltalk because it's what I use, nor promote DSM because it's what we sell: I'm just a sceptic who has been convinced by the evidence.

Insightful comments from Stevey's Blog Rants (via Stefan Tilkov):

[W]hen you dig down into a programming language, and you get past all the hype and the hooplah, what you find is a set of policies and decisions that affect your everyday life as a programmer in ways you can't ignore, and that no amount of hype will smooth over.

If your language is sitting on you like an invisible elephant, and everyone using the language is struggling to work around the same problems, then it's inevitable that other languages will come into play. Libraries can make you more productive, but they have almost no effect on the scalability of the language.

Every language has a complexity ceiling, and it's determined by a whole slew of policy and design decisions within the language, not the libraries. The slew includes the type system (with its attendant hundreds of mini-policies), and the syntax, and it also includes the language's consistency: the ratio of rules to exceptions.

Java's demonstrating quite clearly that at a certain level of complexity, the libraries and frameworks start to collapse under their own weight. People are always writing "lightweight" replacements for existing fat Java libraries and frameworks, and then the replacements get replaced, ad infinitum. ...

[I]t's pretty clear that OOP alone doesn't cut it; it has to be supplemented with Language-Oriented Programming and DSLs. And all languages, DSLs and general-purpose languages alike, have to be designed to maximize consistency; each inconsistency and special-case in the language adds to its conceptual overhead and lowers the complexity ceiling.

That's so well written, and so close to my own experience and thoughts, that I can only add one thing: if you make a DSL (textual or graphical) and you make a mistake or need to extend it, you can. If the language is provided by a vendor, you can't. And if it's designed by committee...

Avi Bryant, father of Seaside, on why he uses Smalltalk:

People are often surprised when I tell them which language I choose to work in. But where else I find one as expressive as Ruby, with a VM as sophisticated as Hotspot, an IDE as good as Eclipse or better, and a community with 20 years of experience at using those three pieces as an integrated whole? Frankly, I couldn’t make any other choice with a straight face.

Aside from mere trivialities like being 2-3 times more productive to work in than Java, Smalltalk is more satisfying on a number of levels. As Eliot Miranda's sig put it:

The fact is,

No matter how closely I study it,

No matter how far I take it apart,

No matter how far I break it down;

It remains Consistent.



I wish you were here to see it!

But sure, we all prefer staying in our comfort zone. If you make me use another language than the one I like and use the most, I'll feel just the same about it as if a non-Smalltalker is asked to do Smalltalk. The difference is just that even in one hour of Smalltalk you see enough of its benefits to make up for the annoyance of having to get to grips with something different. I've used a bunch of languages for more than an hour, but so far the only ones that I feel have given me something new are Forth, Smalltalk and Rexx. And there's only one of those in which I'd want to write actual systems, as opposed to little programs. If you want to try Smalltalk, I'd recommend Dolphin on Windows or VisualWorks if you're willing to invest a little more effort.