show all comments


D-TDD: Destruction Test Driven Development

January 24, 2013 18:00:25 +0200 (EET)

If you've ever seen a child learn to stack blocks, you'll know that the greatest pleasure isn't derived from the beauty or height of the structure. No: the squeals of joy are reserved for when he knocks it down, and the order of the tower is replaced by the chaos of flying blocks.

Last Friday evening I took an equally constructive approach to work on the MetaEdit+ 5.0 multi-user version. We're at the stage where we have the single user version tested and released, and "the first build that could possibly work" of the multi-user clients, with normal user testing showed no problems. So I set up the server and a couple of clients on my PC, and scripted the clients to bash the server as fast as they could with requests. In each transaction, a client would try to lock and change 10% of the repository objects, then either abandon or commit the transaction.

As that seemed to bubble along quite happily, I started up my laptop too, and used a batch file to start a MetaEdit+ client (from the original PC's disk) and run the same script. And again for a fourth client, whereupon I remembered a PC in the next office was unused, and I knew its MAC address so could start it up with WakeOnLAN. (You really learn to appreciate Remote Desktop Connection, VNC and WakeOnLAN when it's -29°C...)

By the end of the evening, I'd squished a couple of bugs: the more cycles you run, the less chance there is of bugs being able to hide. I'd also progressed to four PCs, running a total of 12 clients. Over the course of the weekend, I occasionally poked my nose into a remote desktop to see how things were doing, and another bug was found (apparently in Windows XP's networking, since it didn't appear on the other platforms; still, easy enough to just retry if it occurred).

At some point I restarted the experiment with the bug fixes in place from the start, to get a consistent set of data. At that point the original repository had grown from 40MB to over 4GB, as each client was creating several times the original amount of data each hour. As I woke up with the 'flu on Monday, the experiment continued in a similar fashion through to my return to work on Thursday. The last session was started Wednesday afternoon, with up to 20 clients, and by the same time Thursday its server had processed 1TB of disk I/O, in under 10 hours of CPU time and only 32MB of memory:

So, what do we learn from this?

  1. Destruction testing is fun! Just as much fun as with the blocks, and even more so: when you catch a bug as it happens, and fix it in the server on the fly, the tumbling client blocks reassemble themselves into the tower.
  2. Destruction testing is necessary. There are some bugs you'll never catch with code inspection, manual testing or unit tests. They might occur only once a year in normal use - and at a customer, where all you get is an error log. By forcing them out of the woodwork with massive destruction testing, you can see them several times, spot the pattern, and confirm a fix.
  3. Destruction testing is easier than ever before. Earlier, the operating system, hardware, or time would be the limiting factor more often than not. Now, normal PCs are reliable and performant enough to keep the focus on what you want to be tested, rather than how to keep enough PCs running long enough to test it.
  4. Destruction testing is not scalability testing. It may stray into that area, but it has a different purpose. The repository used in MetaEdit+ has been tested with hundreds of simultaneous users, so that wasn't in doubt. The point here was to flush out any bugs that had crept in with the new MetaEdit+ and platform versions.
  5. Destruction testing is not bulletproof. There are plenty of bugs that it won't find, but it will find bugs you can't find another way. Since you can't test everything to destruction, concentrate on testing lower-level services called by everything, or just the most common services. Other kinds of testing are better at covering the breadth of functionality.


200 year software

June 23, 2008 22:40:00 +0300 (EEST)

A customer from Norway heard from a friend of a project with software dating back several decades (not as uncommon as you might think). He sent me a link to, and a request to blog about, an article on Software that Lasts 200 Years, written by Dan Bricklin. Since Dan is one of the creators of VisiCalc, he ought to know a thing or two about the topic. His main thesis is that software for certain tasks in society (libraries, public records etc.) needs to be built, maintained and indeed thought about in a different way from today's software: as infrastructure akin to roads or bridges.

Dan Bricklin is always interesting, I've found myself coming across his site several times. I don't really have any great insights on what he says in the article, so I'll just add some thoughts on a few bits. I like his comment that open source fans should concentrate on getting trains to run on time rather than persuading railways to run on Linux. The basis of open source was that it's nice if people who write software have access to the source code of other software, not so much about making software and giving it away for free to people who will never look at the source code. Both are fine aims, but sometimes these days the focus is only on the secondary aim and people forget about the primary aim.

Dan is refreshingly realistic in accepting that there will be multiple ways to achieve "software as infrastructure": companies, government, open source communities. Some of his views seem a little naïve, e.g. the idea that licensing professionals is just to improve quality: I think economists generally say an important function of licensing is to restrict entry, thus increasing scarcity and helping justify high prices for existing practitioners. Similarly his thoughts on standards bodies assume they only work for the common good, not to restrict competition or force competitors to compete on existing companies' terms. (I'm deliberately pointing out only the other side of the coin here; the truth includes both sides.) In contrast to what he says (admittedly back in 2004), major enquiries have indeed been held when software has failed, but since the parties involved are mostly private companies rather than government (unlike for bridges), they aren't often public.

Those are some of my opinions, but I'm sure Dan has better grounds for his: read the article to get the full picture. (Mind you, I suspect a good economist or analyst could do a lot better than either of us: if any readers have good links, add a comment!).

I still occasionally use the first piece of commercial software I wrote (a sound editor for Casio's CZ range of synthesizers) and the first programmer's text editor I used at work (Intel's aedit) -- both from around 1986. People born in that year have now graduated from university -- a sobering thought. Not just because of how much it dates me, but because of how little has changed. That university graduate has certainly changed and matured a lot more in 22 years than our industry or the tools we use.


Why are the Japanese so much better?

January 10, 2008 01:25:49 +0200 (EET)

Ever since we first started getting customers in Japan a few years ago, I've been consistently impressed by their professionalism, attitude and achievements. Juha-Pekka recently sent me a fascinating paper* comparing the productivity of developers across the world. The study covered over 100 projects in a variety of domains, with a balanced spread across Japan, the United States, India, and Europe. The most eye-opening result was a comparison of the output per developer per month (in KLOC units of source code), and of the defect rate in the year following release of the project (in defects/KLOC):

Japan: 0.469 : 0.02, Europe 0.436 : 0.225, US 0.270 : 0.400, India 0.209 : 0.263

The myriad factors affecting individual projects mean the actual figures aren't really that important or trustworthy. Even so, the massive difference between Japan and the rest of the world is worth thinking about. Interestingly, the biggest difference in development practices between the regions was how many projects created "detailed designs": 85% in Japan, but only 32% in the US. In India, the figure was 100%, as might be expected if these were outsourced projects.

Some code generation was used in around 50% of projects in all regions, with Japan actually using it least (41%). Given the survey took place in 2001-2002, presumably the majority of generation was from fixed language tools and wizards, where the developer is expected to edit the results. It would be fun to see how DSM projects would look on the graph: obviously several times greater in terms of output, but would American DSM users be able to match Japanese hand-coders' defect rates? 20 times higher quality is a pretty tall order, even for DSM!

* Michael Cusumano, Alan MacCormack, Chris F. Kemerer, and William Crandall: Software Development Worldwide: The State of the Practice, IEEE Software Nov/Dec 2003.


Linked list patented in 2002?

March 20, 2007 12:27:53 +0200 (EET)

Via Grady Booch:

Fellow IBMer Jim Conallen pointed me to this report which indicates that the linked list data structure has been patented. A quick search at the USPTO verifies that this claim is real: the abstract for this patent may be found here . The patent was filed on September 26, 2002, the inventor being Ming-Jen Wang of LSI Logic Corporation , and subsequently US Patent 7028023 was granted last year, on April 11th, 2006.

I wonder just how many times this patent application was processed by software using linked lists in its implementation?

Looking at the abstract, I'd actually say that this isn't a patent for linked lists, at least not in the traditional sense. I can't access the images, but it seems that the idea is that each data element in the list has at least two pointers stored with it. The first pointer is to the next element as in a standard linked list, and the second pointer is to another element. Following the first pointers gives the canonical order of the list, whereas following the second pointers gives a differing ordering. If the elements are people, the canonical ordering might be sorted by last name, and the second ordering sorted by first name. Other orderings, e.g. order by age, can be added by adding an extra pointer to each item.

Personally, I'm happy to let Ming Jen Wang have his patent. Hard-coding the number of possible orderings right in the data structure seems like a bad idea. Linked lists are great for insertion and deletion, but the usage here seems to be more that a single, unchanging list needs to be sorted in many ways. In that case it would make more sense to allocate a separate array of pointers for each ordering. Adding and deleting orderings is then easy, and using each is fast. If you also want to add and delete items to the main list, simply make each ordering array itself a linked list.

So, while I would say this patent probably doesn't transgress too badly in terms of prior art, I would still have rejected it The USPTO home page says a patent must be "new, useful [and] non-obvious", and this for me fails the "useful" test. Even if someone shows an area where it's useful, then it still fails the "non-obvious" test. Teach linked lists to a class, then ask "but what if you wanted a second ordering too", and I bet more than one will come up with this.

Ironically, the USPTO home page currently trumpets their new accelerated process, citing a case where the average review time of 2 years was cut to 6 months. The trick? Rather than the USPTO searching for prior art, they let the applicant do it. Right...


Hard data is hard, but the evidence mounts up

February 06, 2007 15:57:42 +0200 (EET)

Interesting snippet from Usenet: according to these posts by Geoff, Smalltalk churns out programs several times faster than Java:

IBM keeps function point counts on their projects. My understanding is, for a wide range of projects, ST comes out way ahead. ... ST has gotten as high as 60 FP's per 100 man hours, Lotus Notes 30, Java 15 to 20, and PL/1 5.

Having programmed in all those languages, I'd say those figures are pretty close to the mark. Smalltalk is perhaps benefitting slightly from the fact that the best programmers tend to prefer it. Maybe for the same programmer, Smalltalk would be "only" 2.5 -- 3x as fast as Java.

In the end, though, it's always a question of horses for courses. Each of those languages has its own sweet spots: application areas where it shines. Even using PL/M didn't feel bad for building mainframe firmware, because we didn't have to stretch the language outside its comfort zone. Also, there won't be figures for using Lotus Notes to build firmware, but there will certainly be Notes-like applications built in Java and probably PL/1.

As Geoff says, the figures above are for a wide range of projects. One thing I find interesting in Smalltalk is its ability to build a wide range of applications well. The libraries and other goodies accompanying Cincom VisualWorks, for instance, cover everything from 3D CAD to Web Services. When you find a new area that doesn't have a library, creating one for your own needs is easy and feels natural. Smalltalk really does seem to be the best generic programming language for building domain-specific languages.

How much does this contribute to the productivity figures? Nobody is going to convince me that using a period rather than a semicolon to end a line of code is going to make a 3x difference in productivity. Even the superb collection classes and IDE can only account for so much -- the less so as bit by bit others manage to copy them. Being a dynamic language helps in and of itself, but most of all because it seems to enable or promote the creation of DSLs -- at least it seems so, given that the three "DSL friendly" languages that always get mentioned are LISP, Smalltalk and Ruby.

For any given project, measuring the productivity of several languages is hard: you can't build the same system twice with different languages, but with the same programmers with the same level of understanding. Measuring similar projects is possible but can still be dubious: maybe one language is just good at that type of problem. It's only when you see a big difference over a wide range or projects that you can really start to think that one approach might really be shown to be better than another. I don't endorse Smalltalk because it's what I use, nor promote DSM because it's what we sell: I'm just a sceptic who has been convinced by the evidence.


If your language is sitting on you like an invisible elephant

September 27, 2006 14:53:03 +0300 (EEST)

Insightful comments from Stevey's Blog Rants (via Stefan Tilkov):

[W]hen you dig down into a programming language, and you get past all the hype and the hooplah, what you find is a set of policies and decisions that affect your everyday life as a programmer in ways you can't ignore, and that no amount of hype will smooth over.
If your language is sitting on you like an invisible elephant, and everyone using the language is struggling to work around the same problems, then it's inevitable that other languages will come into play. Libraries can make you more productive, but they have almost no effect on the scalability of the language.
Every language has a complexity ceiling, and it's determined by a whole slew of policy and design decisions within the language, not the libraries. The slew includes the type system (with its attendant hundreds of mini-policies), and the syntax, and it also includes the language's consistency: the ratio of rules to exceptions.
Java's demonstrating quite clearly that at a certain level of complexity, the libraries and frameworks start to collapse under their own weight. People are always writing "lightweight" replacements for existing fat Java libraries and frameworks, and then the replacements get replaced, ad infinitum. ...
[I]t's pretty clear that OOP alone doesn't cut it; it has to be supplemented with Language-Oriented Programming and DSLs. And all languages, DSLs and general-purpose languages alike, have to be designed to maximize consistency; each inconsistency and special-case in the language adds to its conceptual overhead and lowers the complexity ceiling.

That's so well written, and so close to my own experience and thoughts, that I can only add one thing: if you make a DSL (textual or graphical) and you make a mistake or need to extend it, you can. If the language is provided by a vendor, you can't. And if it's designed by committee...


Smalltalk remains...

December 02, 2005 01:32:00 +0200 (EET)

Avi Bryant, father of Seaside, on why he uses Smalltalk:

People are often surprised when I tell them which language I choose to work in. But where else I find one as expressive as Ruby, with a VM as sophisticated as Hotspot, an IDE as good as Eclipse or better, and a community with 20 years of experience at using those three pieces as an integrated whole? Frankly, I couldn’t make any other choice with a straight face.

Aside from mere trivialities like being 2-3 times more productive to work in than Java, Smalltalk is more satisfying on a number of levels. As Eliot Miranda's sig put it:

The fact is,

No matter how closely I study it,

No matter how far I take it apart,

No matter how far I break it down;

It remains Consistent.

I wish you were here to see it!

But sure, we all prefer staying in our comfort zone. If you make me use another language than the one I like and use the most, I'll feel just the same about it as if a non-Smalltalker is asked to do Smalltalk. The difference is just that even in one hour of Smalltalk you see enough of its benefits to make up for the annoyance of having to get to grips with something different. I've used a bunch of languages for more than an hour, but so far the only ones that I feel have given me something new are Forth, Smalltalk and Rexx. And there's only one of those in which I'd want to write actual systems, as opposed to little programs. If you want to try Smalltalk, I'd recommend Dolphin on Windows or VisualWorks if you're willing to invest a little more effort.