Later, I realised that most programming errors fit into two broad categories: the obvious and the subtle. Obvious errors are those whose source can be easily pinpointed (even if fixing the problem takes a while). The subtle are typically those where cause and effect are separated, making identification of the root of the problem difficult (often, when eventually located, such problems are easily fixed). The looop problem above was, to a relative novice programmer, a very subtle problem (years of experience have taught me that looking for daft spellings and in my own programs is a good initial target when debugging).

There are, to my mind, two tricks to debugging. The first is to try and turn subtle problems into obvious problems; however subtle problems are typically inherently subtle and unamenable to such treatment. The second is to try and speed up the solving of obvious problems. For me, the main tool for solving obvious problems is the humble backtrace which, when an exception occurs, shows one (in some manner or another) the call stack and, hopefully, what file and line number each entry therein is associated with. Given the following trivial program:

func head(l):
    return [l[0], l[1 : ]]

func main():
    head([1])
    head([])

Traceback (most recent call at bottom):
  1: File "/tmp/head.cv", line 6
  2: File "/tmp/head.cv", line 2
  3: (internal), in List.get
Bounds_Exception: 0 exceeds upper bound 0

Backtraces like the above can be found in most modern programming languages like Java. They are immensely useful and form precisely half of my debugging toolkit, the other half being printf - in my view of the world, these two tools obviate the need for debuggers. The power of backtraces is most obviously felt in those languages that don't have them. C programs typically need to be run through a debugger to get a backtrace, meaning that errors in programs running in production can be extremely difficult to diagnose. The first Haskell program I wrote had the head error in it. The resulting message just said Prelude.head: empty list with nothing else to help me - no line numbers or even file names. Needless to say, it took me a long while to work out what this meant, and how it happened (if I remember correctly, I passed an empty list to a library function which, in turn, called head). Unsurprisingly that was also pretty much the last program I wrote in Haskell - languages that turn should-be-obvious errors into subtle errors are of no use to me. [Apparently Haskell now has some in-built, though hardly easy to use, support for backtraces. The implications relating to Haskell's prioritisation of features are, to my mind, highly amusing.]

Python was the first language I saw that took backtraces a little bit further, printing (when possible) the line of source code associated with each part of the backtrace. A Python-esque backtrace looks roughly as follows:

Traceback (most recent call at bottom):
  1: File "/tmp/head.cv", line 6
       head([])
  2: File "/tmp/head.cv", line 2
       return [l[0], l[1 : ]]
  3: (internal), in List.get
Bounds_Exception: 0 exceeds upper bound 0

I realised early on in Converge's development that knowing merely the line number of an error was only part of the problem. Often a specific sub-expression within a certain line is the relevant part of the backtrace, and the rest of the line is noise. Converge therefore recorded the column (i.e. offset within a line) where each error is associated with, meaning that backtraces looked like the following:

Traceback (most recent call at bottom):
  1: File "/tmp/head.cv", line 6, column 4
  2: File "/tmp/head.cv", line 2, column 13
  3: (internal), in List.get
Bounds_Exception: 0 exceeds upper bound 0

Converge's backtraces stayed like the above for quite some time, until recently when I realised that knowing the start column associated with an error is only part of the story. What one really wants to know is the start and end of the associated expression. A small tweak to the parser, and a huge (but mechanical) change to the compiler, and Converge backtraces could tell one how many characters in the line an error was associated with:

Traceback (most recent call at bottom):
  1: File "/tmp/head.cv", line 6, column 4, length 8
       head([])
  2: File "/tmp/head.cv", line 2, column 12, length 4
       return [l[0], l[1 : ]]
  3: (internal), in List.get
Bounds_Exception: 0 exceeds upper bound 0

A short while later, the answer hit me: what the backtraces need to do is to highlight the relevant sub-expression within the line. Here's a screenshot of the above error running in an xterm with the latest version of Converge:

As you might notice, the tiny little difference here is that the part of each line pertinent to the error is in bold and underlined. Knowing that, one can instantly see that the first of the two list lookups on line 2 is responsible for calling List.get incorrectly. Interestingly, my first attempt at this put the offending fragments only in bold, but since whitespace can sometimes be a significant part of an error, underlining can be a useful aid. In the case where the associated source code is split over multiple lines, the first relevant line of source code is printed with ... added to the end of the line to inform the user that the printed line is not the end of the story.

As I explained in a previous entry, when Converge DSLs are translated into Converge ASTs, individual call stack entries can be associated with more than one source location. This means that backtraces tend to be rather long, which previously made tracking down the cause of an error tedious - loading multiple files into text editors and continually flipping back and forth to xterms is not fun. Extended backtraces become a real life saver in this regard. Here's an example where a DSL incorrectly tries to subtract a string from an integer:

Looking at this backtrace, an experienced programmer will be able to quickly surmise that, given the exception message, the most likely candidate for this error is in the ex4.cv file (which, as the eagle-eyed may notice, is code written in the DSL - Converge's errors work with both the base language and with embedded DSLs). Imagine trying to debug this with a traditional backtrace: there's a lot of information in the backtrace, and there would be no indication as to which part of it is more likely to be responsible for the error.

From a practical point of view, Converge's extended backtraces have no run-time penalty for correct code, and users don't have to do anything to enable them - they're a standard part of the system. Extended backtraces can be found in -current versions of Converge (at the time of writing, Converge's support for Curses under Windows is weak, so underlining doesn't work there - it's a quick, fun little project for someone who's interested).

So, going back to the start of this entry, how do Converge's extended backtraces help with debugging? Well, they might help turn the odd subtle error into an obvious error, but that's an incidental benefit. What I think they do is make solving obvious errors much quicker than previously. In the sort time since I've had extended backtraces, I've noticed that I've often been able to almost instantly fix errors that before might have taken me a couple of minutes. Given the number of programming errors I make, the cumulative time saving is most welcome.

In summary, I think that Converge's extended backtraces are a real boon to programming. To the best of my knowledge, Converge is the first language with such backtraces in - I hope it won't be the last! ]]> http://tratt.net/laurie/tech_articles/articles/designing_sane_scoping_rules scoping rules. In this article I'm going to outline why Converge has the scoping rules it does.

The first thing I need to do is to outline the problem. Although all my examples are framed in terms of a typical imperative programming language, the underlying concepts all translate fairly directly to functional languages too. Here's the simplest example possible:

x := 2
...
y := x

The first major issue with regards to variables is global vs. local variables. Take the following code which is intended to represent the top-level of a source file:

x := 2

func f():
    x := 3

f()

Retro-fitting sane scoping rules is not easy if the first version of your language used the above scoping rule (note the deliberate use of the singular) - any change of scoping rule(s) has a high chance of breaking programs in nasty ways. Some of the BASIC-type languages I first used solved the backwards compatibility problem by making variables global by default but allowing variables in functions to be declared as local. This allows one to rewrite the above example as follows:

x := 2

func f():
    local x
    x := 3

f()

Most programming languages adopt a slight variant of the above rules which are fairly easy to understand in practice. Essentially variables with the same name as a top-level variable reference that top-level variable directly, while all other variables are local. So in a C-type language the following code contains two variables: a top-level x and a y local to f:

x := 2

func f():
    x := 3
    y := 4

f()

The next major design challenge for scoping rules is much more subtle and confuses many of us to this day. Knowing that the global keyword in Python declares that assignment to a specified variable doesn't make it local to that scope, consider the following Python code:

x = 2

def f():
    x = 3
    
    def g():
        global x
        print x
        x = 4
    
    print x
    g()

f()
print x

$ python scope.py
3
2
4
$

It might at first seem that Python's scoping rules are simply silly; actually, they're not unreasonable and they're shared by most programming languages (e.g. Java). Why? The problem is that the function g might outlive f. Here's a simple example:

func f():
    x := 2
    func g():
        return x
    return g

f()()

Scheme was the first language that presented a practical solution to this problem of nested scopes in the form of closures. The standard way that closures are defined is guaranteed to confuse and I'm not going to repeat it. They're actually very simple: essentially each function allocates heap memory to store variables on. Thus if an inner function outlives an outer function there is no problem in referencing variables in the outer function even if the stack space has long since disappeared, since a function calls variables can outlive the function call itself.

[As an aside, the fact that closures need to allocate heap memory (although it's often possible to statically analyse such allocations away) has been used as an argument against them in languages such as Java. That's the chief reason that Java has all sorts of complications like inner classes, final variables and so on: Java resisted closures, and then had to resort to hacks to get a poor facsimile of its functionality. It's hard to imagine any decent programming language being built now that doesn't implement closures (Converge certainly does), so closures are gradually losing their exotic tag (which, I suspect, is based on their definition and not their utility, which is far from exotic).]

When I was designing Converge, I put some effort into deciding what its scoping rules were going to be. I wanted to make things safe by default (e.g. no global by default-type nonsense), and to make closures easy to deal with. Converge's scoping rules are (or, at least, were), I believe, the simplest of any imperative programming language. They boil down to this: assigning to a variable makes it local to a function; unless a variable is declared nonlocal, in which case scopes are searched from inner to outer to find the matching variable. That's it. Rewriting the Python example into Converge yields the following:

x := 2

func f():
    x := 3
    
    func g():
        nonlocal x
        Sys::println(x)
        x := 4
    
    Sys::println(x)
    g()

func main():
    f()
    Sys::println(x)

$ converge s.cv
3
3
2
$

I said above that Converge has - or had - the simplest of any imperative programming language (at least as far as I'm aware). Some time after the first publications and release of Converge, the Python team decided to fix their scoping rules for their backwards-compatibility-breaking Python 3000 release. PEP 3104 contains the eventual proposal they came up with. Interestingly it is identical to Converge's scoping rules even down to using the nonlocal keyword. Please note that I'm not suggesting that the Python team copied Converge uncredited (and even if they did, I wouldn't mind - I actively encourage people to take the good ideas from Converge and put them in other languages). However I think it shows that these are good scoping rules and that eventually imperative programming languages will evolve towards something like them.

The open question is this: why has it taken us, as a community, 50 years or more to define two simple scoping rules (assignment is local; nonlocal successively searches outer scopes) and one simple implementation technique (closures), and why have we taken so many wrong turns (in this article I haven't enumerated the silliest, such as dynamic scoping)? I suspect the answer, if it could be definitively uncovered, would give one a very interesting insight to the subject of computing in general. ]]> http://tratt.net/laurie/tech_articles/articles/some_lessons_learned_from_icon One of Converge's lesser known influences is Icon. Although it's a relatively obscure language itself, a surprising number of people have heard of Icon's direct predecessor SNOBOL. I first heard of Icon through Tim Peters and Python, where Icon was the inspiration for Python's generator feature (basically procedures that whose execution can be resumed to produce multiple values). However Icon has a much richer, and definitely unique, approach to expression evaluation than any other imperative programming language. When I started working on Converge, I decided to go back to the source, and see how much of Icon's unique approach I could incorporate into Converge.

This article is a personal look back at Icon's influence on Converge, and the successes and failures resulting from that influence. It's quite probable that much of it reflects my slightly superficial understanding of Icon - apart from its innovative approach to expression evaluation, much of the language has an old fashioned feel to it, sometimes appearing like a dynamically typed version of Pascal, and this prevented me from ever wanting to write anything large in it. This isn't a criticism of Icon as such - it was ahead of its time in many ways - but is merely an observation that modern programmers expect something slightly different from their programming languages. Hopefully, even with the above caveat, this article contains something of value to those interested in programming languages.

Why is Icon interesting

Icon's syntax is derived from Algol, so the following complete program prints 1 to 9 inclusive:

procedure upto(x)
  i := 0
  while i < x do {
    suspend i
    i := i + 1
  }
end

procedure main()
  every x := upto(10) do write(x)
end

Failure and if: a beautiful combination

d := Dict{"a" : 2, "b" : 8}
if d.has_key("a"):
  Sys::println(d["a"])

d := {"a" : 2, "b" : 8}
try:
  v = d.has_key("a")
  print v
except KeyError:
  pass

In Converge one can express this common idiom thus:

x := Dict{"a" : 2, "b" : 8}
if v := x.find("a"):
  Sys::println(v)

Variables should not spring into existence with a default value

procedure main()
  if 1 < 0 then x := 0
  write(x)
end

x := Dict{"a" : 2, "b" : 8}
if v := x.find("c"):
  ...
else:
  Sys::println(v)

Procedures shouldn't fail by default

procedure upto(x)
  i := 0
  while i < x do {
    suspend i
    i := i + 1
  }
end

What became quickly apparent is that while this is reasonable behaviour for generators, it can be a disaster for normal procedures. Look at this code and see if you can work out what will happen when it's executed:

procedure f(x)
  if x > 0 then return 1
end

procedure main()
  write(f(-1))
end

Although this might seem trivial, the problem becomes significantly worse when it's embedded in a large body of code. f is an example of what I informally think of as the dangling if return problem - in other words, it's quite easy to write a procedure where different values should be returned, but where one branch of an if incorrectly neglects to actually return anything. I find this occurs quite often during the early stages of development when one is fleshing out functions. On several occasions I spent several hours tracking down instances of this problem.

The question I asked myself was whether this feature was worth the pain. A quick analysis of real world code quickly showed that the vast majority of functions aren't generators, and indeed only ever return a single value. Therefore optimising for the exceptional case (generators) is an odd design choice.

I considered two solutions to the problem. First, one could syntactically differentiate generators and procedures, with the former failing by default and the latter not. Second, one can make all procedures not fail by default. Adding new syntax is a decision that should never be taken lightly, and since generators aren't that common, I couldn't justify the added conceptual complexity. Therefore Converge functions became similar to Python functions, with their default return action being to return the null object.

Generators are useful, but easily hidden

l := []
...
for l.append(g())

Backtracking is rarely useful

Combining conjunction with for allows compact expressions such as the following to be expressed:

x := "abcabaacvcabcbab"
for Sys::println(i := 0.iter_to(x.len()) & i % 2 == 0 & x[i] == "a" & i)

There are two problems with the above. Most obviously, it is incredibly difficult to read from a human perspective; indeed, it would be far better written as a couple of if statements within a for body. The second problem is that the form of backtracking used is often too weak to be useful. Icon allows variables in conjunctions to have assignments undone during backtracking, but even this isn't really enough (I didn't even bother implementing such functionality in Converge, because I couldn't really see its use). Full backtracking often involves undoing assignments within objects and so on, and no imperative language can ever hope to do this automatically.

Out of all the systems I've written in Converge, only one has made any practical use of the built-in backtracking, and even then it needed some manual help to be truly useful.

The fail variable causes bizarre behaviour

Converge simplified this, so that fail was a variable pointing to a semi-special object (in similar fashion to the null object). However the fail variable proved, in admittedly rare cases, to be conceptually troubling. It's troubling in Icon too, although in slightly different ways. What do you think happens in Icon if I define l as the following and then iterate over each of its elements and print them out?

l := [1, &fail, 3]

l := []
put(l, 1)
put(l, &fail)
put(l, 3)

l := []
put(l, 1)
t := &fail
put(l, t)
put(l, 3)

Converge's attempts to simplify thus were somewhat successful, but left one hole. While one could successfully evaluate the following list in Converge:

l := [1, fail, 2]

Sys::println(l[1])

fail was edged out of Converge in stages. A while ago, the recommendation became that the only safe idiom involving fail was return fail. Perhaps surprisingly, this was not an onerous restriction. Nevertheless it still failed to prevent the problem of fail being a definition in a module. Therefore the fail variable was finally banished entirely from Converge recently (although it lives on internally in the VM). fail is now an expression in Converge's grammar, and is equivalent to return &fail in Icon. Thus finally, all of the bizarre behaviour associated with the fail variable is forgotten, and I now sleep easier at night.

Do something different

When I was starting Converge, I agreed whole heartedly with this aim, but had no idea how to achieve it. Indeed, when Converge started, I only had the vague intention of chucking a few seemingly useful influences into the melting pot (initially Python, Icon, ObjVLisp, and ultimately Template Haskell when I'd got the mix of the former three languages about right). I had that vague feeling that either everything interesting had been done before, or that the remaining interesting things were outside of my grasp. To be honest, I sort of gave up on the aim of being innovative, and just concentrated on trying to mix Converge's influences together in a coherent way. What has been interesting is that as Converge has developed, new challenges have presented themselves, and I've had to find answers (some better than others) to such challenges. And in so doing, I think Converge has grown some genuinely innovative features, such as DSL blocks, and its approach to error information for macros amongst others. And in Converge's case - unlike Icon, as I understand it - the journey isn't over. Indeed, since I think Converge is currently nearing the end of the beginning, there's bound to be new challenges ahead, some of which will lead to new innovations of one sort or another, some ultimately successful, some ultimately not.

When I read about Icon's aim, and looked at the end result, I assumed that the innovative ideas in Icon had sprung out of nowhere. I, on the other hand, had no idea how to bring such ideas, fully formed, into the world. What's become clear to me is a restating of Edison's famous rule. Innovation is often a matter of perspiration over inspiration. Now my advice to those who wish to do innovative work is simple: if you put the miles in, you'll raise your head one day and discover that you're in a place where no-one else has ever found themselves in before. It's a good feeling, and one that benefits us all, as collectively we gradually discover what works and what doesn't.

Updated (December 10 2007): Used a clearer example in the Failure and if: a beautiful combination section. ]]> http://tratt.net/laurie/tech_articles/articles/ieee_software_special_issue Roel Wuyts and I were lucky enough to guest edit the current issue of IEEE Software magazine on dynamically typed languages. We had some very good submissions, and narrowing down the selection was quite a challenge!

The IEEE have helpfully put a few of the articles from the special issue online for free. Get them while you can! The full issue is available in paper form (obviously), or online, although you need IEEE membership to access the other articles. ]]> http://tratt.net/laurie/tech_articles/articles/how_difficult_is_it_to_write_a_compiler Recently I was discussing Converge with someone, and mentioned how little time the core compiler had taken to implement (no compile-time meta-programming, limited error checking, but a functioning compiler nonetheless) - only a few days. The chap I was talking to looked at me and told me that he didn't believe me. I laughed. Actually, he really didn't believe me. Initially that surprised me, and I wondered why he might think that I was pulling his leg. But then I thought back, and only a few years ago I would probably have had the same reaction. This article is my attempt to explain his reaction, and why it's no longer the case.

When I was younger, there were three things in computing that seemed so complex that I had no idea how anyone had ever created them; certainly, I didn't think I would ever be capable of working on such things. The imposing triumvirate were hardware, operating systems, and programming languages. Although electricity has always baffled me, the hardware people have done an amazing job on interoperability of components in recent years, such that I've never felt the need to gain a low-level understanding of how electrons whizz around my systems. Operating systems were partially demystified for me as I moved into the world of open source UNIX-like operating systems (for the record, I've been an OpenBSD user for 8 years or so), where one could poke and prod virtually any aspect - although kernels retain an aura of mystery for most of us. Despite the fact that I was familiar with several languages, and had even implemented a primitive assembler language (which, to my surprise, found its way into a real system - my first large-ish programming success), real programming languages remained a mystery to me for much longer. The obvious question is: why?

Programming languages are, for most of us, an integral part of the computing infrastructure. Only a small handful of languages have ever had a wide impact, and by the time a programming language becomes popular it already probably has a decade or more of history behind it. This means that when the average programmer first comes across a new language, it has had substantial development behind it, libraries and documentation, developed its own culture, and undoubtedly had several flaws uncovered; not to mention lots of little platform portability hacks, and often complex optimisations. Faced with this wodge of sheer stuff, most peoples reaction is either to shrug their shoulders and accept it at face value (the majority), try to understand it but not know where to start (a minority), or spot how similar it is to every other language (a tiny handful of people). Because the first two categories massively outnumber the third category, a general feeling - which we're not necessarily consciously aware of - has developed that programming languages are special, and therefore that only special people can create them (a veiled insult, if ever I saw one). Thus if I suggest to people that, despite having created a fairly fully featured programming language, I'm no more capable a programmer than the next man, they dismiss it as false modesty.

So let's start looking at things in more detail. If you break a programming language down, it only has a handful of things to it. What's more, the variance between different languages is - despite the unfortunate zealotry which often clouds debate - exceedingly small. In fact, at a high level there are really only two things which any language must have: a definition and an implementation. Sometimes a paper definition is produced up front, and only when this is complete is an implementation created. Sometimes the latter defines the former; but conceptually these are two separate things. Basically the definition says things like statement S does X, and the implementation actually takes in a source file and compiles S such that it really does do X. Defining a language is very simple: any fool can do it (the skill comes in defining a good language, but that's a different story). We can then break the implementation down into three components: a compiler, libraries, and a run-time system. Again, these are conceptually separate things, although sometimes they are combined (e.g. most Python programmers do not distinguish the compiler from the run-time system). Libraries are not difficult to create, although they are time consuming. Sometimes the run-time system will be a VM, sometimes it will be bundled up with the executable created by a compiler; regardless, we'll assume for the purposes of this article that a reasonable VM is already available.

Most people don't have much trouble understanding the high level view, but it's when they start trying to understand the compiler - the thing that ultimately they interact with - that things start getting more complex. The problem is that looking at implementations like Python (quite complex), Java (very complex), or GCC (something beyond merely very complex), reveals too much detail to easily uncover the big picture. The Converge compiler, at the moment at least, is rather simpler (although, like any other program, it is growing slowly but surely as it accretes features) and is a nice snapshot of a compiler. It has three basic stages:

1. Read in a source file, and create a parse tree.
2. Turn the parse tree into an abstract syntax tree.
3. Turn the abstract syntax tree into object code.

Parsing

If we have an input file such as the following:

import Sys

func main():
  Sys::println(2 + 3)

What is its parse tree? Well, in order to uncover that, we use a parsing system, which given a description of a language structure - a grammar - can automatically create a parse tree. Parsing has a bad reputation, often deservedly so: commonly used parsing technologies are often horrible to use, although it is perfectly possible to make much more pleasant parsing technologies. Regardless of that, this is the only thing in a compiler where you will need to check an external source to understand a bit more about grammars (Wikipedia's page on formal grammars is a decent start). An indicative chunk of the Converge grammar is as follows:

top_level ::= definition ( "NEWLINE" definition )*
          ::=

definition  ::= class_def
            ::= func_def
            ::= import
            ::= var_def ( "," var_def )* ":=" expr
            ::= splice
            ::= insert

import      ::= "IMPORT" import_name import_as ( "," import_name import_as )*
import_name ::= "ID" ( "::" "ID" )*
import_as   ::= "AS" "ID"
            ::=

top_level
  -> definition
    -> import
      -> <IMPORT import>
      -> import_name
        -> <ID Sys>
      -> import_as
  -> <NEWLINE>
  -> definition
    -> func_def
      -> func_type
        -> <FUNC func>
      -> func_name
        -> <ID main>
      -> <(>
      -> func_params
      -> <)>
      -> <:>
      -> <INDENT>
      -> func_decls
      -> expr_body
        -> expr
          -> application
            -> expr
              -> module_lookup
                -> expr
                  -> var_lookup
                    -> <ID Sys>
                -> <::>
                -> <ID println>
            -> <(>
            -> expr
              -> binary
                -> expr
                  -> number
                    -> <INT 2>
                -> binary_op
                  -> <+>
                -> expr
                  -> number
                    -> <INT 3>
            -> <)>
      -> <DEDENT>

If you've understood this part, congratulations - you've got over the main stumbling block to creating a compiler.

From parse tree to abstract syntax tree

-> expr
  -> binary
    -> expr
      -> number
        -> <INT 2>
    -> binary_op
      -> <+>
    -> expr
      -> number
        -> <INT 3>

From abstract syntax tree to object code

INT 2
INT 3
ADD

Conclusion

In conclusion, the main difficulty for most of us in creating a compiler is overcoming the cultural absurdity which tells us that mere mortals can't create compilers. They can. I did. You can too. ]]> http://tratt.net/laurie/tech_articles/articles/when_are_macros_useful Many of us are used to being told that macros are useful. However few of us really understand why. A large-ish, if gradually dwindling, number of people use C, and its preprocessor, on a regular basis. This gives them access to a crude, dangerous, but surprisingly effective, macro system. However the type of macro system we are told is most useful is that of a LISP-like language. Commentaries, such as this article extolling the virtues of Scheme-like macros, continually reinforce the notion that macros are a programming nirvana. Claims of an order of magnitude improvement in developer time when macros are used are not uncommon. And because most of do not, and probably have never, used a language with a real macro system, we tend to be somewhat in awe of the programming intelligentsia who broadcast such messages.

For quite some time I have held a somewhat different opinion. For modern - and I use that word very deliberately - programming languages, I believe that LISP-like macros in their raw form aren't hugely useful. Some justification for this position is thus in order.

I designed the Converge language which is one of the few modern programming languages with a macro system. Because its macro system is fairly directly inherited from Template Haskell, it's rather more verbosely referred to as a compile-time meta-programming facility; but really, it's just a macro system. From a practical point of view, there is only one substantive difference between Scheme-like macros and Converge macros. Scheme explicitly identifies macros, and then function calls which reference that macro magically turn into macro calls. In Converge, macros are just normal every-day functions, but the call site of the macro is explicitly identified. From an expressivity perspective, the two approaches can be considered equivalent.

Adding a macro system into Converge was no small task. I had to understand a lot of things that my lazy side would rather have glossed over and I had to make innumerable mistakes before I got to a reasonable design. Fairly early on in this process I realised that there were only ever likely to be a few normal Converge programs that were likely to benefit from raw LISP-like macros. To see why, we need to take another step back.

Compared to Converge, LISP in its purest form is almost unimaginably spartan. In fact, most of the successful programming languages that date from around the early 70's or earlier, tend to lack features that most programmers now take for granted (although at least LISP and its descendants feature automatic memory management). As a general rule I am all for simplicity in life, being something of a simpleton myself. However simplicity is not an end in itself. Stone benches have an integrity, and air of stability about them, that no sofa can match; but I've not been to many houses with a stone chair in the front room. Thus an inevitable side effect of spartan languages is that people need to encode extra functionality in order to make life somewhat more bearable. Macros are an incredibly powerful way to encode such functionality in LISP-like languages. For example, you want an object orientated style system on top of LISP? Use macros. Thus macros are an integral part of the modern LISP experience: they allow users to raise the level of abstraction of the programming language.

The reason why raw macros are not especially useful for most Converge programs is that the base language itself is fairly feature rich. This is one reason why I used the word modern earlier. For example, no one in their right mind is likely to use macros to create an OO layer in Converge; it already has a perfectly serviceable one. In fact, for the majority of uses of macros in LISP, the chances are that it's not worth the effort to create an analogue in Converge. The LISP community has traditionally thought that raw macros raise the level of abstraction of any programming language they're inserted to; in other words, they raise the level of abstraction relative to its starting point. My experience on the other hand is that raw macros instead raise the level of abstraction to an absolute level. Put crudely, if macros raise the abstraction level to X, and your language is abstraction level X-1, then macros will be a gain; but if your language is already at abstraction level X you're not going to notice much improvement.

Assuming you agree with me that raw macros aren't hugely useful for modern programming languages, you might reasonably ask: why did you continue implementing such a thing in Converge? Here we see why I've used the term raw macros earlier. Converge has raw macros because they are the lowest common denominator of compile-time meta-programming (and thus far the shipping Converge system uses precisely one raw macro call, and it's not a particularly crucial one). However Converge also contains a second feature, the DSL block which is a simple layer on top of raw macros which allows arbitrary syntaxes to be neatly embedded in a Converge file and compiled out, while still retaining excellent debugging support.

It's too early to state with confidence whether DSL blocks are a successful or practical means of improving the level of abstraction of Converge. However it does give some insight into the question posed at the beginning of this article. Macros are useful when they give the user the ability to rise above the base programming language. Thus raw macros are a boon to LISP, but offer little to Converge. DSL blocks seem to confer an advantage to Converge, but the programming languages of the future may subsume such functionality.

I do not think that there is a fundamental law of the programming universe which says macros always increase the level of abstraction. Macros aren't an end in themselves. If programming languages incorporate macros in such a way that they help users raise the level of abstraction, then they are useful. The way in which macros achieve that will evolve as programming languages evolve. And if macro technology fails to keep up, or proves inadequate for the job, then macros will no longer be useful. Already I think that LISP-style raw macros are gently heading towards obscurity. Perhaps languages such as Converge and Metalua, as immature as they currently are, will point to a new chapter in macro technology and dissemination. ]]> http://tratt.net/laurie/tech_articles/articles/filling_in_a_gap Recently I've made a previously lop-sided part of Converge much more internally consistent, and in so doing realised a useful new connection between two language features. Since it's to do with a unique part of Converge, it makes an interesting little study. In essence, Converge has a macro-esque system that is inverted from the traditional LISP/Scheme style. LISP macros are special constructs, with some ordinary looking function calls actually being macro calls. In Converge (as in Template Haskell), macros are normal functions or expressions; macro calls however are explicitly identified. In the following example, m is a normal function that intuitively is used a macro by the splice $<...> in main:

func m():
  return [| Sys::println("m") |]

func main():
  $<m()>

func my_dsl():
  return [| ... |]

$<my_dsl("""...
...
...""")>

Therefore Converge soon grew a second splice operator which I subsequently named the DSL splice operator. It is used as follows:

func my_dsl(dsl_block, src_info):
  ...

$<<my_dsl>>:
  ...
  ...
  ...

So that was where it was left for a long time. The traditional splice operator spliced unmodified ASTs in, and the DSL splice operator spliced in ASTs with extra src infos.

Recently it occurred to me that something was missing. Consider the following code:

func f():
  return [| 1.foo |]

func main():
  $<f()>

Traceback (most recent call at bottom):
  1: File "test.cv", line 2, column 12
Slot_Exception: No such slot 'foo' in instance of 'Int'.

This might sound quite limited, but is, at worst, on a par with any existing macro system I've yet come across. Many macro systems don't record any error information when creating or splicing in ASTs. About the best that I've seen is some Scheme variants which record the splice location of any error; however if a complex AST was spliced in, the user is given no clue as to which part of the AST led to the error.

At this point, the comparison with the DSL splice operator should be obvious, although it escaped me for quite some time: the DSL implementation function called by a DSL splice can customise its error reporting based on the input DSL block. However what we want for the above example isn't manual customisation of the error reporting: we want it to be created automatically. I therefore recently merged a patch into Converge which means that when a traditional splice is performed, the spliced-in AST automatically has added to it the src info for the splice location. For the above example one now gets the following run-time exception:

Traceback (most recent call at bottom):
  1: File "test.cv", line 2, column 12
     File "test.cv", line 5, column 12
Slot_Exception: No such slot 'foo' in instance of 'Int'.

Traceback (most recent call at bottom):
  1: File "test.cv", line 125, column 2
  2: File "test.cv", line 54, column 4
     File "test.cv", line 112, column 2
  3: File "test.cv", line 78, column 118
     File "test.cv", line 113, column 3
Slot_Exception: No such slot 'foo' in instance of 'Int'

So it became obvious to me while I was implementing this new functionality that I had created a symmetry - in the sense of a mirror image - of sorts between the two types of splice. Traditional splices automatically add source information about the splice location, whereas DSL splices don't. If I can think of shorter names to capture this, I may well retrospectively rename the two splice operators, as this concept is the one that most usefully captures the difference between them. More importantly this exercise gave me - if no one else - a more profound insight into splicing. I find this sort of insight, which is a relatively rare event, deeply satisfying: and it all comes from solving a little problem, filling a little gap, and then making connections after the fact. ]]> http://tratt.net/laurie/tech_articles/articles/are_multicore_processors_the_root_of_a_new_software_crisis The advent of such machines is having an odd effect on many in the software community, which is having what amounts to a collective crisis of confidence in its ability to fully utilise such machines. But before exploring this further, let's take a step back in time.

The software crisis

Then a few years back, I was at an OMG dinner (in Disneyland in California, but that's a detail I'd prefer to forget), and among the 10 or so people there, the topic of conversation shifted onto the software crisis. Various people shook their fists and banged the table - metaphorically speaking - decrying the terrible state of software. Then Jim Rumbaugh - of UML fame, and who retired from IBM / Rational last year - said something that I initially dismissed, but realised some time later was incredibly profound. To paraphrase Jim: "We've forgotten how incredible software is today. I loaded Photoshop onto my computer last week and within minutes I was manipulating photos in ways that would have been impossible for even world experts a few years ago. And that photo editing requires a reliance on many large, complex software systems that have been packaged in a way that my family of non-experts can fairly easily use."

What Jim was saying changed my thinking on software. While software could certainly be better, couldn't everything in life? The fact that things can be improved doesn't mean that the current state of affairs is intolerable in an absolute sense. In actual fact, for normal people the software that they interact with is pretty decent these days. It may not be perfectly reliable, but it's generally more than reliable enough (gone are the days where machines need to be rebooted 5 times a day to keep them stable). It may not be as easy to use as it could be, but normal people manage to do most of the tasks they need to without huge problems and that's the ultimate acid test.

A substantial reason why we worked our way out of the software crisis is that we have become much better at developing software over the last 20 years. We have better languages and tools; we understand many of the problems more thoroughly; we have been able to disseminate knowledge about software creation fair and wide; and we have become much better at reusing the increasingly large number of mature, stable programs available.

While one still does hear people witter on about the software crisis from time to time, this is gradually diminishing because the reality of software today is that it's largely fit for purpose.

Is there a multicore software crisis?

When I was talking about the software crisis earlier, I missed out the other factor in the demise of said crisis: hardware. Todays computers are so incredibly fast, so capable of dealing with mind-boggling quantities of data, that they bear little relation to those of 25 years ago. The huge increase in horsepower has made many software practises that were previously untenable - e.g. using dynamically typed languages like Python - more than useable. Consequently we have been able to develop software with increasingly little concern of the underlying hardware. In my opinion, reasonably priced PCs are more than fast enough for virtually every task that normal people throw at them, and this has been the case for the last 5 or 6 years. Hardware speed increases since that point have been largely lost on your average computer user, because the machines were already fast enough.

The last sentence of that last paragraph is absolutely key for me: for the vast majority of tasks, for the vast majority of users, machines are already fast enough. Of course we'll take more speed if it's given to us, but the lack of speed isn't generally holding us back any more.

So if you're developing a desktop application, or a standard web site, or a back-end processing system, the chances are that your development tools, languages, and methods are actually largely adequate. More accurately, the next generation of tools, languages, and methods will probably be a useful - but not radical - evolution of the current generation.

Why then are many people preaching that we need to rip all of our tools, languages, and methods to shreds and start again?

Are there situations where we need to rethink software for multicores?

I do however think that there are certain classes of problems that could benefit substantially from multicores, but which are not efficiently decomposed into coarse-grained processes. The most compelling for me is computationally intensive scientific applications. Some of these applications crunch numbers like there's no tomorrow, and often have aspects which are highly parallelizable at the fine-grained level. It might also be that some computer games could benefit similarly (given that many of them are number crunchers aimed at the entertainment domain), but frankly I'm so out of touch with that area that I don't feel qualified to offer an opinion.

My fundamental points here are: most people won't notice the difference in speed from multicores; for those of us who do benefit, coarse-grained process decomposition pushes utilisation more than high enough; and only a few very specialised domains will really benefit from multicores.

How do we best utilise multicores?

For those rare, specialised domains such as computationally intensive scientific applications I think a new approach is needed. I say this not to advocate change for changes sake, but because I believe that the developers of such applications are unusual in that they are prepared to absorb a large amount of implementation pain if they can significantly improve their execution time.

Step forward functional languages without mutable state.

I often enjoy teasing the FP community. Frankly FP has never shown itself as being a practical way to develop most systems: cute 10 line programs simply don't reflect the ugly realities of changing requirements and developers of varying abilities. But FP without mutable state has two inherent advantages over imperative approaches. First, it can largely do away locks, since locks are generally only needed to protect mutable state. Second, functional programs are often amenable to more analysis than imperative programs, and consequently parallelization optimisations are more likely to be identified automatically.

I think that this is an area where FP could finally find its niche. The path has been somewhat mapped out by Erlang, but Erlang is an outlier in that, while it's an FP language without mutable state, it doesn't have static types. Every other FP language without mutable state that I know of has a static type system. Most such type systems are merely odd on a good day, but wilfully obscure on a bad day - they get in the way too often. If someone can come up with a statically typed language which is as relatively easy to use as Erlang, but can provide large parallelization benefits, then there will be a specific class of real user out there who will gobble it up.

Conclusions

My main contention is that multicore machines aren't really going to make a big impact on most people or most developers. The vast majority of software will continue to be developed using methods that are familiar in tone to todays developers, and the resulting software will be sufficiently efficient, feature rich, and stable. However for a small class of users, existing techniques are lacking. In todays software world, which is more tolerant of heterogeneous systems than ever before, this provides a niche opening for FP if it can be packaged in a reasonable fashion.

I do want to make clear though that I don't think that FP in any of its forms will ever take over the world, but I'm going to be interested to see if FP finally manages to carve out a distinct niche. ]]> http://tratt.net/laurie/tech_articles/articles/the_high_risk_of_novel_language_features general feel. In fact some languages make virtues of this: until recently Python was proud of the fact that it had only one novel language feature, on the basis that it contained only those constructs which had proved themselves in prior languages.

The obvious question is: why don't different programming languages have novel language features? I think the reason can be most clearly seen by example. When Java appeared it had precisely one novel language feature - checked exceptions, a language feature which has no precedent. Personally when I think of the single worst feature of Java, the feature which most turns me off the language, it is - wait for it - checked exceptions. By trying to enforce good programming practice, they make life so annoying that many people actually use empty catch blocks, which means their code is less reliable than it would have been without checked exceptions. I know that I'm not alone in thinking that this is a fundamentally bad thing. Here you have the quandary: novel language features carry with them an exceedingly high risk. Most novel language features turn out to be one or more of irrelevant, dangerous, or annoying; checked exceptions come under the latter two categories. Novel language features carry with them such a high risk of failure that most sensible language designers avoid them whenever possible.

Earlier, I said that (until recently) Python had only one novel language feature. It's a small, rarely used feature: the else clause on for and while loops. One uses this as follows:

for ...:
  ...
else:
  ...

Until very recently - and assuming one discounts the novel language features related to compile-time meta-programming and DSLs - Converge also contained only one genuinely novel language feature. You may well be able to spot its lineage however: I refer of course to the exhausted and broken clauses on for and while loops. In Converge one can use these as follows:

for ...:
  ...
exhausted:
  ...
broken:
  ...

However, for many months I have been aware that there is an idiom in Converge that has sullied my code. Here's an example of the idiom:

X := 0
Y := 1
  
...

if a == X:
  // do something related to X
elif a == Y:
  // do something related to Y

else:
  raise "XXX"

A couple of days ago, when programming something with lots of enums, I realised that I had to implement something to prevent my code from being sullied further with this idiom. So I started implementing a switch statement. Half way through adding an entirely new construct to the language, I had a revelation: what if there was a variant of the if statement whose default else action was to raise a shouldn't have got here exception? And so, with some trepidation, but also a slight sense of impulsion, I implemented such a feature. It's called No Default If or ndif and for the above example one would write:

ndif a == X:
  // do something related to X
elif a == Y:
  // do something related to Y

So there you have it - the base Converge language has its second novel language feature. I hope it will be useful, but I'm far from sure. For the next 18 months or more, I will worry frequently as to whether ndif will share the same fate as Java's checked exceptions. What happens if I've polluted my nice language with a disastrous feature? The next time that you complain that languages don't contain enough novel new features, try and remember the language designers quandary.

Updated (January 26 2008): In fact, ndif might not be as unique as suggested in this article - Erlang's normal if statement acts similarly to ndif. Erlang's exception-raising can be turned off by adding the equivalent of elif true: pass. This would appear to represent a fundamental difference in language design: Converge's if caters to the common (at least 95% in my experience) case, whereas Erlang's appears to optimise the less common case. My thanks to Thomas Figg for pointing me at this facet of Erlang. ]]> http://tratt.net/laurie/tech_articles/articles/evolving_dsls musing might be being rather polite - rather, I have heard several people arguing vehemently that creating a DSL inevitably leads to doom when the requirements for the DSL change. This is a very interesting point, because in my opinion the ability for a DSL to evolve is critical.

Paul Hudak got a lot right in a sequence of papers he published on DSLs in the mid-90's (including this web-friendly version). Specifically he notes that DSLs generally start tackling a small problem, then need to grow bigger as more aspects of the problem are tackled. [He also noted - and I think he may have been the first to articulate this so eloquently - that DSLs eventually tend to evolve into a badly designed general purpose language, but that's beyond the scope of this entry.] The way in which this evolution of requirements happens is almost always unpredictable, because it is the act of building and using the DSL that gives users the insight to change their requirements. In my mind, Hudak is entirely correct; I believe that the terms DSL and need to evolve go hand in hand.

The question one has to ask oneself is thus fairly simple: are DSLs hard to evolve? I think the answer is that, yes, today DSLs are hard to evolve. Why is this? Well, there are two types of DSL in common use. The first is standalone (often called external) DSLs such as Make. The second is integrated (often called internal) DSLs such as those that Hudak talks about, or those that are frequently talked about in conjunction with Ruby. These two types of DSLs are fundamentally different. Standalone DSLs are flexible, but an awful lot of work to create, because in a sense they're a complete implementation of a mini-programming language. Integrated DSLs aren't very much work to create, but they tend to have an unhealthy coupling to their host language, which means they have many limitations imposed upon them both in terms of what they can express and how they can express it. The difference between most standalone and integrated DSLs is so severe that I sometimes wonder if the umbrella term DSL is entirely helpful, but that's an argument for another time.

The irony is that standalone and integrated DSLs share one thing in common: their implementations are typically hard to change. The reasons for this are rather different. Standalone DSLs have fairly large implementations, and are subject to all the problems that any non-trivial implementation suffers from e.g. the interaction between components is often complex and brittle. Integrated DSLs on the other hand are often small but rather hackish in nature; they frequently rely on stretching often somewhat obscure language features to near breaking point. At some point, either the language feature can be stretched no further or, worse, the whole hackish facade comes crumbling down. Please don't get me wrong: I enjoy a cunning hack as much as the next man, but cunning hacks are not what I want to base a whole approach on.

In my opinion, while DSLs are implemented in one of these two ways, they will always be hard to evolve. Therefore I agree with those who point out that, at the moment, implementing a DSL is an almost guaranteed way of giving oneself huge problems when evolution rears its inconvenient head. My argument is that the approaches I've outlined above are fundamentally flawed. What one wants is a way of implementing DSLs with relatively little code but which don't rely on abusing language features. Since small, well written programs are generally considered fairly evolvable, this should give one a reasonable chance of having DSLs that are evolvable. This has been one of my goals in the recent new version of Converge; it's far too early to tell if that's been achieved yet, but I'm fairly convinced already that this approach is, at the very least, no worse than the traditional approaches. ]]> http://tratt.net/laurie/tech_articles/articles/more_meta_matters metacircularity entry and documents one such occurrence.

Here is the problem I was facing in a nutshell. In the new version of the Converge language I was making much more use of meta-classes to hide the general ickiness of primitive datatypes. A simple example is the File class. When one opens a file for reading or writing, the resultant object needs to have extra space in memory to record the C-level file handle. Other types of objects may similarly need to have an arbitrary extra space to record some low-level information. Many languages attempt to hide this fact altogether, but that tends to rule out some useful things, and makes the language hard to extent. Far better in this case to have a Meta_File class which specifies how new File objects are to be created; users can also add their own meta-classes to create arbitrary types of objects as they see fit.

The Meta_File class (it isn't actually called that in Converge, but this name makes things easier to explain in this entry) is in C and its definition looks like this in hybrid Converge and C code:

class Meta_File(Class):
  func new(path, mode):
    Con_Obj *new_file_object = malloc(sizeof(normal object) +
      sizeof(extra space for file handle));
    ...

class File metaclass Meta_File:
  func read(num_bytes):
    FILE *handle = (((u_char *) self) + sizeof(normal object))->
      handle;
    ...

File := Meta_File.new("File", [Object], [func new(): ...])

class Read_Line_File(File):
  func readline():
    ...

The fix for this is obvious enough: the Read_Line_File class needs to declare that its metaclass is Meta_File:

class Read_Line_File(File) metaclass Meta_File:
  func readline():
    ...

So I set about devising a mechanism which would ensure that subclasses would, by default, use the same metaclass as their superclass whilst still being ObjVLisp in spirit (Python's __metaclass__ attribute, for example, is decidedly non-ObjVLisp in style). Eventually I came up with a mechanism that I was happy with, and having used it for a while, decided that it was worthy of being recorded in a research paper.

I then started hunting around for all the past work I could find on metaclasses, stumbling along the way on papers that I had seen at some point in the past, but had not been able to digest. Most of what I came across had nothing to say about the issue above, but I saw a couple of references to a concept called Metaclass Compatability. Simplified somewhat, this refers to the potential problem when a language with multiple inheritance inherits from two superclasses which have different metaclasses. An interesting read is from Nicolas Graube (or the similar Bouraqadi-Saâdani et. al., also available via CiteSeer). The problem of metaclass compatability is, frankly speaking, a largely theoretical problem - it's a corner case which I struggle to believe is likely to occur often (if at all) in practise, and the treatment of it is really rather dense.

What's interesting about the metaclass compatability research is that, once one strips away the theoretical densenes, one discovers that the solutions that are proposed for metaclass compatability effectively present a solution to the problem I outlined earlier in this entry - and which are very similar to what I eventually came up with. My first feeling was of disappointment that I hadn't discovered something novel. My second feeling was also of disappointment: if I'd been able to interpret this research up front, I might have saved myself a lot of effort. But ultimately I realised that I'd merely been a victim of the fundamental problem whenever one discusses theory and practise: even when each route leads to the same answer, that route is often impenetrable to the other side until both independently arrive at the same answer. ]]> http://tratt.net/laurie/tech_articles/articles/my_strategy_for_dealing_with_email For one reason or another, I receive quite a bit of e-mail - not as much as a few people I've heard of, but certainly enough. As time has gone by I have developed a few simple techniques that mean that I very rarely feel that my e-mail has got the better of me. Much of what I'm going to say here is not that surprising, but there are a couple of things that I think might be of interest.

My first point is that e-mail is very hard to read "flat". That is, many people I see have all the e-mail they receive delivered straight into their main inbox. This guarantees doom. Typically one gets all sorts of e-mails: private e-mails, work e-mails, mailing list e-mails, professional e-mails and so on. As I suspect most other people do, I read these different types of e-mails in different mental "modes": I don't respond to work e-mails in the same manner that I reply to private e-mails, for example. Context switching is a difficult thing to do in general, and I am particularly bad at it; it can take me several seconds to move to the appropriate mode when moving from a work e-mail to a private e-mail which, when multiplied, by many e-mails can be a significant time soak. Having observed many other people who read their e-mail "flat", it's clear to me that it's not just me who finds this type of context switching - this is, to put it mildly, an unproductive and highly intimidating way of reading e-mail.

All e-mail system have a folder feature whereby e-mails can be sorted into folders. As my e-mails are delivered to my system, they are automatically put into appropriate folders, one for each notional "subject" (e.g. one per mailing list, one per job function etc.); those e-mails which can't be automatically filed are dumped into my main inbox folder. I happen to use procmail for this on my server - it's an awful bit of software, but it just about does what I need it to. For most people the default filtering facility in programs like Thunderbird is perfectly adequate. When I open my e-mail program up, my e-mails are already sorted into related groups which I batch read, thus lowering the context switching cost to near zero.

Getting the number of folders right can be a bit tricky - too many seems to be worse than having too few. This also leads me to an important observation: folders are only useful if e-mails are in them before you read them. I see a lot of people read an e-mail and then carefully file it in a folder; while this fulfills a deep internal need to housetidy, it's counterproductive. I've seen far too many people say "I'll find that e-mail" and then have to look in 7 different folders because they're not sure which one they've filed it in. On modern e-mail programs there is no problem having tens of thousands of e-mails in a folder, so it's important to resist the urge to file for filings sake. The only reason I file e-mails into a different folder than the one they were automatically placed in is if the automatic filter (for whatever reason) wasn't able to do its job.

The other main strategy I use for e-mails is unusual. I'm sure that nearly everyone has, at some point, tried to put off whatever task they should be working on by pressing the "get mail" button, and then being grateful when a new e-mail appears to distract them from the task at hand. So some time back I changed my e-mail collection program (of which more later) so that it has two modes of e-mail retrieval: "full" and "quick". In "full" mode it collects mail from all of my folders. In "quick" mode it only downloads new e-mail for the most important subset of folders. I use "full" mode only once a day, first thing in the morning, and it downloads a frightening amount of e-mail. After that I use only "quick" mode, meaning that not only can I not distract myself with unimportant e-mail throughout the day, but that important e-mails get instant priority.

My little hack has had one unexpected side benefit: it has decreased context switching costs even further. The reason is simple: in the morning when I download reams of e-mail, I sometimes find a mailing list has had 100 messages in 24 hours. If I'd read those messages as they were posted, I'd have spent a lot of time switching to "read mailing list" mode and back again. But because there's 100 messages in one folder, I can run them through them at a rate of knots, reading only the messages I'm most interested in. Often only a couple of messages per day on any given mailing list will be of genuine interest, but it takes me less than 2 or 3 minutes to process the whole batch. The amount of time this has saved me when reading e-mails is quite astonishing: I read perhaps 80% of my e-mails in a concentrated block, and in quick order, at the beginning of the day, and am rarely distracted by e-mail subsequently.

The only possible snag is how to make ones e-mail program have "full" and "quick" modes. For reasons that those of you who know me will no doubt snigger at, I use a command-line e-mail collection program called OfflineIMAP. Because it's written in Python, one can do all sorts of weird customisations to it. My particular hack is based on the fact that OfflineIMAP ignores command line arguments it doesn't understand. So if I invoke it like this:

offlineimap -o quick

def folder_filter(foldername):

  if "quick" in sys.argv:
    return foldername not in ["INBOX.bugtraq", "INBOX.openbsd-arm",
      "INBOX.openbsd-misc", "INBOX.openbsd-ports",
      "INBOX.openbsd-tech", "INBOX.spam", "INBOX.Trash"]
  else:
    return 1

I have no idea what the anaology of this simple little hack is in other mail programs such as Thunderbird, although I'm sure it can be done one way or another. All I can say is that it has increased not only my e-mail productivity, but also my overall productivity - a win win situation! ]]> http://tratt.net/laurie/tech_articles/articles/debugging_driven_development What I'm not sure that any of these methodologies tackle sufficiently is a far more down-to-earth part of development: how to ensure developers stay motivated during the development process. What I mean by this is how can developers be kept interested in the task at hand, so as to ensure that the software they are working on moves forward at a reasonable rate? This is an overlooked factor because it is hard to define and even harder to measure. Programming is generally a long drawn out affair; even small percentage losses in productivity can have fairly large real world effects on timeliness, and on the quality of the delivered product. Since ultimately it is programmers who do the programming, the lack of focus on this area does us all a disservice.

I've known for quite some time that I suffer from a habit that is common to many programmers. When I finish a challenging part of a program - after having spent an awful lot of time looking at incorrect output of one form or another - I find myself so surprised that things are working that I then spend a silly amount of time rerunning the program and admiring the correct output. It's almost like I don't believe that it's really correct. When I eventually pull myself away from the programming equivalent of navel gazing, I find myself in a familiar, but unappealing, situation: I have a functioning, but incomplete system, and I need to add the next feature on the list to it. But adding another feature means that everything is going to stop working properly, and it's going to take an awful lot of work to get back to another point where the system will be working properly. And so I stare and stare at the screen, semi-paralyzed by the thought of breaking my perfectly functioning (if incomplete) system. Of course, eventually I pull myself together, make the first stages of the necessary change for the new feature and then I'm away, only for the cycle to repeat itself when the system reaches its next stable state. I've come to realise that all I'm going through in such instances is the programming equivalent of writers block; the fear of setting pen to paper for fear of starting down the wrong track.

Interestingly, over the past couple of years I've inadvertently stumbled on an effective technique for avoiding this problem in many circumstances. It's taken me a little while to analyse exactly what I was doing, but now that I have, I feel confident enough to coin a new development methodology (hoping that no-one has coined it before): debugging driven development.

The first thing to note from my previous description is that whenever a system I was developing got to a stable point, I paused (sometimes fairly considerably) before eventually starting on the path to the next stable point. These pauses are clearly a, if not the, major development bottleneck. As the length of these pauses seems largely outside of my control, logically - and although it may be counter-intuitive at first - the only solution to reducing the number of times I pause is to reduce the number of stable points the system is in. The less times the system reaches a stable point, the less times I pause my development.

One defining characteristic of all genuinely keen programmers is that they dislike their systems to have any errors in. If a programmer stays late after work one day, it's far more likely that they're trying to debug their system than it is that they're adding a new feature in. Therefore what I found myself doing was continually leaving some small part of my system in such a state that it needed to be debugged. So as soon as I finished one feature, there was a little bit of the system screaming at me that it needed to be fixed. Disliking any system screaming at me, I would beaver away until the message was gone. At which point another message would have popped up somewhere, demanding to be fixed. And so on and so forth. Suddenly I found the number of times I was pausing to navel gaze had diminished significantly. This may well sound like a recipe for anarchy at best, or low-quality systems at worst but the aim of this isn't to prevent the system ever reaching a stable state, nor is it the aim to leave small, and easily forgotten, bugs in. Rather the aim is to minimise the number of times the system reaches a stable state - outside of those dictated by the release schedule - by leaving blindingly obvious cock-ups in of the sort that stop the system in a predictable fashion whenever it is run. In practise I've found that ultimately the programs I've created using this approach have had significantly less bugs than those I created previously.

I've gradually refined this technique, and these days I often use the XXX exceptions I outlined in my previous entry about exceptions to insert annoying messages into the system at any point where a feature hasn't yet been filled in. Sprinkling around XXX errors leads to a situation where mentally I think I'm in a continual state of debugging because such messages appear as program aborts, but really I'm not debugging in the traditional sense of the term: I'm just using it as a carrot (or is it a stick?) to force me to continue to program. My overall productivity has increased quite significantly due to this technique.

Maybe this technique only works if your brain is wired similarly to mine, but I have a sneaking feeling it might be something that works for anyone else who takes pride in their programs. Debugging driven development might sound masochistic, or even deranged, but it works for me and it might work for somebody else - who knows? ]]> http://tratt.net/laurie/tech_articles/articles/make_exceptions_work_for_you As such I don't think there's anything wrong with a small fear of exceptions. However it seems to be gradually becoming a part of programming culture to 'do something' about exceptions, and there's really only thing that can be done to them: suppress them. While certain limited classes of exceptions - 'file not found' for example - are generally fairly benign, others - an incorrect array index for example - are far more serious. All too often, although the programmer thinks the program will keep running after an exception, instead it hobbles to a slow and painful death.

This mindset seems particularly prevalent amongst Java programmers. First, Java's horrible notion of checked exceptions positively encourages exceptions to be suppressed, just to try and keep the number of try ... catch constructs to a semi-reasonable level. Second, in multi-threaded Java applications, an exception which reaches the top level in one thread terminates only that thread; this gives the impression that exceptions in multi-threaded programs are somehow less serious than in their single threaded brethren. I have seen many Java programs randomly suppressing some exceptions, and allowing others to spew backtraces others to stdout, but continuing on anyway; this seriously undermines my confidence in the programs' correct running. But the funny thing is that people just say 'oh, ignore that exception, it doesn't seem to cause any harm.' This is a mindset that I find alien - to me, exceptions are always serious, and they always need to be fixed.

I have come to look at things in a completely different light. Although seemingly perverse at first, I don't mind exceptions being raised in my program. Obviously I don't want my program to have flaws of any kind, but since it would be foolish to think that I can create a program without flaws, the sooner flaws are identified, and the more accurate the information about them, the better. The backtraces with line numbers that most decent programming languages produce when an exception is raised are, in my opinion, the single most useful debugging aid possible.

Jim Shore's Fail Fast article articulates a similar philosophy, using assertions to terminate a program when an assumption is violated, but I find assertions to be useful only in certain limited circumstances. I actually like to take things one step further in a crude sort of fashion. I have long realised that when I'm programming a new function, I'm generally only coding the bits of the function necessary to make my current use case function correctly. The other bits of the function that will make it fully general I fill in at some later point. This, I feel fairly confident in stating, is a common development strategy. The problem with it is that sometimes one forgets to fill in the other bits of the function. Then some poor unsuspecting fool comes along, feeds a value to the function which it can't properly cater for, and gets a random results. Random results are the worst thing that can happen in a program, as tracking down something random generally degenerates into guesswork.

So what I now do is harness the power of exceptions to stop a program dead in its tracks (at least, unless one is using Java). When I come to the point where I think 'two things could happen here, but I only want to code for one of them at the moment', I raise an exception. I don't care what the exception is, merely that it is raised, and that it is easy to find in the source code. In Python and Converge, assuming I only wanted to cater for the case that x == 0, I use the following idiom:

func f(x):
  if x == 0:
    // do whatever
  else:
    raise "XXX"

With a little bit of thought, it's generally possible to get a similar effect in languages which don't have native exceptions. For example in my C programs I define the following macro:

#define XXX do { printf("XXX exit file %s line: \n", __FILE__, \
  __LINE__); abort(); } while (0)

So I say don't fear exceptions - make them work for you. ]]> http://tratt.net/laurie/tech_articles/articles/home_directory_synchronization scp) from machine to machine. This was a pain, and even with just two machines I occasionally overwrote files with old versions, or went on a trip only to discover I didn't have the latest version of a particular file on my laptop. With more than two computers the problem becomes disproportionately more difficult.

My solution to this problem is not a novel one, but nor does it seem to be particularly well known. I had the germ of the idea around three years or so ago, and largely got it working before finding that Joey Hess had already eloquently described most of the important steps; I used some of Joey's ideas to refine my setup. The idea that's fairly completely described by Joey is 'use version control to store files in your home directory.' Version control systems such as CVS are generally used so that multiple developers can work on the same source code and share their changes in a controlled fashion amongst each other. As this implies, on each developers' machine lies a (largely) identical copy of the shared source code. However there's no reason to restrict this to being of use only when multiple people are involved. If one has multiple computers, using version control software simply means that each contains an identical copy of shared files.

The benefits of taking this approach are, from my experience, almost impossible to overstate. My life has not only become significantly easier by significantly reducing the chance for mistakes, but I've also been able to be significantly more cavalier about moving between new machines, adding new machines to my menagerie, and even simply reinstalling existing machines. Of course for most normal people out there, this won't be an advantage at all since it fulfils a need you don't have, and uses a mechanism you won't want to understand, but if you're a serious computer user I think you should consider it.

I suspect one of the reasons why this method is rarely used - I know a grand total of one person in real life who uses something approaching this technique - is because of the use of "version control system" in the above text. Version control software is traditionally scary (most of the tools were one or more of big, slow, and unreliable), and of course it is seen as being applicable only to source code. In practice, even with simple tools, neither of these points is valid. Using this technique does require some thought, and it does take getting used to, but once one is used to it, the benefits significantly outweigh the disadvantages.

One thing that's interesting is that I see the list of pros, cons and irrelevancies a little bit differently than Joey and other similar write-ups.

Pros
• 1=. Automatic file synchronization across multiple machines.
• 1=. Multiple distributed backups (I always have a minimum of two copies of my latest data in locations separated by over 100 miles).
• 3. No real reason to ever remove files from your home directory since there's no significant organizational penalty for keeping them around.
• 4. Allows easy use of staging servers. I use this to develop my web site locally on various machines, before seamlessly committing it to my web server.
• 5. Ensures home directory is kept 'clean' since a fresh checkout will not check out irrelevant files (e.g. all the cruft files associated with running LaTeX, or those resulting from a C compilation) since those will never have been added to the repository.

Cons
• 1. Getting your existing files into shape to make the move to this system can be time-consuming (it probably took me around 4-5 hours).
• 2. Adding files to the repository (and maintaining the list of files which shouldn't be added to the repository) is tedious, but fortunately takes relatively little time once one is used to it.
• 3. You really need access to a server that is available anywhere on the Internet to get the most out of the technique. As most interested parties will have a DSL line, this is a very minor con.

Irrelevancies
• 1. Being able to get old versions of your files is useful. I have used this feature once. And that was just to see what would happen.
• 2. It's not practical with binary data. In fact, there's no problem with this, provided you're sensible. See divide your binary data into three types a little later in the article.

So now that I've been using this technique for a few years I feel that I have a few useful suggestions for anyone tempted to go down this highly recommended route.

Use a commonly available version control system.

An oft used alternative is Subversion but I wouldn't touch that with a barge pole, since it appears to be a project with the limited ambition of just replacing CVS. Unfortunately while they fixed some of CVS's more obvious deficiencies, they've introduced some tear-inducingly stupid new flaws. I've seen several corrupted repositories because using BSD-DB or similar for a storage backend is an obviously bad move.

At some point, one of the more advanced systems like Darcs or bzr might be well known enough to use here. But not for a few years yet I suspect.

Think before you name and add files.

A corollary of this is that the layout of the top-level directories in your home directory is extremely important. I have the following:

• .private
• audio
• bin
• misc
• photos
• research
• share
• src
• tmp
• web
• work

Divide your binary data into three types.

1. Binary data which is both 'irreplaceable' and doesn't change regularly, should be checked into the repository. Photos come under this heading.
2. Binary data which has no intrinsic value should be considered local to a particular machine. This means that it can be deleted without consequence.
3. Binary data which it is useful to synchronize, but which can be recreated by other means if necessary, is synchronized by a lighter weight mechanism. Audio comes under this category (since I own every CD I have converted into oggs, I can recreate this data if necessary) as does some programs' data (I use this to synchronize my RSS readers data).

Some lateral thinking can lead to useful savings in terms of the amount of binary data you store. For example I store only the large versions of my photos in my repository, but I've set up Makefile's so that the thumbnails and web pages that allow one to sensibly view these files are created after checkout (or any changes to the photos). Although the saving of around 15% that I get in this particular case might not seem very significant, this actually translates to a useful saving when checking out a fresh repository or manipulating files because binary data tends to dwarf textual data in size.

E-mail is special.

Automate your setup.

The second script is much less obvious: I call it cvsbootstrap. When I have a freshly installed machine, I put just this one script on there and it connects to my server and downloads all the various CVS modules etc on to the new machine. This makes the process of installing a new machine painless. The script takes two arguments maximal and minimal which determine which modules are checked out (minimal is used on irregularly used machines or on those whose security I do not entirely trust). Since I use this script relatively infrequently it is often broken by the time I use it on a new machine since I may have changed the layout of my setup in some minor way, but even when it only does 75% of the job I need it to do, it still saves me a couple of hours of remembering how long-forgotten part of my setup works. I tend to fix the error that occurred, and then check it in without any testing which reflects the unusual nature of this script.

Create a complete backup before you try this.