More Meta Matters

August 30 2006

Every so often, computing theory and practise collide, and one finds that the same answer has been produced by coming it at it from the two extreme angles. Alas, it is so difficult to locate the common ground between these two extremes that this is a rare occurrence; but it does happen. This entry is in some senses a follow-up to the metacircularity entry and documents one such occurrence.

Here is the problem I was facing in a nutshell. In the new version of the Converge language I was making much more use of meta-classes to hide the general ickiness of primitive datatypes. A simple example is the File class. When one opens a file for reading or writing, the resultant object needs to have extra space in memory to record the C-level file handle. Other types of objects may similarly need to have an arbitrary extra space to record some low-level information. Many languages attempt to hide this fact altogether, but that tends to rule out some useful things, and makes the language hard to extent. Far better in this case to have a Meta_File class which specifies how new File objects are to be created; users can also add their own meta-classes to create arbitrary types of objects as they see fit.

The Meta_File class (it isn't actually called that in Converge, but this name makes things easier to explain in this entry) is in C and its definition looks like this in hybrid Converge and C code:

class Meta_File(Class):
  func new(path, mode):
    Con_Obj *new_file_object = malloc(sizeof(normal object) +
      sizeof(extra space for file handle));
    ...

The File class is then defined as:

class File metaclass Meta_File:
  func read(num_bytes):
    FILE *handle = (((u_char *) self) + sizeof(normal object))->
      handle;
    ...

It's important to note that, in an ObjVLisp style system, this is really short hand for explicitly creating the class File by calling the new slot in the Meta_File object (which is effectively the Class.new function):

File := Meta_File.new("File", [Object], [func new(): ...])

So far, so good. Now let's assume the user wants to make a subclass of File which sports a new method readline which reads in a line of text rather than a fixed number of bytes. It would seem that this is a reasonable defintion:

class Read_Line_File(File):
  func readline():
    ...

However if one creates an instance of Read_Line_File then Object.new rather than Meta_File_New.new will be used to create the object: no space will be set aside to store the C-level file handle. Fortunately in Converge, while nothing bad happens (i.e. the program doesn't throw a wobbly at the C-level), trying to do anything much with the resulting object will lead to an exception being raised as the VM notices that it is not being given a chunk of memory with a C-level file handle in it.

The fix for this is obvious enough: the Read_Line_File class needs to declare that its metaclass is Meta_File:

class Read_Line_File(File) metaclass Meta_File:
  func readline():
    ...

This makes everything work as expected but, to my mind, is distasteful. There is now a strong coupling between a class, its metaclass, and its subclasses. At best it leads to annoying, easily fixable errors; at worse, it makes refactoring extremely difficult because one has to change the metaclass of each subclass. Although many people do tend to get unduly vexed about coupling of elements within a program - any realistic system is going to have a reasonable degree of coupling, no matter how many patterns etc. one uses - it is better to avoid coupling when possible, especially when it is this pervasive.

So I set about devising a mechanism which would ensure that subclasses would, by default, use the same metaclass as their superclass whilst still being ObjVLisp in spirit (Python's __metaclass__ attribute, for example, is decidedly non-ObjVLisp in style). Eventually I came up with a mechanism that I was happy with, and having used it for a while, decided that it was worthy of being recorded in a research paper.

I then started hunting around for all the past work I could find on metaclasses, stumbling along the way on papers that I had seen at some point in the past, but had not been able to digest. Most of what I came across had nothing to say about the issue above, but I saw a couple of references to a concept called Metaclass Compatability. Simplified somewhat, this refers to the potential problem when a language with multiple inheritance inherits from two superclasses which have different metaclasses. An interesting read is from Nicolas Graube (or the similar Bouraqadi-Saâdani et. al., also available via CiteSeer). The problem of metaclass compatability is, frankly speaking, a largely theoretical problem - it's a corner case which I struggle to believe is likely to occur often (if at all) in practise, and the treatment of it is really rather dense.

What's interesting about the metaclass compatability research is that, once one strips away the theoretical densenes, one discovers that the solutions that are proposed for metaclass compatability effectively present a solution to the problem I outlined earlier in this entry - and which are very similar to what I eventually came up with. My first feeling was of disappointment that I hadn't discovered something novel. My second feeling was also of disappointment: if I'd been able to interpret this research up front, I might have saved myself a lot of effort. But ultimately I realised that I'd merely been a victim of the fundamental problem whenever one discusses theory and practise: even when each route leads to the same answer, that route is often impenetrable to the other side until both independently arrive at the same answer.

Link to this entry