I sometimes find myself asked to give advice on how organisations should go about creating software, but often my advice doesn’t gel with those who sought it. Sometimes that’s because only one answer was ever considered acceptable; sometimes I am ignorant of important wider context and my answer is unworthy of consideration.
But, most often, I believe it’s because both sides have different underlying assumptions about the nature of software. These assumptions are so deeply embedded that most of us rarely, if ever, explicitly think about them. It took me many years to realise I had made such assumptions, and many more to be able to articulate them somewhat coherently.
In this post I’m going to define, as best I can, what I consider to be the triad of interacting factors that best define the nature of software. These are inevitably high-level, but my experience is that they are just about specific enough to allow one to make reasonable quality predictions about a given piece of software. I’m not claiming these thoughts to be original, nor do I claim software to be unique in having these factors as their fundamentals.
The triad
When we’re disagreeing with others about some aspect of software, we tend to focus on concrete factors we can easily define and argue about: choice of programming language, approaches to testing, file name conventions, and the like [1]. Some of these choices do clearly make software development easier or harder, but they don’t give us a deep insight into a specific piece of software, let alone software more generally.
Are there deeper, more fundamental aspects of software that can help us think beyond surface-level matters? I’ve come to think that a triad of interacting factors best explains what software is and, by extension, why software is difficult:
-
Software occupies a liminal state between the constraints of the physical world and an anything-goes fantasy world. We frequently mistake the constraints that software faces.
-
Our ability to specify what a given piece of software should be is limited by the circular specification problem. We nearly always have to fully build the software in order to know precisely what we want it to be.
-
Software is subject to the observer effect. The act of seeing the software in action changes what we – or more often others – think the software should be, sometimes radically.
Let’s now tackle each factor in turn.
Software occupies a liminal state
The physical world is subject to externally imposed constraints: we may sometimes be annoyed by the effects of gravity, but we cannot ignore them. In dreams, or fiction, we often find ourselves in a fantasy world where physical constraints can be arbitrarily relaxed: people can fly, teleport, or even travel through time.
Software occupies the liminal state between the physical world and fantasy. Hardware imposes several hard constraints upon software, most of which are widely understood (e.g. computers have finite RAM, CPUs only go so fast, and so on).
Software, in contrast, is subject to relatively few hard constraints. However, if we try and make software deal with every imaginable possibility, it will probably be too complex and too buggy — and we won’t be able to deliver it in reasonable time for a reasonable price. We therefore impose a great number of soft constraints upon our software to simplify our lives and that of users. For example, we might make our program only able to take one input file, or assume that people only live at one address at a time, and so on.
The relative merits of soft constraints are hard to quantify or to relate to the real-world. Two reasonable people can easily differ over whether adding (or removing) a given soft constraint is a good or bad idea, because so much of what each side is thinking about resides in a fantasy world that we struggle to explain to others. I might believe from past experience that a certain data-structure, though simpler, will lead to unacceptably bad performance in the long run, but I may struggle to quantify that in any meaningful way until the bad case is actually hit.
Exacerbating this are three factors.
First, we inherit soft constraints: when we use a library or operating system, for example, its soft constraints are now added to ours.
Second, we often don’t fully understand the problem domain we’re working on. We frequently add inappropriate soft constraints based on the problem domain — often without realising we are doing so!
Third, many seemingly hard constraints are actually historically contingent soft constraints [2]. For example, most of us – and I include myself in this – have difficulty thinking beyond the operating system principles embedded in Unix, which itself directly derives from the operating systems of the 1960s: there’s no reason that software has to be structured in monolithic processes as it is now; no reason why we can’t have non-transient identifiers to processes [3]; no reason for C, or even a C-like language, to be the de facto interchange language; and so on.
What makes all of this even harder is that there really are hard constraints in software, and some soft constraints are sufficiently hard that they’re better thought of as actually hard in most situations. For example, my operating system imposes various soft constraints on my software, many of which I could fix if I spent several years understanding the relevant parts of the kernel, which is rarely realistic.
The circular specification problem
The only way to know exactly what software we want to build is to fully specify it: without doing so, there will be gaps between our vague ideas and harsh reality. However, a complete, abstract specification is, in general, at least as much work as creating the software itself — in many cases, it is substantially more work.
The act of creating software is therefore, in nearly all cases, also an act of specification [4]: we have to create the software in order to specify it. That means that we almost never really know what software we’re building before, or even while, we’re building it. The opportunities for misunderstanding by individuals, and miscommunication between individuals, about what should be built is thus huge.
One common suggestion is that we could avoid this problem if we were as diligent as civil engineers and used an equivalent of blueprints. As Hillel Wayne’s excellent “Are we really engineers?” article makes clear, what differentiates software engineers from civil engineers (amongst others) is less kind than degree. Civil engineering may use blueprints (or models), but many details of the eventual structure will not be specified in them. Many details will be trivial (wall colours, door knob styles, etc.) but some are not: some buildings are built or refurbished with few if any non-supporting walls so that the first people to move in can put in the structure that best suits them.
An equivalent of blueprints sounds exactly what software needs! But, alas, it hasn’t worked out like that. Blueprints work because humans find it relatively easy to map them to their existing, shared, knowledge of real-world structures. The problem for software, as we saw earlier, is that there is no existing real-world structure we can map a software “blueprint” too [5].
One counter-argument is that this just reflects the relative youth of our subject and we’ll get better at creating specifications (or models) as we mature. Indeed, during my working lifetime, we have become notably better at producing reliable, complex, software — and at lower cost to boot! One reason for this is that whenever we can encode a category of software as a library or domain specific language [6], we implicitly make specifying the resulting software easier. As time goes on, we have applied these techniques to more categories of software.
However, it seems to me that at the same speed as we make one category of software more easily specifiable, we increase our demands on bespoke software. Unlike civil engineering, there are few hard constraints stopping us increasing our ambitions for software, and so we keep doing so. It seems plausible to me that our ability to specify software – which I expect will continue improving – will remain a semi-constant distance behind our ambitions for software.
This is compounded by the fact that there are some aspects of software that we do not know how to meaningfully specify. It’s relatively easy to specify functional correctness (e.g. “this function should return a number which is less than or equal to the maximum number in the input list”) but non-functional properties (e.g. memory usage, performance [7]) vary between hard and impossible to meaningfully specify, at least with current techniques.
The observer effect
The act of seeing software in action changes what we think the software should be. I think of it as akin to the observer effect in physics.
As the software we are building comes into being, we spot challenges and opportunities, often in equal numbers. Whether we’re fixing a flaw in our specification, or taking advantage of a new observation we’ve made, we often feel compelled to take our changed view of the world into account when creating the software.
Often, though, the person creating the software is not the observer. When we create software we tend to have a very specific use-case in mind. When others see the software, they bring to it other possible use-cases that we had not considered, some of which require changes to the software.
It’s difficult to overstate how often this happens: every time that I have given a piece of software I’ve written to other people, they have used, or wanted to use, it in ways that had not occurred to me. The observer effect thus differs from the circular specification problem in that even a perfectly specified system can still be subject to the observer effect.
This is exacerbated in many organisations who, often without really thinking about it, put impermeable layers between programmers and users. This seems to derive from a flawed assumption that only software “architects” need to interact with users. However, programmers have to fill in so many details from inevitably vague specifications that nearly every programmer is to some extent also an “architect” [8]. The greater the distance between programmers and users, the more profound the observer effect is likely to be.
The observer effect often implies considerable extra work. A common attempt to avoid this extra work is to denigrate users, ascribing their feedback to stupidity or incompetence. A less common variant is that a small number of people with iron will simply refuse all change requests, normally in the name of prioritisation. Either way, trying to fully avoid the implications of the observer effect tends to lead to terrible software.
Closing thoughts
The triad of factors I’ve presented above don’t tell us everything about software, but I believe they tell us a lot that is useful. What I find particularly interesting about the factors is the way they continually interact with each other (e.g. the observer effect is often the cause of the circular specification problem; the circular specification problem gives copious opportunities for the observer effect; etc.).
Going back to the start of this article, how do the triad of factors – whether you agree or disagree with them! – help or hinder communication between individuals over software? There are many ways of looking at this: I’ll give three different examples to give you a flavour.
By understanding that software occupies a liminal state between fantasy and reality, I have found it possible to clarify in my mind what are hard and what are soft constraints. This is helpful in two directions. Sometimes I’ve convinced people that seemingly hard constraints (e.g. software that seems fundamentally single-threaded) are not only soft constraints but can, with relative ease, be removed. Sometimes I’ve had to point out that some “obviously” soft constraints (often, but not always, those inherited from other software) are prohibitively difficult to remove.
The circular specification problem and the observer effect both imply that some degree of iteration in software development is inevitable. As soon as one accepts that logic, one must then also accept that software development will be subject to friction (i.e. unexpected inefficiencies). When I plan the future for a piece of software, I expect to encounter ongoing friction, and I try to allow for that [9]. I remain astonished at how many organisations assume that a master plan for their software can be rigidly followed.
I once dealt with an organisation who had identified a significant performance problem with their software. They were part way through implementing a technically complex, expensive, performance fix and wanted advice. I dug into the technical details but realised that I wasn’t sure how much the proposed performance fix would help. I suspected that the observer effect had been ignored, but my first conversations with users were fruitless, because they had already been told, and believed, that the proposed fix would make their life better. Eventually I realised that I needed to sit down with users and see how they used their software in the moment, irrespective of the proposed fix. It soon became clear that the real problem they faced was not what the proposed fix was tackling, and I proposed a simpler, cheaper, fix. The observer effect was real, even though the organisation had ignored it, and the observers themselves had misinterpreted it!
Ultimately, I hope that by naming these factors, and explaining as best I can what I mean by them, that they might help future conversations between people of good will who are trying to deal with the many challenges involved in creating software. Even if you disagree with them, having something concrete to disagree with might be useful!
Acknowledgments: Thanks to Dan Luu and Hillel Wayne for comments.
Footnotes
Some of these factors are also transient. We guess (or assume) which are transient or not at our peril.
Some of these factors are also transient. We guess (or assume) which are transient or not at our peril.
In other words, the current state of software engineering was not inevitable. Some decisions in the past could easily have gone another way, and we’d have ended up be in a different – possibly better, possibly worse – present to the one we actually inhabit.
In other words, the current state of software engineering was not inevitable. Some decisions in the past could easily have gone another way, and we’d have ended up be in a different – possibly better, possibly worse – present to the one we actually inhabit.
I talked about the problem of “PIDs” in my reflections on writing Unix daemons. We could, for example, allow users to obtain “pointeresque references” to a process, where the process might only finally be removed from the system when there are no remaining references to it.
I talked about the problem of “PIDs” in my reflections on writing Unix daemons. We could, for example, allow users to obtain “pointeresque references” to a process, where the process might only finally be removed from the system when there are no remaining references to it.
There are, or at least should be, some exceptions such as aviation. They are rare.
There are, or at least should be, some exceptions such as aviation. They are rare.
UML has shown convincingly that it’s easy to give a high-level sketch of software, but attempts to turn it into something approaching the detail of blueprints did not end well. I don’t think that’s a reflection on UML per se, but reflects a deeper reality.
UML has shown convincingly that it’s easy to give a high-level sketch of software, but attempts to turn it into something approaching the detail of blueprints did not end well. I don’t think that’s a reflection on UML per se, but reflects a deeper reality.
Examples of this include things like e-commerce sites: most e-commerce sites need slight customisation, which is really programming in disguise, but few need much flexibility.
Examples of this include things like e-commerce sites: most e-commerce sites need slight customisation, which is really programming in disguise, but few need much flexibility.
Although it’s rarely thought of this way, the famous Knuth quote “premature optimization is the root of all evil” can be seen as a reminder not to assume that we can accurately specify performance in advance.
Although it’s rarely thought of this way, the famous Knuth quote “premature optimization is the root of all evil” can be seen as a reminder not to assume that we can accurately specify performance in advance.
Another way of looking at this is that the gap between software “architects” and programmers is much smaller than the gap between civil-engineering architects and (say) brick-layers.
Another way of looking at this is that the gap between software “architects” and programmers is much smaller than the gap between civil-engineering architects and (say) brick-layers.
The very nature of friction means that one can not anticipate its exact nature or magnitude in advance. What one can do, though, is not to create a plan such that the only possible route to success rests on encountering no friction at all.
The very nature of friction means that one can not anticipate its exact nature or magnitude in advance. What one can do, though, is not to create a plan such that the only possible route to success rests on encountering no friction at all.