Forty years of building abstractions, tools and systems
I’ve been programming since the early days of home computers and writing software professionally for more than four decades.
Looking back across that time, the technologies seem almost impossibly varied.
Z80 and 6502 assembly. BASIC. Pascal. 8086 assembly. C and C++. DOS, OS/2, Windows, Mac and Unix. CORBA. Java. XML, SOAP and REST. Payments and security. Distributed systems. APIs. Cloud platforms. And, most recently, AI-assisted software development.
It would be easy to tell that story as a succession of technologies.
But recently I started looking back through some of the software I’d written, including source code I hadn’t seriously examined in decades, and something else became apparent.
The technology kept changing. The problems didn’t.
Again and again, I seem to have encountered some awkward piece of machinery, tried to understand how it really worked, separated the essential complexity from the accidental complexity, and then built something so that the same problem didn’t have to be solved repeatedly.
Sometimes that meant a library.
Sometimes a platform abstraction.
Sometimes a programming language.
Sometimes an interface definition language.
And sometimes it meant deciding that the abstraction everyone else was reaching for wasn’t actually necessary.
It Started with Small Machines
My first computers in the early 1980s were machines where understanding what happened underneath wasn’t optional.
Memory was measured in kilobytes. Processors such as the Z80 and 6502 were small enough that you could understand a great deal of what the machine was actually doing.
I programmed in BASIC, Pascal and assembly.
Later came 8086 assembly and DOS, followed by C and C++. BASIC also reappeared professionally.
Those early machines taught me something that has stayed with me ever since:
Abstractions are enormously useful, but understanding what lies underneath them matters.
Perhaps unsurprisingly, it wasn’t long before I started creating abstractions of my own.
RAMBASIC — 1985
In 1985 I wrote something called RAMBASIC.
It was a fairly rudimentary preprocessor that added structured constructs to BASIC, including things along the lines of:
@if
@case
@procedureThe BASIC implementation underneath didn’t suddenly acquire those capabilities. RAMBASIC translated the higher-level representation into something the underlying environment understood.
I certainly wasn’t thinking about language-design principles in those terms at the time.
I simply had an environment that did almost what I wanted, and built something on top of it that made it easier to express the program I actually wanted to write.
It turns out I would revisit that idea.
Several times.
Getting Underneath DOS
Another early project was a library I wrote in C for building DOS TSRs.
For anyone who has successfully erased DOS terminology from memory, TSR meant Terminate and Stay Resident. A program could terminate while leaving part of itself resident in memory, later appearing in response to a hotkey or interrupt.
My library was used to build an early popup hypertext help system.
That required dealing directly with the awkward realities of DOS: interrupts, memory, screen state, keyboard handling and interaction with whatever application happened to be running underneath.
The purpose of the library was to make all of that machinery manageable for the application using it.
Again, the pattern was there.
Understand the ugly bit once. Put a sensible interface around it.
OnScreen/2
During the OS/2 era I wrote OnScreen/2, inspired in part by Vernon Buerg’s excellent LIST utility for DOS.
It was primarily a programmer’s file viewer.
It included syntax highlighting for multiple programming languages, a very fast plaintext search based on Boyer-Moore, regular-expression searching and a hexadecimal viewer.
I recently went back through the surviving source.
That was an interesting experience.
There are ideas in there that I still recognise.
There is also code that makes me very glad modern C++ gives us RAII, standard containers and smart pointers.
Thirty years provides a wonderfully safe distance from which to review your own memory-management decisions.
But OnScreen/2 also contains the beginnings of something that became much more important to me later:
portability as an abstraction in its own right.
When Portability Was a Real Problem
Modern C++ programmers can take a remarkable amount of platform independence for granted.
That wasn’t always the case.
I worked across DOS, Windows, OS/2, Mac and Unix systems. At Workhorse, for example, I worked on the AT&T Rhapsody system across Unix and Microsoft Windows.
Unix itself was hardly a single target.
Linux, Solaris, HP-UX, AIX and the other Unix variants shared a heritage and broadly similar APIs, but maintaining a substantial common C or C++ codebase across them exposed differences surprisingly quickly.
One of my early inspirations in this area was the Rabbit Software book on software portability.
Eventually, portability became a problem I wanted application developers not to have to keep solving.
That led to VPTK.
VPTK — the Virtual Platform Toolkit
In the late 1990s I developed VPTK, a Virtual Platform Toolkit written in C++.
The objective was to hide differences between operating systems and Unix implementations sufficiently that applications could maintain a largely uniform codebase across Linux, Solaris, HP-UX, AIX and ultimately Windows.
VPTK grew well beyond a collection of portability macros.
It provided abstractions around threading, synchronisation, sockets, streams, files and other platform facilities.
It also included Bind/Bindable, an intrusive smart-pointer model intended to make object ownership explicit and eliminate many of the memory-management bugs that were extremely easy to create in C++ at the time.
Another part was a fault-tolerant server architecture consisting of a controller supervising multithreaded workers. If a worker crashed, the controller could automatically restart it.
I recently reviewed the surviving VPTK source.
Some of the architectural ideas hold up surprisingly well.
Some of the implementation doesn’t.
In particular, I found concurrency mistakes I certainly wouldn’t reproduce today.
That distinction matters.
A good abstraction can eliminate an entire class of problems — but only if the abstraction itself is correct.
Putting a clean API around something complicated doesn’t magically make the implementation underneath correct.
I still have to understand the machinery.
Into Distributed Systems
By the end of the 1990s, my work had increasingly moved into distributed systems.
At IONA I was working in the CORBA world, where distributed objects and portable enterprise systems were very much at the centre of software architecture.
Then, around 2000, I moved into payments with Orbiscom.
Payments would become the industry in which I spent much of the next quarter-century.
But even there, I kept encountering variations of the same engineering questions.
How do I define contracts rigorously?
How do I isolate implementation details?
How do I make distributed systems reliable?
How do I build security into the architecture?
And how do I stop every development team from solving the same infrastructure problem independently?
OIL — the Orbiscom Interface Language
Around 2005/2006, while at Orbiscom, I created OIL — the Orbiscom Interface Language.
OIL provided a rigorous way of describing software interfaces.
From an OIL definition I could generate artefacts including client-side code, server-side stubs, XML schemas and interfaces to Oracle stored procedures.
The surrounding technology continued to evolve.
OIL ultimately supported SOAP-style messaging and JSON-based REST interfaces as well.
Looking back, what interests me isn’t whether an interface happened to use XML, SOAP, Oracle or JSON.
Those were representations and implementation technologies.
The important part was separating:
what the interface means
from:
the repetitive machinery required to implement it.
Define the contract rigorously at the right level.
Generate the plumbing underneath.
That idea would return too.
Twenty-Five Years of Payments
Payments became a major part of my career from 2000 onward.
At Orbiscom I worked on tokenised card payment technology, long before tokenisation became something consumers routinely encountered through mobile wallets and modern card networks.
I subsequently spent many years at Mastercard.
Over that period my work touched payment systems, security, authentication, distributed platforms and large Java systems.
The scale changed enormously from the computers on which I started.
The underlying engineering questions often didn’t.
A payment system cannot simply be mostly correct.
Failure modes matter.
Concurrency matters.
State matters.
Security boundaries matter.
Operational behaviour matters.
Abstractions matter enormously, because no human being can reason about an entire large distributed system at once.
But abstractions can also become part of the problem.
Which leads to something I’ve become increasingly convinced of.
Complexity Has to Earn Its Place
I’ve apparently spent quite a lot of time over the years being sceptical about fashionable complexity.
Cloud isn’t automatically the answer.
Microservices aren’t automatically the answer.
Distributed systems are certainly not automatically the answer.
Adding another framework doesn’t necessarily simplify anything.
Neither does adding another architectural layer because the diagram looks cleaner afterwards.
This might sound slightly contradictory coming from someone who has spent much of his career building abstractions.
I don’t think it is.
There is an important difference between removing complexity and hiding complexity.
A good abstraction means a difficult problem can be solved correctly once rather than badly a hundred times.
A bad abstraction merely relocates the complexity, adds some of its own, and makes the original problem harder to see.
So one of the principles I’ve gradually arrived at is:
Complexity has to earn its place.
Sometimes a distributed architecture is exactly what a problem requires.
Sometimes a process needs asynchronous messaging, durable state, retries, leases and distributed coordination.
And sometimes a PostgreSQL table will do perfectly well.
Knowing the difference is engineering.
Nova — Back to Language Design
More recently I created Nova.
Nova is an API specification language that transpiles to OpenAPI.
It embodies two related ideas that matter to me.
The first is rigorous API definition control.
Rather than treating a large, free-form OpenAPI document as the most convenient design language, Nova provides a controlled language in which I can define the API itself.
The second is standards-based interoperability.
Nova isn’t intended to replace OpenAPI with a proprietary ecosystem.
It generates industry-standard OpenAPI schema documents.
The relationship is deliberately simple:
Developer intent
│
▼
Nova
│
▼
Rigorous API definition
│
▼
OpenAPI
│
▼
Existing industry toolingThe developer works at the level appropriate to defining the API.
The wider ecosystem gets the standard representation it already understands.
Nova isn’t OIL rewritten twenty years later.
The problems are different.
But I can certainly see the family resemblance.
OIL ───────► rigorous interface definition
│
▼
generated implementation
artefacts
Nova ───────► rigorous API definition
│
▼
industry-standard
OpenAPII hadn’t consciously set out to revisit the same architectural idea.
Apparently some ideas take a while to leave me alone.
And Then There Was BASIC. Again.
This is where the recurrence becomes slightly ridiculous.
Recently I’ve also been developing BASCAL and Pasic.
BASCAL returns to the world of classic Microsoft BASIC, providing structured programming facilities while remaining rooted in that programming environment.
Pasic goes considerably further.
It is an experimental Pascal-inspired language that transpiles to Microsoft BASIC.
That creates some interesting implementation problems because Pasic supports concepts that BASIC wasn’t naturally designed to provide.
Its generated runtime supports things such as procedures and functions, recursion, activation records, reference parameters, pointers and dynamic arrays.
The programmer works with the higher-level semantics.
The compiler and generated runtime deal with representing those semantics in the much more constrained underlying environment.
Which gives me this rather improbable sequence:
1985 2020s
RAMBASIC ───────► BASCAL ──────────────► Pasic
│ │ │
└──────── structured programming ──────┘
over Microsoft BASICRoughly forty years after writing a preprocessor to make BASIC more structured, I somehow found myself building languages that make BASIC more structured.
I can’t even blame that one on changing business requirements.
The tooling has improved considerably, at least.
Two Ideas That Refused to Go Away
Put those projects alongside OIL and Nova and two rather long-lived threads emerge.
Languages
1985 2020s
RAMBASIC ─────────► BASCAL ─────────────► PasicDifferent levels of sophistication, but a recurring question:
Can I give the programmer a better language without requiring the underlying machine to understand it?
Interface Definition
2005/06 2020s
OIL ───────────────────────────────────► Nova
│ │
▼ ▼
Rigorous interface Rigorous API
definition definition
│ │
▼ ▼
Generated implementation Industry-standard
artefacts OpenAPIThese aren’t the same systems.
They don’t solve exactly the same problems.
But both start from another question I seem to keep asking:
What is the right level at which the developer should express intent?
Once I have that, machinery can translate the intent into whatever representation the underlying systems require.
Reusable Infrastructure
The same thinking appears in more conventional infrastructure projects.
My more recent Mango4j work extracts recurring infrastructure concerns so that applications don’t continually have to reinvent them.
Obsinity similarly reflects my preference for understandable, conventional building blocks where possible rather than introducing infrastructure simply because it has become fashionable.
The names and technologies differ.
The question remains:
Which parts of this problem should every application really have to solve for itself?
If the answer is “none of them”, there may be an abstraction waiting to be built.
If the answer is “this problem doesn’t actually need another abstraction”, then I shouldn’t build one.
Both conclusions are useful.
Explain What’s Behind the Abstraction
There’s another part of this that matters to me.
I don’t particularly like abstractions presented as magic.
An abstraction should make understanding the implementation optional, not impossible.
Most developers shouldn’t need to understand every detail of a thread scheduler to use a mutex.
But when behaviour matters — particularly when something fails — somebody needs to understand what’s underneath.
That means documenting not merely how to use an abstraction, but why the abstraction exists.
What problem was it designed to solve?
What assumptions does it make?
What does it guarantee?
What doesn’t it guarantee?
Where does the underlying machinery leak through?
I’ve increasingly found that explaining those things is as important as writing the abstraction itself.
Perhaps that’s another consequence of starting on machines where the distance between the program and the hardware was very small.
Eventually every abstraction leaks.
It helps to know what’s on the other side.
Then AI Changed the Economics
One thing really has changed dramatically in the last few years.
The cost of turning an idea into working software.
I’ve accumulated a lot of ideas over four decades.
Some became products.
Some became libraries.
Some became prototypes.
Others remained notes, half-finished experiments or simply ideas filed mentally under:
That would be interesting if I ever had the time.
There was never enough time.
AI-assisted software development has changed that.
It has dramatically reduced the mechanical distance between an idea and something I can actually run.
Implementation, tests, documentation, refactoring, investigation and experimentation can all happen much faster.
That’s one reason projects such as Nova, BASCAL, Pasic, Mango4j, Obsinity and others have appeared in relatively rapid succession.
Many of those ideas didn’t suddenly occur to me because generative AI arrived.
Some had been sitting around for years.
AI changed whether they were economically worth pursuing.
That distinction matters.
AI didn’t give me the ideas. It changed which of them I could afford to pursue.
And that has been unexpectedly liberating.
After forty-plus years, I’ve accumulated enough unfinished ideas to keep several engineers busy for quite some time.
AI gives me a realistic chance to explore considerably more of them.
But there’s an important caveat.
AI can generate code very quickly.
It can also generate the wrong code very quickly.
It doesn’t remove the need for engineering judgement.
If anything, reducing the cost of implementation shifts more of the value towards deciding what should be implemented in the first place.
I still have to ask:
- Is this the right problem?
- Is this the right abstraction?
- Does this really need to be distributed?
- Does this need another dependency?
- What happens when it fails?
- Is the generated code actually correct?
- Am I making the system simpler, or merely generating complexity faster?
That last question may be the most important one.
Forty Years, One Engineering Thread
If I were to draw the whole thing now, I wouldn’t draw a conventional career timeline.
I’d draw something closer to this:
UNDERSTAND
│
▼
DEFINE
│
▼
ABSTRACT
│
▼
BUILD
│
▼
EXPLAIN
│
│
┌─────────────┴─────────────┐
│ │
▼ ▼
REMOVE COMPLEXITY UNDERSTAND WHAT REMAINS
│ │
└─────────────┬─────────────┘
│
▼
┌─────────────────┐
│ RAMBASIC │
│ OnScreen/2 │
│ VPTK │
│ OIL │
│ Payments │
│ Nova │
│ BASCAL │
│ Pasic │
│ Mango4j │
│ Obsinity │
└─────────────────┘
│
▼
COMPLEXITY HAS TO
EARN ITS PLACEThat describes the journey better than a list of employers, programming languages or job titles.
The processors changed.
The languages changed.
The operating systems changed.
The scale changed enormously.
The industry changed.
The abstractions certainly changed.
Apparently the underlying instinct didn’t.
Understand it.
Define it.
Abstract it.
Build it.
Explain it.
And make the complexity justify itself.
After more than forty years, I’m still working on variations of the same problems.
Fortunately, they’re still interesting.