The YouTube channel, Technology Connections has a tradition of “No Effort November” videos, taking a break from highly-researched, longform content to produce something shorter and more improvised. I have been procrastinating finding time to write lately and so thought taking that gimmick could give this dying blog a new lease on life. With that preamble/excuse for what will follow...
The UNIX Philosophy
Since I’m not doing research, I can’t tell you when “the UNIX philosophy” started or who is to be credited with its invention. What I can say is that (a) programmers treat the philosophy in reverent terms and (b) it’s embodied clearly in some programs built by programmers and for programmers and practically nowhere else. So: the philosophy.
If you are a regular computer user, you may never have booted up a command line (cmd or Powershell on Windows, Terminal on Mac). If you open up one of those programs, you will be confronted with a dark rectangle with a prompt where you can type text. Press enter and, unless you have mastered the incantations, you’ll be greeted with something like this.
Riveting stuff. What can this strange rectangle do? Actually, almost anything. Before computers had “graphics” they had a terminal display: you just had a keyboard to enter text and a display that could show you a few rows and columns of text at a time.
To make a computer do anything, you have to give it instructions, run a program. The very earliest programs were executed directly on a computer. But pretty quickly computers become powerful and widespread enough that this direct execution becomes limiting. You really want your computer to be able to run multiple programs at a time. To that end was the “operating system” (OS) born. An OS is a program that directly controls the machine and provides a home for other programs.
UNIX was an early operating system (or rather family thereof, which would go on to spawn Mac and Linux and thereby iOs and Android) and foundational to how this whole “getting multiple programs to live comfortably and usefully together” story plays out. The solution: define general interfaces for that interaction—at the core here are files, places where text may be read from or written to, and “standard input/output,” a standard (as the name suggests) place to read and write text—and provide a library of useful programs that operate on that interface from which other programs may be legod together. The UNIX philosophy refers to the approach to those foundational programs: try to build programs that have a single purpose and a simple, standard interface that makes it possible to plug that program into other programs.
To make this somewhat concrete, take two foundational UNIX programs: ls and grep. ls (short for list because when you have a tiny CRT monitor you want to shave all the characters you can) lists the files in a directory and outputs them as text on the standard output, one file name per line. grep takes a piece of text to look for and searches each line of text and outputs the ones that contain the search pattern. If say you were looking in a folder of photos for one you knew was called “something Birthday Party something,” you could enter the following commands into your command line ls Photos | grep Birthday Party and it would return all the files in photos called “something Birthday Party something.” That way you don’t need a “find my photo called Birthday Party” program or a “find my photos called whatever” program or even a “find my whatever files” program. You just need a “list the files” program and a “search the thing in a list” program and a way to put them together. (Although UNIX has a “find my whatever files” program; it’s called find. As Barbossa says, the code is more guidelines than rules.)
Why the UNIX Philosophy Did Not Win
In the first instance, this UNIX philosophy is a design philosophy, but it can be more broadly used to describe a consumer preference for programs designed in this way, to be uni-taskers that interoperate well with other programs. But while that preference exists, it exists strongly only within a niche of users: neckbeards (affectionate). So although the UNIX philosophy is reflected fairly well in how the programs at the foundational layer of modern computing work, it can barely be seen in the applications where users actually spend most of their time (outside again of the neckbeard community). There are a range of causes here, moving from the generic to the historical.
“Good and bad design.” If you read any basic design book, the author will spend some time talking about affordances, features of a design that clue the user in on how they’re supposed to use the object. A door will have a handle on the side you’re supposed to pull and a flat bit of metal on the side you’re supposed to push, or else it will find itself in the pillory list of 99% Invisible. Software is too complicated, usually, to have such simple affordances, but even where it needs a manual to explain its mistifying UX, most consumer software at least has a clearly legible answer for “what am I supposed to be using this for?” Photoshop exists to edit in Simpsons’ characters into wedding photos, and to give Adobe direct access to your bank account. Chrome exists for uploading those edited photos onto Facebook and to give Google direct access to your every waking thought and desire. Programs designed according to the UNIX philosophy have a unique user experience problem where it might be very clear what the product does, but not at all clear why you would want what the product does. After all, it’s only in combinations with other programs (as in our example) that these building block programs become really useful; therefore the UNIX philosophy demands considerable knowledge and creativity of its consumers to see results.
“The Contradictions of Capitalism.” Talk to any product person in tech, and you’ll hear them talk about adoption rates (how quickly are people starting to use our product) and retention (how long do they stick around once they start using it). Under this rubric you’ll also hear the term switching cost, how difficult it is for a user to switch from using a competitor’s product to using yours or, in the other direction, how expensive to switch from you to a competitor. Tools built according to the UNIX philosophy naturally have extremely low switching costs. After all, they’re meant to do one straightforward thing and act on generic interface, so that it’s in theory quite easy for a user to replace your version with another. Therefore, profit-oriented firms have little reason to put a program built according to the UNIX philosophy to market and so the principal examples of such program are free and open source programs. Insofar as the market is the primary funding mechanism for the production of programs, most programs will be built instead in a closed, “walled garden” form.
“The Internet.” Early, open visions of the internet rhyme with the UNIX philosophy. The internet would consist of an ecosystem backed by common interfaces (HTTP, HTML, DNS) where anyone could host a website and different websites would interface with one another across these standards. However, the technical fact that high speed internet means that the data necessary for a task can live on the server side rather than the client side means that web applications can become even more of a walled garden than a local application. At least with Word its files live on your machine and can be opened in other programs, which can provide a way for “UNIXy” programs to interface with the non-UNIXy ecosystem (e.g. I can use my beloved
pandocto bridge between the UNIXy world of text editors over markdown folders and the corporate hellscape of people emailing docx files back and forth). With Word online, Microsoft controls not just the format, but the file itself and how it may be accessed.“The App Store.” Another genuine disadvantage of the world of UNIX programs is security. The world in which I make things happen on my computer by plugging together various programs operating over my files is a world in which any of those programs may be doing some real gnarly shit and we are in the world of full caveat emptor. If I typo an
rmand blow away all my data, git gud scrub. Partly because Google and Apple are user focused benevolent dictators who don’t want you to pwn yourself, but mostly because they’re capitalist hellbeasts who want to own everything forever, when presented with the opportunity to radically rethink the UNIX legacy for their phone operating systems (while still building on its technical legacy under the hood), the two companies decided to put all programs that would run on your phone in sandboxes. This means that each program has its own data and lacks the power to maliciously tamper with other programs. And of course the price of this beneficence is that Google and Apple own the marketplace on which programs can be listed for their platforms and take their cut. This also means that the system itself is actively hostile to UNIXy programs.
AI and the Revenge (?) of the UNIX Philosophy
After having its lunch money stolen by jocks and retreated to DnD hour in the basement with its nerd friends, the UNIX philosophy might be having a glow-up. Someday it may even get to kiss the prom queen. This is happening with AI on three different fronts.
UNIXy programs are how AI agents get shit done. The learning curve I was describing earlier that makes operating UNIXy programs difficult for humans? Well UNIXy programs tend to be text in/text out affairs (so a natural fit for a “language” model), they have been around and in use by the kind of people who use StackOverflow for fun (so are well represented in training data), and used well unlock the ability to “use a computer” for a model (so labs have every incentive to build massive RL envs to train their models on how to use these programs). As a result, frontier models are superhuman levels of good at operating these programs. And while the labs have made significant progress at training models to handle the clicky clicky drag and drop nightmare that is the walled garden of non-UNIXy programs, this remains by nature less token efficient and considerably slower than having the model blast out a mess of bash.
AI-forward software firms, convinced that most program use going forward will be done by AI agents on a user’s behalf, rather than users themselves, have extended the interfaces of their programs to be more accessible to AI. In this world of MCPs (the protocol for an “AI-accessible” server invented by Anthropic and donated to the Linux foundation), the internet is returning, if only in a very limited way to its open, UNIX-adjacent roots.
LLMs themselves are UNIXy. All models have essentially the same “text in/text out” interface of a UNIX program. This means that “agents,” the programs built on top of LLMs, can be assembled in the same way that a program can legod together out of UNIXy building blocks. Paired with (1), this means that an LLM itself can become a new kind of foundational text-based interface to a computer in the same way that a command line is, but one with considerably lower learning curve. (For an inkling of what that might look like, see Ben Vallack’s videos.)
Will the UNIX philosophy be the new belle of the ball? Well, we’ve seen this dance before. In the days of the early Internet, open standards had such a pride of place because, in a new technological arena, user adoption is more important than user retention. So firms are less concerned with shutting out their competitors in a space and have more appetite for collaborative efforts to define open standards that makes the space as a whole more attractive. Over time, the focus shifts back towards competition within the space wherein players try and establish their own walled gardens (but within the open field defined by earlier partial collaboration). Smart phones are perhaps a clear example of how by contrast a “walled garden of walled gardens” may emerge when the entire emerging space is owned by a single firm. So far it seems generative AI is more like the early internet in having a degree of openness; however, the new category of the Grok bot, the Meta Muse, the OpenAI dots, indicate firms are again trying to build a closed ecosystem on top of of an open foundation. It would not surprise me if the UNIX philosophy again slinks back to the remit of nerds like me.