Against the spirit of no effort November, I read a paper for this post. More in the spirit, I only read one and, as the post itself will demonstrate, I didn’t think about it very hard. That paper was John Danaher’s 2016 Philosophy and Technology article “The Threat of Algocracy.”
Intro: What’s an Algocracy?
Aristocracy rule by the aristocracy (helpful). Democracy rule by the demos (more helpful). Algocracy rule by algorithms. (Most helpful.) Think actuarial algorithms that decide how much you will need to pay for the privilege of getting to pay for medical care. More to this government target here (though I don’t think we should entirely forget about private actors here), think of the IRS asking the computer who to audit (not the rich, please and thank you), or DOGE asking Grok how many Africans should be left to die (as many as possible, ideally).
Algos are bad because (a) hidden, you can’t as someone affected by the algorithm know all the data that went into the process and so on what basis the decision was made, and (b) opaque, even if you had all that info, the algorithmic process itself is in general too complex to be tractable. So the public in general cannot understand why a decision affecting them made on the recommendation of an algorithm was made. But the tradition of liberal democratic theorizing holds some version of an intelligibility requirement on the legitimacy of a governing scheme: people should be able to understand (in general) how the scheme is being applied to them. So, for instance, in the courts, the judge’s role is not only to apply the laws of the land but to explain the basis on which they are applying those laws. An algorithm is not a conscious being, it cannot explain itself. It just does, and may God have mercy on your soul.
Is That Right?
Danaher explores some ways in which algocracy could perhaps be “accommodated,” i. e. in which an algorithm could be used in a governing scheme while being not so objectionable.
Human intervention: either at the level of the selection and implementation of algorithms (to insist on using interpretability), or via an “appeals process” whereby a person affected can demand review or even seek a second, human decision on an automated decision. Problem: at best this probably makes algocracy less hidden, but if the algorithmic decisions are made on “Big Data,” as one expects they would be, then they’re still opaque in the sense of being too complicated for our puny brains.
Epistemic enhancement: if the problem of opacity comes down ultimately to the fact that humans are stupid, why not make humans less stupid? Problem: human brains would have to be made way, way bigger and it would probably cost too much in terms of nootropics and hats.
Algorithm on algorithm violence. Step one: have “the public” carry around their own recording devices and collect massive amounts of data on the system’s decision making. Step two: ??? Step three: democracy is saved. The only somewhat plausible value of that “???” is using your own algorithms to reverse engineer and challenge how the government is employing its algorithms. Problem: you don’t understand how your algorithm works.
I think there’s more to say particularly about 3, particularly in a world with LLMs.
Can Algorithms Explain Themselves?
Interpretability is a purely economic problem for software engineering firms. If their software fails or produces a bad outcome, they have a strong incentive to be able to understand how that outcome came about so as to fix the cause and prevent future recurrences. In cases where the firm faces liability for that outcome, this need is especially acute. The problem is that, as we’ve been saying, algorithms tend towards uninterpretability. The whole point of software engineering is that a computer performs arithmetic so much faster and more cheaply than a human that it becomes economical to perform massively super-human levels of computation all the time and everywhere. And, as Turing pointed out before computers really existed, it is trivially easy to write an algorithm whose outputs you yourself cannot predict, so the fact that humans wrote the code does not make the working of the algorithm more amenable to human understanding (especially when each individual human will work on only the smallest fraction of a system, and especially, especially where the behavior of the system depends substantially on the presence of external information, whose volume is truly incomprehensible).
Nevertheless, software producers need to be able to understand the workings of their software (note, this is a potential, disperse understanding rather than any realized understanding in any mind) well enough to fix their shit when it breaks. There are three features of software that make this problem somewhat tractable: (a) the final outcome of a computational process is reached by many small steps that build on one another, (b) software can be “architected,” arranged in layers and discrete systems so that the broad shape of a distributed computational process can be made to some extent predictable, and (c) the intermediary steps and interactions between systems can be logged, written to an external record so that the steps may be reconstructed after disaster has struck.
For an outsider, it is hard to fathom the scale of data computer systems produce about their own operation, or to the sophistication of the computer systems that are responsible for collecting, storing, and making effectively searchable this data. To give some indication, I worked for a time at a database service for a hyperscaler. Our largest user was the cloud provider’s logging service. We had a whole team for working on the collection, storage, and querying of the database’s own logs. (Which were, yes, partly stored on the database itself; things get fucky-wucky in the guts of the cloud.)
Nevertheless, even a good logging system does not mean interpretability is solved. And this for several reasons.
You can’t log what is outside the boundary of the system. Negative events (the system produced a bad result because it was missing data because the data entry guy went on vacation) don’t show up on the record, at best they can be inferred. Computer systems generally cross firm boundaries (you store your sales data in Salesforce and your HR stuff in Workday, etc.) and so no firm has the complete logs of a computation. (Though investigation can be federated.)
Logs are necessarily incomplete. Logging everything is impossible. There’s an infinite regress, for one. But just on a practical level, a computation has so many steps that the programmer has to decide which ones are worth logging. (Or they can be bad at their job and fail to insert logs where they really should.) Some information must be kept out of logs for legal reasons. (The GDPR, bureaucracy’s triumph over algocracy.) Sometimes your logging system loses things. To save money on storage you configure your logging to be purged after some period, so the account might be deleted.
Irreducible complexity. In some cases the output of a computation is an aggregate over billions+ of inputs. For the above economic reasons, not all of that input would be captured in the logs. But even if it were, the full story can be so vast that explaining it can be intractable. (This is complex, however. Size itself is not what makes a computation intractable. There may be simple stories of where the data came from and how it is being aggregated that restore interpretability.) An especially relevant example here is neural networks, where the mathematics is intelligible but the data so big and so arcanely put together that we have made very limited progress in interpretability.
With tools to operate a system’s logs, in addition to tools to inspect its configuration, its operational state, and access to its source code, a seasoned software engineer familiar with the system stands a good chance of explaining some aspects of its behavior. This is far from full interpretability and raises the specter “epistocracy”: only a limited elite, selected by virtue of knowledge, will be able to shape the governing scheme. However, I think these observations do provide some comfort. The interpretability of a computing system is not a fixed fact about the system, but rather an aspect that can be invested in.
We may also note that the ability to operate theses systems is now not the exclusive purview of the engineer; coding agents have commoditized this ability. Does this address the epistocratic challenge or just transmute it into a different algocratic (or a corpo-algocratic) nightmare? This is genuinely tricky. One thing I believe, that I perhaps owe an argument for in a future post, is that agents can be a source of testimonial knowledge. So if I ask my agent “why did this program screw the pooch here?” and it searches the logs and reads the code and spits back an answer, I can come to know why the program behaved in a certain way by believing that answer.
And what if the computer system in question is (or has a component) an AI? We have just cited those systems as particularly uninterpretable. And, supposing the AI is an LLM, I don’t think you can get any real insight into why an LLM behaved in a certain way by asking it “why’d ya do that?” LLMs cannot introspect. Their chain of thought is not a chain of thought in a straightforward, behavior-explaining way. At best you get an educated guess based on a general theory of LLM behavior, and at least at this point in time such theories are not good or comprehensive. In fact this is a massive, looming economic problem that the industry as a whole has a very partial story for. (Though not no story. Moderations, evaluations, deterministic workflows, good old-fashioned logging; these get you somewhere.)
Where Do We Stand?
I am less (?) pessimistic about algocracy than Danaher at least. This is not because I think that the problems of hiddenness or opacity can be solved for algorithms. Instead, I think that they are gradated rather than modal features of a governing scheme (a scheme can have a greater or lesser tendency to opacity rather than being opaque or transparent full stop), and further all schemes will fall somewhere or other in the gray area. Even a perfect judge making decisions on the basis of laws passed by a democratic system will be speaking for laws passed for reasons that are now lost, because those who gave them have died, and which in any case no longer includes the reasons of those now living. We are in every moment choked by the hands of the dead. With an algorithm, there is at least some hope of unburying the machinery.