In August 2016, the Verrückt water slide had been opened for 2 years before it decapitated a child. 50 meters high and shaped like a ski jump, the ride accelerated rafts up to 70 MPH. At the jump, Caleb Schwab’s raft went airborne and struck the netting’s support beam. Far from being a freak accident, however, a later investigation discovered that a dozen other injuries had been inflicted during its operation and that the design process of the slide had consisted in part of sending rafts filled with sandbags down and adjusting the angles until they stopped flying off the track.
When I taught engineering ethics, I introduced the class with this example as a piece of clickbait. There are however a couple of serious points to be made. The first is that while the construction of the slide was morally negligent, it is unclear if it was criminally negligent. The trial against the park owner and designer ended inconclusively; laws at the time permitted self-certification. (Legislation was tightened in response to this incident.) The second is to think through the value of safety. This is a clear case of way-too-little regard being paid to safety. But risk is something that comes by degrees and so the only perfectly safe ride is no ride at all.
An engineering project will have a massive web of morally relevant implicatons. Presumably a project is considered because it offers some clear benefit. The roller coaster is good fun. The apartment block will allow more people to live in a desirable area. The train line will unlock economic development between the connected areas. But there will also be costs. The financial costs of constructions, yes, but also environmental harms, degraded quality of life for some (the train will be loud and interrupt some residents’ sleep), risks injury or death for others. Some projects ought not to be undertaken while others should. What’s the difference?
One class of accounts recommends “cost-benefit analyses.” You assign some (dollar) value to every likely foreseen outcome of a decision. This might look like, e.g., figuring out how many people’s sleep will be impacted by the train’s passage, assigning some value to an hour of lost sleep ($20? $50?) and multiplying by number of expected hours. Or saying “there’s a 1% chance this ride will kill someone over its lifespan, say a life is worth a million bucks, call that 0.01 deaths so ten grand down the hole.” Once you’ve totted up all the figures, you look at if it comes out positive, in which case, hey, it’s a good thing! At least if it compares favorably to other options, you should go ahead and do it.
Lately on my Substack feed there’s been some hubbub about a Dario Amodei story. Back in his grad student days, Amodei wrote an article for GiveWell (a meta-charity aimed at determining how efficient other charities are at doing good) describing his decision to donate (on a 2010 grad student’s salary!) $ 10,000 dollars to the childhood vaccination charity Village Reach. He gives Stop TB as the alternative he seriously considered, which provides tubercolisis care for children and adults. Now that Dario Amodei is no longer a Princeton graduate student, but instead the head of an AI lab about to IPO for over a trillion dollars, his critics have dug up that article and taken issue with the following passage.
I feel that adults are capable of deeper and more meaningful experiences than are infants, and also deeper connections with other people, so an adult death seems worse to me than an infant death (though both are of course bad). Trying to quantify exactly how much worse is very subjective and can also seem calculating (“how many babies would you kill to save an adult?”), but on a practical level one is forced to make difficult decisions with limited funds, and in my case I’d say that I think an adult death is perhaps 2 or 3 times worse than an infant’s death.
Critic responses that I’ve seen range from
Infants are obviously more morally valuable than adults and only a monster would think otherwise.
Not only does this seem calculating, it is calculating. There’s something monstrous or at least confused in assigning relative values to different lives and trying to make a moral decision based on cost effectiveness.
Now, obviously, the reason that this has reemerged 16 years after being written is that some people hate AI in general and Dario Amodei in particular; I’m not interested in that here. Rather the question on my mind here is (2), when (if ever) is “moral arithmetic” appropriate?
My engineering ethics students tended to be split between a group who was instinctively comfortable with cost-benefit analysis, for promising a scientific-seeming way to settle moral questions, and those who felt there was something cold and evil about it. To try and echo some instinctive objections.
Human life is infinitely valuable. Assigning a finite value to human life is to say it could be okay to kill someone if there were enough benefits to doing so, but this is categorically wrong.
To attempt to assign concrete values to outcomes is morally hubristic and even imperious. It’s to declare yourself the sole arbiter of what’s good.
Cost-benefit analysis is inherently consequentialist, but consequentialism is false. Some things are wrong even if the benefits outweigh the costs.
Cost-benefit analysis is unrealistic or too limiting a framework given (a) our state of uncertainty about what the consequences of any decision will be and/or (b) the uncertainty we should have over our values.
Cost-benefit analysis is an alienating way to make decisions. Instead of responding directly to our reasons or being guided by our moral sentiments or exhibiting virtues or reflecting on the moral law as such (depending on your preferred account of what good moral decisionmaking looks like), it asks us to behave as actuaries. Deciding to take a course of action because cost-benefit analyst selects it is “one thought too many.”
There is a lot here that’s rich and potentially compelling, but I think these debates can also easily become confused. Some preliminary points to try and hedge off some unproductive avenues.
It is entirely possible to believe that cost-benefit analysis is the right tool for only some jobs. You definitely shouldn’t whip out the calculator when deciding whether to drive a friend to the airport. (You should do it because you care about your friend.)
You can think that cost benefit analysis is the right tool for a decision without thinking that it should be left up to any one individual to arbitrate how it’s used. Perhaps a democratic process should feed into how we do the analysis. Perhaps the analysis should feed into a downstream process.
The procedure itself does not tell you what or how you should value. Even using the dollar as the unit is mere convention. You can even put in “non-utilitarian” consequences. If you think lying is wrong in itself (but not overwhelmingly wrong, which is implausible), just include “involves lying” as one of the costs.
It is possible, though challenging, to use cost-benefit analysis in a way that more finely reflects uncertainty. You can do it multiple times for different likely outcomes or value judgments and weight based on probability. You can risk adjust if that floats your boat. (I think risk adjustment is lame, but that’s a separate post.)
The “this is bad decision making” is what the rest of this post is about.
Here’s where we get to the positive, albeit cautious defense of cost-benefit analysis or “formal calculation” in moral decision making. Here’s the argument.
Some decisions, depending on the stakes and the complexities involved, demand that we put more care into making a good decision. (Assuming we have time to make a decision; if it’s split second, you go with your instincts.)
If you need to put care into making a good decision, then you should apply the best technique you can to help ensure that your time spent investigating actually improves the quality of your decision making.
Cost-benefit analysis is part of the best technique we have available for doing so.
Therefore, cost-benefit analysis should sometimes be used.
To see where this argument is getting at, let’s consider a range of techniques for decision making.
Going with your gut. Don’t use any explicit thinking in making a distinction. Trust whatever combination of instincts, emotions, and dispositions are coursing through you at the moment and go with them.
Make a disorganized list of things you could do and an unranked list of considerations, then trust your gut.
The above, except you try and assign some sort of priority to various considerations. You still go with your gut at the end because this is not an algorithm.
The above, except you assign numerical weights to the considerations. You can treat this as an algorithm (add up the considerations for each output), but it’s really still a vibe.
The above except you attempt to be systematic and (to a point) exhaustive in your considerations. So if a consideration applies to one possible choice, you think about to what extent it also applies to other choices.
I think 5 is basically cost-benefit analysis. To be clear, I’m thinking of cost-benefit analysis as a discipline but not a procedure. A recipe is a procedure. Mise en place is a discipline. The main reason I think it’s a good discipline is that it prevents the practitioner from doing “fake thinking.”
To see what I mean by this, let’s consider the claim “writing is a technology for thought.” This is sometimes put in a sloganistic form “you haven’t really thought until you’ve written your thinking down.” Why should this be the case? Well, when your thinking stays in your head, it’s very easy for it to be circular or messy or confused. Considerations may occur to you that point towards or against your favored conclusion, but you can only hold so many of them in your mind at once, you can easily forget prior considerations. You can also easily nod along to yourself, “yeah, that sounds good,” without really interrogating what you’re thinking. At the least, once a consideration is on the page it has a more fixed form. It can be related to other considerations, which are also on the page, it can be interrogated further.
Now, not all writing is equally good as an aid to thinking. It’s easy for “stream of consciousness” writing to reproduce many of the demerits of unaided thinking. An objection appears in the text, it is noted and then cleanly forgotten. Entire realms of consideration are skipped. There are further techniques (e.g. argument mapping, structured writing) that should be applied to determine a discipline of writing that actually supports clear thinking. (The irony of making these statements in a “no effort” post is not lost on me.)
Decision making is just one form of thinking. It is no surprise then that it is benefited by writing. But the numerical form of the calculation enforces a further discipline: that one is honest and clear about one’s own values.
This is a tricky point it’s worth being somewhat precise about. I don’t think it’s rationally possible with our limited cognitive capacity to have rationally defensible and fully settled views about the relative priority of our values in different circumstances. But I also don’t think we’re starting from zero. Rather, we have some inchoate sense that say “humans are more important than bunnies are more important than shrimp.” After some period of reflective equilibrium, those inchoate priorities can become at least ballpark estimates. Similarly, quantifying outcomes (or a rough probability distribution over outcomes) requires us to be honest about what we believe, rather than handwaving.
Cost-benefit analysis is an element in a broader process of thinking and deciding. The “costs” and “benefits” have to come from our thinking about the world and our value; there are other techniques for thinking through those, but all techniques form an overall assemblage. If this analysis yields outcomes that seem wrong, then that can help us diagnose where we need to look further. And of course these disciplines may be practiced in more or less complete a form; often it’s a simple form of “napkin math” and often that suffices.
To return briefly to Amodei. The section of the article critics focus on is the part that looks the most straightforward cost-benefit analysis: assigning a conversion factor between different values and doing a basic “which one gives me the more bang for my buck” calculation. But this is not the whole of the article and the charity that comes out ahead in that selection loses overall. What caught my eye and where I want to end is with where Amodei ends.
I hope (though I cannot be sure) that my donation will save the lives of 20 children (which is what the cost-effectiveness numbers work out to). That’s a truly staggering benefit, and honestly it came at very little cost to myself: I don’t much miss the new car I didn’t buy, and I’ll gladly make the same sacrifice next year in order to donate again. What did feel very emotionally taxing was reading (and in most cases, agreeing with) all the negative analysis of charities at GiveWell and elsewhere. I found it difficult to evaluate everything in a critical fashion while still holding on to the compassion and optimism that originally inspired me to donate. It’s tough to find the right balance between caring and hard-nosed realism, but it is possible, and it is, as far as I know, the only way to truly change the world.
It is a common enough occurrence in, for example, caring for a sick parent, to find it difficult to hold on to your motivation to care while bearing through all of the strain, the sleepless nights of caring. Bearing through that and having patience requires the application of psychological techniques (distracting oneself, meditating on fond memories, etc.). The caregiver can frequently give care while not feeling a present motivation or a present love. Technique can empower our moral reasons by letting us hold true to them even where they cannot guide us themselves.
Given that we have moral reasons to make significant sacrifices to “do the most good” (which is true, but I won’t argue it here), we should expect that trying our best to achieve this should be taxing. The world is enormous and enormously complicated and so it is terribly taxing to figure out how our decisions may effect it. This makes it all the more urgent that we apply discipline to the problem.