Trust in Numbers

by Theodore M. Porter

This investigation of the overwhelming appeal of quantification in the modern world discusses the development of cultural meanings of objectivity over two centuries. How are we to account for the current prestige and power of quantitative methods? The usual answer is that quantification is seen as desirable in social and economic investigation as a result of its successes in the study of nature. Theodore Porter is not content with this. Why should the kind of success achieved in the study of stars, molecules, or cells be an attractive model for research on human societies? he asks. And, indeed, how should we understand the pervasiveness of quantification in the sciences of nature? In his view, we should look in the reverse comprehending the attractions of quantification in business, government, and social research will teach us something new about its role in psychology, physics, and medicine.


Drawing on a wide range of examples from the laboratory and from the worlds of accounting, insurance, cost-benefit analysis, and civil engineering, Porter shows that it is "exactly wrong" to interpret the drive for quantitative rigor as inherent somehow in the activity of science except where political and social pressures force compromise. Instead, quantification grows from attempts to develop a strategy of impersonality in response to pressures from outside. Objectivity derives its impetus from cultural contexts, quantification becoming most important where elites are weak, where private negotiation is suspect, and where trust is in short supply.

Genres: Science, History, Nonfiction, Philosophy, History Of Science, Economics, Social Science

328 pages, Paperback

First published February 17, 1995

Reviews

[Alex · Rating 4 out of 5]

'trust in numbers' asks why administrative and scientific organizations quantify, and begins by claiming that the standard story, namely that quantification and formal models are inferentially more powerful than verbal theories, is insufficient. porter does not claim that quantification doesn't work, and instead sidesteps this discussion entirely to flesh out the administrative side of the story. the first several chapters essentially discuss measurement: administrative difficulties due to lack of information, and lack of information due to lack of standardized measurement. for this portion of the book 'statistics' essentially means 'counts of things,' and porter discusses extensive pushback in accounting and medical communities against counting, in favor of richer, more nuanced understand dependent on expert judgement.

the claim is then that national administrations became more powerful, and this increased power and bureaucratic scope allowed nation-wide measurement and standardization, which porter describes as social technologies. throughout this argument porter hints that these social technologies are shaping reality in addition to measuring reality -- i would have like to see connections drawn between his argument the performativity hypothesis, for example.

after this we arrive at what seems to me to be the core of the argument: that quantification is not an epistemic tool, but rather a social one. quantification is supposed to produce objective information, and objective information is necessary for policy decisions, or really any decision where interested parties will try to steer towards a given outcome. this claim is supported by a comparative study of doctors, accountants and engineers in europe (mostly britain and france) and the united states -- in europe, the experts are largely insulated from powerful political adversaries, and institutions that house experts have social and political status that allows them to operate autonomously and without much oversight. in these settings, experts quantify, but do so very informally, and do not share their results publicly; they are trusted to be objective already. in the united states, experts do not have this political cover, and they deploy an extreme form of quantification to certify their disinterestedness. in a related study of academics, porter shows similar effects of disciplinary disunity or newness, and the consequences of this institutional frailty.

the argument is compelling, and i would love to see it applied to more modern phenomena: the decline of universities and expertise in general, and more particularly the replication crisis and modern pushes for reproducibility. for example, the book suggests that our current focus on reproducibility is not so much about improving scientific practice (where, for example, is the theory work on how to conduct science correctly that would accompany such an effort?) as it is about certifying objectivity by making scientific results reproducible by literally everyone. i would like to see this idea taken further.

i also find porter's len an amusing framework to investigate the replication crisis. porter discusses the rote use of statistic tests (against all statistical advice!) in psychology as a tool to grant credibility to psychological experimenters in the face of disciplinary disunity (i am reminded of brian caffo and roger peng's "p-values are just the tip of the iceberg" paper). anyway, we see the same thing happening now: "solutions" to the replication crisis like pre-registration, which do not actually accomplish good inference, merely transparency. there are hints here of a futile feedback cycle: psychologists adopting mechanical procedures for inference to bolster their credibility, mechanical procedures for inference being impossible, such that the resulting inferences are bad, and this negatively affecting the credibility of psychology.

all in all, fairly dry writing, but full of interesting information. absolutely essential in that it considers how inference practices are responses to social pressures, rather than choices based on epistemic virtue. more data analysis needs to consider the goals, influence, and power of parties interested in data analytic outcomes, and this is a promising start. i look forward to work that can discuss this sociological aspect of data analysis, as well as technical and epistemic matters simultaneously

[Rodrigo Medel · Rating 4 out of 5]

This is a classic book in history of science. By comparing the role of numbers and the acceptance of objectivity criteria in France, England and USA, the main thesis of the book is that, contrary to what is commonly thought, quantification in the different spheres of society did not arise from the science of nature and then passed to social activities. Rather, the arrow seems to be exactly the reverse. Accountability and control of expert and elitist groups by society seemed to have played a critical role, with statistics and model building being only secondarily accepted as a regular practice by scientists.

[Chunyang Ding · Rating 3 out of 5]

A very thorough but dry book. Porter does a very good job laying out his argument, that the usage of numbers in several disciplines (economics, sciences, accounting, engineering) originates from the growth of a field, and the inability for gatekeepers to otherwise trust the participants of that discipline. However, during the transitionary process, numbers are often used as weapons to demonstrate objectivity, even if the process in which those numbers are created (ie, the methods for accounting for unquantifiable properties, the instruments by which data is collected and created, the statistical methods used for analyzing that data) is tightly controlled and bent to whatever that field already considers to be their "Truth". More than anything, Porter's book seemed to me to be a warning against implicitly trusting numbers, or people trying to say "Just the facts", because there is *always* some kind of social implication to it.

However, the presentation of this research is incredibly dull. The entire book is presented very academically, where minor details are as important as the primary theses of the manuscript. The portion on French engineering was especially difficult for me to follow, and even the section on the American Corps of Engineers, a topic that I have found fascinating since reading Desert Cadillac, was made to be grueling with the sheer quantity of details included. Porter makes an excellent resource for those studying philosophy and history of science, but this isn't a great book for the casual reader.

[Judd Street · Rating 4 out of 5]

Apparently when I read this book in 2016, I gave it five stars with the pithy review below. None of that is wrong - this is a classic text and it does a nice job of making some points regarding standardization - but with an additional seven years of reading STS texts, I'm not sure this is actually the best encapsulation of these arguments. Or, at the very least, I found it a real slog to get through a second time when re-reading it for a course, and am not sure it's the most accessible or approachable articulation of these ideas.

A core project of the book, of course, is to justify why it is that decision-makers are so compelled to work in quantitative worlds. As he argues in the introduction,

The appeal of numbers is especially compelling to bureaucratic officials who lack the mandate of a popular election, or a divine right. Arbitrariness and bias are the most usual grounds upon which such officials are criticized. A decision made by the numbers (or by explicit rules of some other sort) has at least the appearance of being fair and impersonal. (p. 8)

I'm not persuaded by Porter's view that this is a trap for unelected officials in particular, given that politicians seem equally likely to use rhetorics of impartiality and quantitative reasoning to garner support. But, the point holds in general. These quantitative narratives also have a "compatibility [with] positivism with the pursuit of control over nature" (p. 19) which fits with broader societal logics.

Quantitative logics aren't all rhetorical, though. For instance, Porter refers to J. S. Hunter's work on the US National Bureau of Standards, pointing out the ways that mandates (in theory) constrain producers from choosing the most favourable ways of self-quantifying (p.28). They also only hold water because people accept them, such as the way that publics accept school grades or accounting practices as standard and real (p. 45). And, they significantly shape professional practice, as discussed later (p. 116).

This can be particularly problematic when there's disagreement on how to count. Porter explores this with a great example from medicine and ways that normalization needs to be based on disease rather than bed days, lest results be misleading (p. 83) - very a propos in our COVID world.

Porter perhaps deviates some from Scott, who makes similar arguments, because of Porter's focus on knowledge production (versus social control). "But the bureaucratic imposition of uniform standards and measures has been indispensable for the metamorphosis of local skills into generally valid scientific knowledge," argues Porter (p. 21). This is linked, of course, with the desire to achieve objectivity, something quantification promises to deliver on (p. 74)

Porter also links this, in some ways that vaguely rhyme with Collins & Evans, to the nature of expertise. Many of his examples illustrate ways that quantification is an antithesis of expertise, the latter relying on discernment versus "mechanical objectivity" (p. 91). Standardization also frequently seeks to erode individual discretion, for better or for worse, which is often opposed by those like actuaries (p. 111); "In place of precision they offered a profession," Scott recounts (p. 113). Some professions (actuaries) were more vulnerable than others (engineers) to this professional erosion because of social and political power (see p. 138, 142). Put more directly,

Quantification was never merely a set of tools. Making up numbers in deference to political necessity was unacceptable to these engineers. It compromised their status as a disinterested elite and violated standards of mathematical integrity that they took seriously. (p. 118)

Finally, Porter's exploration of the rise of cost benefit analysis as championed by professionals to increase social standing and in the rise of increasing social distrust (p. 189) is also fascinating (Chapter 7), and features a few great quotes (e.g., "Hammon was aware, though, that Adam and Eve felt temptation even before the economic serpent presented them with this apple," p. 187). It also aligns with some remarks on the rising of statistical analysis due to distrust in medicine as well (p. 208-209).

All told, yes this is a good book... but, damn, it's been a while since I've struggled so much to get through a text due to lack of engagement. Adjusting previous 5 star rating to an average of 5 + 3 as a result.

--Dec 24, 2016 Review--
Trust in Numbers joins Scott's "Seeing Like A State" as quintessential texts of STS. Numbering, ordering, and standardizing the world is ultimately part of making it, and Porter does an excellent job of touring various disciplines and histories in order to illustrate these points.

[Amy · Rating 4 out of 5]

I skipped over some of the sections on quantification in engineering, but otherwise, this was the book I had been looking for for a while.

There were a lot of good points: how quantification a tool that comes in when trust in a community is eroded, how data can be used as a tool of accountability (but always a bit antagonistically), how social nature and socialization of the science community give it ~neutral authority, how the professions that rely on statistics the most are the ones that otherwise have little credibility (psychology, economics), how professionals will resist quantification and standardization bc of a belief they "expert subject knowledge" (and how more prestigious professions have been able to resist it better, e.g. medicine v. education.)

Niche history, but good.

[Tin Alvarez · Rating 4 out of 5]

I removed a star for the tedious prose in dense chapters. Otherwise, I would've rated this a five if the writing were as consistently lucid and lively as the prefaces, because it is a wonderful historical treatise tracing the prestige of quantification in governing
knowledge and social life.

[Lorin Hochstein · Rating 5 out of 5]

There are two general approaches to decision-making. One way is to make a judgment call. Informally, you could call this “trusting your gut”. Formally, you could describe this as a subjective, implicit process. The other way is to use an explicit approach that relies on objective, quantitative data, for example, doing a return-on-investment (ROI) calculation on a proposed project to decide whether to undertake the project. We use the term rigorous to describe these type of approaches, and we generally regard them as superior.

Here, Porter argues that quantitative, rigorous decision-making in a field is not a sign of its maturity, but rather its political weakness. In fields where technical professionals enjoy a significant amount of trust, these professionals do decision-making using personal judgment. While professionals will use quantitative data as input, their decisions are ultimately based on their own subjective impressions. (For example, see Julie Gainsburg’s notion of skeptical reverence in The Mathematical Disposition of Structural Engineers). In Porter’s account, we witnessed an increase of rigorous decision-making approaches in the twentieth century because of a lack of trust in certain professional fields, not because the quantitative approaches yielded better results.

It’s only in fields where the public does not grant deference to professionals that they are compelled to use explicit, objective processes to make the decisions. They are forced to show their work in a public way because they aren’t trusted. In some cases, a weak field adopts rigor to strengthen itself in the eyes of the public, such as experimental psychology’s adoption of experimental rigor (in particular, ESP research). Most of the case studies in the book come from areas where a field was compelled to adopt objective approaches because there was explicit political pressure and the field did not have sufficient power to resist.

In some cases, professionals did have the political clout to push back. An early chapter of the book discusses a problem that the British parliament wrestled with in the late nineteenth century: unreliable insurance companies that would happily collect premiums but then would eventually fail and would hence be unable to pay out when their customers submitted claims. A parliamentary committee formed and heard testimony from actuaries about how the government could determine whether an insurance company was sound. The experienced actuaries from reputable companies argued that it was not possible to define an objective procedure for assessing the a company. They insisted that “precision is not attainable through actuarial methods. A sound company depends on judgment and discretion.” They were concerned that a mechanical, rule-based approach wouldn’t work:

> Uniform rules of calculation, imposed by the state, might yield “uniform errors.” Charles Ansell, testifying before another select committee a decade earlier, argued similarly, then expressed his fear that the office of government actuary would fall to “some gentlemen of high mathematical talents, recently removed from one of our Universities, but without any experience whatever, though of great mathematical reputation.” This “would not qualify him in any way whatever for expressing a sound opinion on a practical point like that of the premiums in a life assurance.” —pp108-109

Porter tells a similar story about American accountants. To stave off having standardized rules imposed on them, the American Institute of Accountants defined standards for its members, but these were controversial. One accountant, Walter Wilcox, argued in 1941 that “Cost is not a simple fact, but is a very elusive concept… Like other aspects of accounting, costs give a false impression of accuracy.” Similarly, when it came to government-funded projects, the political pressure was simply too strong to defer to government civil engineers, such as the French civil engineers who had to help decide which rail projects should be funded, or the U.S. Army Corps of Engineers who had to help make similar decisions about waterway projects such as dams and reservoirs. In the U.S., they settled on a cost-benefit analysis process, where the return on investment had to exceed 1.0 in order to justify a project. But, unsurprisingly, there were conflicts over how benefits were quantified, as well as over how to classify costs. While the output may have been a number, and the process was ostensibly objective, because it needed to be, ultimately these numbers were negotiable and assessments changed as a function of political factors.

In education, teachers were opposed to standardized testing, but did not have the power to overcome it. On the other hands, doctors were able to retain the use of their personal judgment for diagnosing patients. However, the regulators had sufficient power that they were able to enforce the use of objective measures for evaluating drugs, and hence were able to oversee some aspect of medical practice.

This tug of war between rigorous, mechanical objectivity and élite professional autonomy continues to this day. Professionals say “This requires private knowledge; trust us”. Sometimes, the public says “We don’t trust you anymore. Make the knowledge public!”, and the professionals have no choice but to relent. On the subject of whether we are actually better off when we trade away judgment for rigor, Porter is skeptical. I agree.

[Alex · Rating 3 out of 5]

This was a good book, but it wasn't what I was looking for, so take my rating with a sizable kernel of salt. I was hoping that it would contain more discussion of standardized testing and teacher accountability, but I only got about three paragraphs of that toward the very end. Instead, this was largely a history of the bureaucratization of the Ecole Polytechnique and the U.S. Army Corps of Engineers. Porter's critique of objectivity is definitely worth the read, though you may find similar critiques in more concentrated form elsewhere.

[Polly Callahan]

from New Yorker article https://www.newyorker.com/magazine/20...
which says that “How Data Happened: A History from the Age of Reason to the Age of Algorithms” (Norton), the Columbia professors Chris Wiggins and Matthew L. Jones ....initial chapters, drawing on earlier work like Theodore Porter’s “Trust in Numbers,” Sarah Igo’s “The Averaged American,” and Khalil Gibran Muhammad’s “The Condemnation of Blackness,”

[Heather Hoyt · Rating 4 out of 5]

I read portions of this for a class and then finished the rest on my own, though I confess that by the end of it, I didn't care very much. I started this book by thinking something like, "Wow, I'm trusting in something that is much more arbitrary than I thought." Numbers can be subjective, just like judgment is. For a moment, I felt like I was looking at the wizard behind the curtain, and but then I find myself not caring about it very deeply and I let the curtain slide back. I trust in numbers, and that seems fine.

[Romy · Rating 2 out of 5]

I had to read this for my college seminar, and while it was incredibly dry, it was very thorough and the concept is an interesting one to think about.

Can objectivity ever really exist?

This book looks at this question and many more like it from a scientific standpoint, and I think I would have enjoyed it infinitely more if it were from a philosophical or humanities standpoint.

I won't give it anything less than 3 stars even though I honestly hated reading it, because I recognize it's better suited for a scholar than an 18 year old, and that doesn't make it a bad book, I just didn't enjoy it.

Edit: I take back what I said, my mind keeps randomly wandering to how much I disliked this book so I'm docking a star :)

[Yisu Z. · Rating 4 out of 5]

观点非常棒；写作风格不太习惯（不断推进到新概念，但又不收回来——。实证的部分也感觉一般般，比较浅、散。其中任何一个实证章节都能单独写一本书。非常帮助开脑洞的一本书。

[Travis · Rating 1 out of 5]

philosophy of science

[Edward Fenner · Rating 3 out of 5]

Interesting but not exactly a page-turner.

[Melissa]

The book's structure is a little problematic.