Abstract

This note was originally written for The Book of Why: The new science of cause and effect (Pearl and Mackenzie, 2018) and was taken out to meet space limitations. I am archiving these thoughts with the hope that they will prove useful for other students of history.

Thomas Kuhn’s classic book The Structure of Scientific Revolutions, published in 1962, was one of the most unlikely bestsellers in history. It sold only 919 copies in its first year, yet by its golden anniversary it had reached more than 1.4 million. Structure introduced a new idiom into our language—“paradigm shift.” Before Kuhn, “paradigm” was a philosophical term that referred to rhetorical devices such as parables and fables. Now everyone from feminists to New Age spiritual healers calls for paradigm shifts, and there is even a “Paradigm Shift” quilt pattern!

Kuhn sought in his Structure to debunk the myth that science progresses steadily from ignorance to enlightenment, through a gradual accumulation of facts and logical deduction. This is the way science is typically presented in our classrooms, but Kuhn argued that the image does not square with reality. Most of the time, he concedes, researchers in a given field do strive to make incremental progress on problems that are agreed to be important, using agreed-upon languages, techniques and assumptions. These tacit agreements constitute the “paradigm” of a given science. But every science, from physics to chemistry to biology, has gone through periods of profound trauma—Kuhn’s “paradigm shifts”—when business as usual just didn’t work. Usually, Kuhn argues, the crisis is precipitated by an “anomaly,” an observation or event that the ruling paradigm cannot account for. Eventually a new paradigm makes sense out of the anomaly, and the science settles back down into its “normal” mode. But the reasons for acceptance of the new paradigm are varied and often have little to do with scientific objectivity. “The claim to have solved the crisis-provoking problems is, however, rarely sufficient by itself,” Kuhn emphasizes. For instance, he notes, “Copernicus’ [heliocentric] theory was not more accurate than Ptolemy’s.” True, sometimes the new theory makes successful predictions that could not have been anticipated. For example, Copernican astronomy predicted that Venus would show phases like the moon’s—which was subsequently confirmed when the first telescopes were invented. But other times, the decisive factors may be individual beliefs or personal idiosyncrasies; national or cultural allegiances; and even a sense of beauty, the epitome of subjectivity. “Something must make at least a few scientists feel that the new proposal is on the right track, and sometimes it is only personal and inarticulate aesthetic considerations that can do that,” Kuhn wrote.

In 2016, I attended a seminar in the history department at UCLA dedicated to the book’s fiftieth anniversary. It was organized by professors Ted Porter and Norton Wise, who were both students of Kuhn at Princeton University. The question naturally came up: What made Structure such a success, even among people with no particular interest in science or philosophy? Wise argued that the cultural mood of the period was ready for Kuhn’s book. The opposition to the Vietnam War, the Woodstock counter-culture, the general mood of melting paradigms and smashing authority figures, could not have been more perfect for a book that took on one of the most revered idols of all: the objectivity of science itself.

Kuhn’s book humanized science and perhaps took it off a lofty pedestal where it didn’t belong in the first place. But it also created its own kind of paradigm shift among historians of science. Suddenly history written from the viewpoint of a present-day scientist became unfashionable. “Whig history” was the epithet used to ridicule history written with hindsight, which focuses on the successful theories and experiments while ignoring failed theories and dead ends. Instead a new democratic style of history writing came into fashion, which treats chemists and alchemists with equal respect, viewing the latter merely as being temporarily out of favor.

Not all historians went along with this romantic style of history telling, and certainly not the scientists. By the late 1980s, some of them began fighting back. Edward Harrison, an astronomer and cosmologist at the University of Massachusetts, joked (but with serious intent) that historians had replaced the Whig interpretation with the “prig interpretation,” which “from fear of being unhistorical commits what it supposes to be the lesser crime of being unscientific.” Even historians have (cautiously) rehabilitated the Whig approach. In 2012, Edward Cronon (then president of the American Historical Association), echoed Harrison’s sentiments as he wrote: “Historians exist to explain the past to the present.” To accomplish that task, he conceded, they should use any tools they have available, even to the point of using contemporary values and standards to interpret the past.

This historiographic debate between the Whiggish and the priggish came to mind as I was preparing to write about one of the most bizarre paradigms of twentieth-century science: statistics and its attitude toward causality. Like all of Kuhn’s paradigms, statistics restricted the questions that its acolytes could ask and made Causality one of the proscribed questions, maybe even the Ur-question whose name could not be spoken. As a foot soldier in the causal revolution of the 21st century, I find it both curious and compelling to ask: How did this paradigm take shape?

Unlike many of the paradigms that Kuhn wrote about, the silencing of causality was not driven by observed anomalies that previous theory could not explain. Rather, it was almost entirely man-made, stumbled upon and then institutionalized by two great giants, Francis Galton and Karl Pearson. Moreover, the statistical paradigm that Galton and Pearson forged in the 1880s and 1890s, instead of resolving the causal problems that motivated its creation, declared them off limits, and went on to become a thriving enterprise that escaped their existence.

As I tell this story I will take an unabashedly Whiggish view, painted entirely by what we know today about the Causal Revolution and its insights. I am well aware of the risks involved. Given the ongoing controversies within statistics, perhaps it would be more prudent to forget what we know, to take the priggish route and marvel, as other historians of statistics have done, at the intellectual accomplishments of the Galton-Pearson school: correlations, chi-square tests, p-values, multivariate analysis, and other tools that have made data science into a science. But I would be remiss in my duty as a student of history and my integrity as a scientist if I did not leverage the modern understanding of causality to “explain the past to the present.” There simply is no other way to understand this peculiar past. Therefore I will start the story, not with Galton and Pearson, but with the person who laid the foundation for modern causal analysis, Sewall Wright.