A Guide to Inferencing With Bayesian Network in Python
December 2, 2021
Review: ‘The Book of Why’ Examines the Science of Cause and Effect
Credit…Basic Books
By
Everyone knows that the cock’s crowing at dawn does not “cause” the sun to rise. Conversely, we have equal confidence “that flipping a switch will cause a light to turn on or off and that a sultry summer afternoon will cause sales to go up at the local ice cream parlor.” Such intuitions are integral to countless practical and moral judgments that fill our daily lives. And yet, as Prof. Judea Pearl and the science writer Dana Mackenzie note in their illuminating new work, “The Book of Why: The New Science of Cause and Effect,” scientists and statisticians lacked a common language until recently to distinguish between these very different kinds of observation. Indeed, within academia, “causal vocabulary was virtually prohibited for more than half a century.”
The absence of an accepted scientific approach to analyzing cause and effect is not merely of historical or theoretical interest. It explains the delay in the surgeon general’s reports on smoking, for example, and likely led to untold avoidable early deaths. New modeling tools have vastly expanded what can be learned from the proliferation of “big data” and will define the potential reach of artificial intelligence more broadly.
The subject of causation has preoccupied philosophers at least since Aristotle. Professor Pearl has deftly used the arc of his own career — first at RCA Laboratories and for the last 50 years at the University of California, Los Angeles (initially in the engineering department and since 1970 in computer science) — to chart the recent history of the subject.
This period broadly coincides with what Professor Pearl terms “the causal revolution.” Three ascending rungs of what he calls the “ladder of causation” serve as the central metaphor driving the narrative of “The Book of Why.” The “revolution” charted in the book, and in which Professor Pearl and his disciples played a crucial role, is what has allowed researchers across a vast range of disciplines to move beyond the first rung of the causal ladder, where they had been perennially stuck.
This lowest rung deals simply with observation — basically looking for regularities in past behavior. Professor Pearl places “present-day learning machines squarely on rung one.” While it is true that the explosion of computing power and accessible deep data sets have yielded many surprising and important results, the mechanics still operate “in much the same way that a statistician tries to fit a line to a collection of points.”
“Deep neural networks have added many layers of complexity of the fitted function, but raw data still drives the fitting process,” according to Professor Pearl. The causal revolution is what has enabled researchers to explore the higher rungs of the ladder.
The second rung of the ladder of causation moves from seeing to doing. That is, it goes from asking what happened to asking what would happen based on possible interventions. Professor Pearl notes that “many scientists have been traumatized to learn that none of the methods they learned in statistics is sufficient to articulate, let alone answer, a simple question like ‘What happens if we double the price?’” “The Book of Why” provides a detailed explanation and history of how and when a model alone can answer such questions in the absence of live experiments.
The top rung of the ladder involves something called “counterfactual” questions: What would the world be like if a different path had been taken? These are “the building blocks of moral behavior as well as scientific thought.” The ability to look backward and imagine what could have been governs our judgments on success and failure, right and wrong.
Once considered a defining characteristic of humanity, in recent decades these topics have not escaped the reach of increasingly complex modeling tools. Those tools have been applied to varied social and scientific problems, including the efficacy of medical procedures, the impact of climate change and the usefulness of social policies.
The Professor Pearl who emerges from the pages of “The Book of Why” brims with the joy of discovery and pride in his students and colleagues. He concedes, however, that his inclination is never to use words when an equation will do: “To me, a formula is a baked idea. Words are ideas in the oven.” The resulting book is not always fully accessible to readers who do not share the author’s fondness for equations.
The sometimes challenging technical aspects of the book should not dissuade anyone interested in the promise of intelligent systems. “The Book of Why” not only delivers a valuable lesson on the history of ideas but provides the conceptual tools needed to judge just what big data can and cannot deliver. Notably, “causal questions can never be answered from data alone.”
While it is “easy to understand why some people would see data mining as the finish rather than the first step,” Professor Pearl reminds us that the most important questions will always require “us, as well as future machines,” to engage in the “work of having to consider and articulate substantive assumptions about how the world operates.”
Despite this well-considered skepticism, Professor Pearl is remarkably optimistic about what artificial intelligence can achieve and even whether we can make machines that are capable of distinguishing good and evil. Regardless of whether one agrees with these provocative conclusions, we can all hope that in any counterfactual world in which that is the case, programmers with the humanity of Professor Pearl will be in charge.
Jonathan A. Knee is professor of professional practice at Columbia Business School and a senior adviser at Evercore Partners. His latest book is “Class Clowns: How the Smartest Investors Lost Billions in Education.”
References
- https://www.nytimes.com/2018/06/01/business/dealbook/review-the-book-of-why-examines-the-science-of-cause-and-effect.html
