(Baker, 2016)
You can also download and read or share a .pdf of the complete text of this essay by scrolling down to the bottom of this post and clicking on the Download tab.
Empirical Science With Uncertainty But Without Reproducibility
In their important essay, “From Scientific Reproducibility To Epistemic Humility,” Joseph Wayne Smith and Saxon J. Smith open and close their argument as follows:
The reproducibility crisis (aka “the replication crisis,” aka “the replicability crisis”) is a disturbing contemporary theoretical problem whereby, in a large and increasing number of scientific fields, ranging from social sciences such as psychology, sociology, and economics, to natural sciences such as genetics, ecology, and medicine, it has been found that numerous studies have not been able to be reproduced by other researchers and sometimes even by the original research teams that retested their own research (Baker, 2016). The philosophical problem raised here is that the so-called “scientific method,” among other things, requires for the empirical sciences that their research be able to be reproduced, hence a failure to be able to do this on a large-scale calls into question the scientificity of the disciplines suffering from this failure of reproducibility….
Breznau et al.’s conclusion is that
idiosyncratic uncertainty is a fundamental feature of the scientific process that is not easily explained by typically observed researcher characteristics or analytical decisions. (Breznau et al., 2023)
Elaborating on this, they make the cogent observation that a much higher level of uncertainty about research findings exists than was previously thought, and that there is a general need for “epistemic humility.” Their conclusion was drawn from a range of empirical studies that used statistical methods and assumptions such as the relevance of the categories of “significant’ versus “not significant” data, with a further focus upon variation in significance, which has also been observed to be problematic in debates about the replication crisis and the foundations of statistical methodology (Mathur et al., 2023).
As we’re construing it, the attitude of epistemic humility towards empirical science is not an all-out or destructive skepticism about empirical science, but instead a measured or constructive skepticism that yields a critical awareness of the proper limits and scope of empirical science. Certainly, the reproducibility crisis calls for epistemic humility towards empirical science. (Smith and Smith, 2023)
In thinking more about Smith-&-Smith’s essay, I started out from Breznau et al.’s thought-provoking observation:
[I]diosyncratic uncertainty is a fundamental feature of the scientific process that is not easily explained by typically observed researcher characteristics or analytical decisions. (Breznau et al., 2023)
This led me to wonder whether there might be an important analogy between the role of uncertainty in quantum mechanics, on the one hand, and the role of “idiosyncratic uncertainty” in the irreproducibility results for empirical science, on the other.
My leading thought was that just as Werner Heisenberg’s famous uncertainty result isn’t the expression of a skeptical problem at the heart of classical mechanics, but instead the discovery that particle physics actually operates according to a non-classical mechanics, so too the irreproducibility results aren’t the expression of a skeptical problem at the heart of classical empirical scientific methodology, but instead the discovery that empirical science actually operates according to a non-classical methodology. Heisenberg’s Uncertainty Principle for particle physics says that the more precisely you measure a particle’s momentum, the less you’re able to determine its position from initial conditions, and conversely (Heisenberg, 1927/1983). Correspondingly, that led me to what I’ll call Hanna’s Uncertainty Principle for empirical science: the more precisely you measure an empirical scientific study’s original set-up conditions, the less you’re able to reproduce its original results, and conversely. For the purposes of argument, let’s assume that both uncertainty principles are true. Interestingly, if Hanna’s uncertainty principle is true, then it also applies to (Heisenberg, 1927/1983).
Heisenberg-uncertainty stems from the intrinsic or real complementarity of the momentum/position pair, considered as “canonically conjugate variables,” independently of any observer effects there might also be, which led me to postulate that there’s an intrinsic or real complementarity between any empirical study’s original set-up conditions/results pair, considered as canonically conjugate variables, independently of any researcher effects there might also be. More specifically, Heisenberg-uncertainty flows from the fact that epistemically isolating momentum from position, or epistemically isolating position from momentum, misrepresents their complementarity: every particle with mass in fact has both momentum[i] and position, but by their very nature as canonically conjugate variables they co-determine each other, hence an isolationist epistemology will always misrepresent the real microphysical phenomenon. Correspondingly, Hanna-uncertainty flows from the fact that epistemically isolating an empirical scientific study’s original set-up conditions from its results, or epistemically isolating its results from its original set-up conditions, misrepresents their complementarity: every empirical scientific study in fact has both original set-up conditions and results, but by their very nature as canonically conjugate variables they co-determine each other, hence an isolationist epistemology will always misrepresent the real methodological phenomenon.
If correct, this in turn means that every empirical scientific study is one-off or unique, and also that since every empirical scientific study attempts to describe some real relational fact (whether causal, essential, or statistical) in nature, then either this one-off or unique empirical scientific study either truly or veridically describes some real relational fact in nature, or else it doesn’t. But then it follows that reproducibility isn’t either a necessary or a sufficient condition of the truth or veridicality of any empirical scientific study. On the contrary, irreproducibility is a basic fact about all empirical scientific studies, whether true or veridical, or not.
Correspondingly, the so-called reproducibility of empirical scientific studies is generated merely as a derivative or secondary methodological effect when there’s a similar imprecision in measuring both the original set-up conditions and the original results of an empirical scientific study. But this doesn’t show anything about the original empirical scientific study, other than that its various reproductions are more-or-less like the original empirical scientific study. Whether the original empirical scientific study is indeed true or veridical, or not, then its various reproductions will be also more-or-less true or veridical, or not. Nevertheless, its various reproductions don’t logically or semantically conduce or contribute to the original empirical scientific study’s being true or veridical, or not, just as the various photocopies of an original text or image don’t logically or semantically conduce or contribute to that text’s or image’s being true or veridical, or not.
Therefore, if my argument in this essay is sound, then while there’s still sufficient reason for epistemic humility about empirical science and its methodology—namely, the fact that all scientific researchers are “human, all-too-human,” i.e., finite, fallible, and thoroughly normatively imperfect in every other way too—it also turns out, by virtue of Hanna’s Uncertainty Principle, that the very idea of reproducibility was fundamentally misguided from the get-go, as if statements in a newspaper could be verified by successfully generating fair copies of the same newspaper or falsified by failing to generate such copies.[ii]
The concept of reproducibility in empirical science has its origins in Robert Boyle’s attempts to secure public support for his research in pneumatics, and in particular for his favored experimental device, the air-pump. But as Steven Shapin and Simon Schaffer have famously and plausibly argued, Boyle’s defense of the concept of reproducibility has everything to do with social conventions governing the conversion of what’s called “belief” by a certain scientific community into what’s called “knowledge” by that community, by means of the technology available to that community, and nothing to do with scientific truth or veridicality per se:
Replication is the set of technologies [“not just the physical reiteration of the practice, but also the virtual witnessing offered by literary technology”] which transforms what counts as belief into what counts as knowledge. (Shapin and Schaffer, 1985: pp. 225-226)
In fact, however,
[i]n Boyle’s view, replication was rarely accomplished. When he came to publish the Continuation of New Experiments more than eight years after the original air-pump trials, Boyle admitted that, despite his care in communicating details of his engine and his procedures, there had been few successful replications…. This situation had not materially changed by the mid-1670s. In the seven or eight years after the Continuation, Boyle said that he had heard “of very few experiments made, either in the engine I used, or in any other made after the model thereof.” Boyle now expressed despair that these experiments would ever be replicated. He said that he was now even more willing “to set down diverse things with their minute circumstances” because “probably many of these experiments would never be re-examined by others, or re-iterated by myself.” Anyone who set about trying to replicate such experiments, Boyle said, “will find it no easy task.” (Shapin and Schaffer, 1985: pp. 59-60)
Therefore, even despite Boyle’s original defense of the concept of reproducibility, his later considered views about reproducibility actually support Hanna’s Uncertainty Principle. In The Logic of Scientific Discovery, Karl Popper remarked, without any supporting argument, that “non-reproducible single occurrences are of no significance to science” (Popper, 1934/1992: p. 66). But if the later Boyle and I are right, then this is nothing but an indefensible dogma, and it also follows that the very idea of the truth-relevance or veridicality-relevance of reproducibility for empirical science is nothing but a Popperian myth.[iii]
NOTES
[i] In fact, some particles are massless: e.g., gluons, photons, and gravitons. So they don’t have momentum, which is the product of mass and velocity. But even massless particles have velocity; and velocity and position are also canonically conjugate variables.
[ii] This is a relevant riff on a cogent remark in Wittgenstein’s Philosophical Investigations about the epistemology of mental imagery: “If the mental image of the time-table could not itself be tested for correctness, how could it confirm the correctness of the first memory [of the time-table for train departures]? (As if someone were to buy several copies of the morning paper to assure himself what what it said was true.)” (Wittgenstein, 1953: §265, p. 94e).
[iii] For example, in the article on “Reproducibility of Scientific Results” in the Stanford Encyclopedia of Philosophy, the critical worry that reproducibility is in fact not a necessary (or a sufficient) condition for the truth or veridicality of empirical science is never even considered (SEP, 2021).
REFERENCES
(Baker, 2016). Baker, M. “1,500 Scientists Lift the Lid on Reproducibility.” Nature 533, 7604: 452-454. Available online at URL = <https://www.nature.com/articles/533452a>.
(Breznau, 2023). Breznau, N. et al. “Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty.” Proceedings of the National Academy of Sciences 119, 4: e2203150119. Available online at URL = <https://doi.org/10.1073/pnas.2203150119>.
(Heisenberg, 1927/1983). Heisenberg, W. “The Actual Content of Quantum Theoretical Kinematics and Mechanics.” NASA Technical Reports. 1 December. Available online at URL = <https://ntrs.nasa.gov/citations/19840008978>.
(Mathur et al., 2023). Mathur, M. et al., “Variation Across Analysts in Statistical Significance, Yet Consistently Small Effect Sizes.” Proceedings of the National Academy of Sciences 120, 93: e2218957120. Available online at URL = <https://doi.org/10.1073/pnas.2218957120>.
(Popper, 1934/1992). Popper, K. The Logic of Scientific Discovery. London: Routledge.
(SEP, 2021). Fidler, F. and Wilcox, J. “Reproducibility of Scientific Results.” In E.N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Summer Edition. Available online at URL = <https://plato.stanford.edu/archives/sum2021/entries/scientific-reproducibility/>.
(Shapin and Schaffer, 1985). Shapin, S. and Schaffer, S. Leviathan and The Air-Pump: Hobbes, Boyle and The Experimental Life. Princeton NJ: Univ. of Princeton Press.
(Smith and Smith, 2023). Smith, J.W. and Smith, S.J. “From Scientific Reproducibility To Epistemic Humility.” Against Professional Philosophy. 3 December. Available online at URL = <https://againstprofphil.org/2023/12/03/from-scientific-reproducibility-to-epistemic-humility/>.
(Wittgenstein, 1953). Wittgenstein, L. Philosophical Investigations. Trans. G.E.M. Anscombe. New York: Macmillan.
Against Professional Philosophy is a sub-project of the online mega-project Philosophy Without Borders, which is home-based on Patreon here.
Please consider becoming a patron!
