Ronald Coase once remarked, “If you torture the data enough, nature will always confess.”
Governments increasingly rely on sophisticated statistical models to justify policies. Before accepting a model-generated number as a basis for public policy, we should ask two questions: How much comes from actual observations and how much from assumptions? And what incentives surround the institutions producing the research?
Consider employment data. The Bureau of Labor Statistics publishes its Employment Situation monthly. Even a century gives us only about 1,200 monthly observations.
Economists cannot conduct the same controlled experiment. We cannot run the American economy 10,000 times, raise interest rates in half the experiments, and compare the results. We have one economy and one history.
Models help overcome this limitation. Researchers estimate relationships from observed data and simulate millions of possible outcomes. This is useful, but it creates a distinction forgotten.
Suppose I estimate a model from 1,200 historical observations and use it to generate one million simulated observations. I now have one million additional numbers. I do not have one million additional observations of reality.
Simulation can multiply observations inside a computer. It cannot multiply information about the real world.
The same problem arises with assumptions. Financial models based on normally distributed returns, for example, can underestimate extreme events because financial returns often exhibit “fat tails.” Change the assumptions and the result may change, although the computer will calculate either answer with extraordinary numerical precision.
Precision of calculation and accuracy of knowledge are not the same thing.
Climate modeling illustrates the broader problem. Climate scientists use instrumental measurements, proxy evidence and sophisticated physical models. But long-term projections necessarily depend partly on assumptions about future emissions, economic development, technological change and physical responses.
A model may effectively tell us: If assumptions A, B and C hold, our model estimates outcome X.
Public debate can transform that into: Scientists have determined X will happen.
Those statements are not equivalent. A scenario is not an observation from the future.
There is another issue economists should recognize: incentives.
Scientific research requires money, and the government is an important source of climate-research funding. This does not demonstrate that scientists alter conclusions to obtain grants. That would require evidence.
But economists routinely study how incentives affect pharmaceutical companies, defense contractors, banks and consumers. Why assume incentives cease operating inside universities?
There may be an even subtler mechanism.
Suppose early evidence favors hypothesis A over hypothesis B. Government directs more funding toward A. Universities establish research centers around it. Graduate students enter the field. More datasets are collected and papers published. That larger infrastructure produces still more evidence associated with A, potentially attracting still more funding.
Evidence influences funding, but funding can also influence the production of future evidence.
Economists have a word for relationships in which causality runs both ways: endogeneity.
Call this the endogeneity of scientific consensus.
This does not mean the dominant hypothesis is wrong. It may have attracted resources precisely because its evidence was stronger. But once a research program becomes institutionally dominant, the size of its literature should not automatically be treated as evidence independent of the institutional process that helped produce it.
The appropriate response is not to reject models or scientific consensus. It is to demand humility about what they can establish.
Before imposing costly policies based on models, ask how much of the conclusion comes from observation, how much from assumptions, whether alternative assumptions materially change the result, and whether credible competing hypotheses received serious examination.
Coase’s warning applies to advocates and skeptics alike.
A million simulations may tell us a great deal about our model. They do not give us a million additional observations of reality.
Science advances not by protecting today’s consensus, but by exposing it to tomorrow’s evidence.
