Noah Smith — Elegant Economic Theories Get Shoved Aside by Data

The first approach -- relying on an assist from econ theory -- is called structural estimation. Basically, you make a theory of how the economy works -- how consumers behave, who owns what and what kind of costs companies face. Then you use available data to see how well the model fits the data, and to figure out the most likely values for the model’s parameters. Those parameters, or the specifics of the model, could include how risk-averse people are or how much it costs companies to change the rate at which they purchase new capital equipment.

The advantage of this method is that it allows you to pose all kinds of interesting questions. You can ask what would happen if racial discrimination suddenly vanished from the workplace -- even if that’s never actually happened in the past.

The downside is that your theory might be wrong and give you the incorrect answers. All those parameter values might be estimates of things that don’t even exist! Ideally, you can check whether the theory fits the data, and only use the theories that check out, but in reality very few models fit the data well enough to pass these tests. Structural estimation is powerful if you have a very good working theory of the world, but that’s a luxury we rarely have.

The second main approach is called the quasi-experimental technique, or natural experiments. This is when you look for a random variation in economic conditions or policy, and you observe the effect of that random variation. For example, suppose that a crazy dictator in a Caribbean nation suddenly decides to send a whole bunch of low-skilled refugees to Miami. You can use that random decision to get an idea of how an influx of low-skilled immigrants affects local wages and employment levels. Or suppose a city runs a lottery, and lets the winners go to whatever school they want. By comparing the lottery winners to the losers (who are stuck with their old schools by pure random chance), you can get an estimate of how much difference school choice makes.

This is a very powerful technique, because it doesn’t require you to have the correct theory. Just look at the effect of A on B. But it also has a severe limitation -- the more the world changes from when and where you did the study, the less useful the result is. Suppose you look at a minimum wage hike that raises hourly pay to $5.05 in New Jersey in 1992. What does that tell you about the likely effect of a plan adopted to raise the minimum wage to $15 in Seattle? Maybe a lot, but maybe nothing at all. Quasi-experimental research becomes less reliable the further you move away from the conditions where the experiment happened -- and you don’t even know how fast the reliability vanishes.…

Bloomberg View

Elegant Economic Theories Get Shoved Aside by Data

Noah Smith | Assistant Professor of Finance, Stony Brook University