Andrew Gelman — Rosenbaum (1999): Choice as an Alternative to Control in Observational Studies

Paul Rosenbaum’s 1999 paper “Choice as an Alternative to Control in Observational Studies” is really thoughtful and well-written. The comments and rejoinder include an interesting exchange between Manski and Rosenbaum on external validity and the role of theories....

Important in the most studies in social science, including economics, are necessarily observational rather than experimental. The question is how to design observational studies to make them as close as possible to experimental studies where tight control of variables is available.

Design problems involves choice that are implicit assumptions. Designers need to carefully choose (assume) consciously and intentionally rather than presume, which runs the risk of hidden assumptions that might have been avoided through greater advertence.

A good example is the Reinhart and Rogoff historical study on the effect of public debt that was vitiated by inadvertence to the different consequences of public debt under different monetary systems. MMT economists immediately pointed out that the presumption that all public debt is the same in its effects is false, owing to operational differences under different monetary regimes historically. This is actually more significant than the computational errors that were discovered subsequently and highly publicized in the media.

The R&R study was highly influential in policy formulation even though MMT economists had pointed out at the time of its release, and this led to very damaging effects when policy based on the study was implemented. This should not have happened in a professional environment.

Statistical Modeling, Causal Inference, and Social Science
Rosenbaum (1999): Choice as an Alternative to Control in Observational Studies
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University

Lars P. Syll — Why testing axioms is necessary in economics

Where do axioms come from and how are they tested?

Axioms are starting points of deductive systems. They are stipulations.

Axioms must avoid the traps of illogic, circularity, and infinite regress.

Axioms in scientific theories are assumptions derived either from induction as generalization from experience (data) or abduction in C. S. Peirce's sense as discovery through "educated guessing."

The objective of scientific inquiry is not to "prove" axioms, since axioms are the basis of proof in a logical system. Axioms function as criteria for syntactical or logical truth, also called "necessity."

Axioms in hypothetical-deductive systems provide the basis for the scientific method. 

Assumptions of representational models stand or fall with the testing of models they are used to construct. 

Scientific models are used to generate hypotheses that can be tested empirically by deriving hypotheses as theorems from the assumptions that serve as axioms for the system.

Failure of a hypothesis as a theorem of deductive system reveals semantic inconsistency or incompleteness and calls the system into question as a coherent explanation of the data. 

This also calls the explanatory model into question as providing a causal explanation based on a representational model purporting to show causal transmission.

Correlation is not causation. Without a theory explicating transmission, there is no properly scientific explanation.

Lars P. Syll’s Blog

Why testing axioms is necessary in economics

Lars P. Syll | Professor, Malmo University

Noah Smith — Should theories be testable?

I don't see why we should insist that any theory be testable. After all, most of the things people are doing in math departments aren't testable, and no one complains about those, do they? I don't see why it should matter if people are doing math in a math department, a physics department, or an econ department.

I think testability starts to matter when you start thinking about applying theories to the real world. This is why I get annoyed when people ignore the evidence in business cycle theory, but not when they do it in pure theory.

Suppose you're studying the properties of repeated games. Who cares if those games represent anything that really exists today?… 

But when you start making models that claim to be about some specific real thing (e.g. monetary policy), you're implying that you think those models should be applied. And then, it seems important to me to have some connection to real data, to tell if the theory is a good one to use, or a crappy one to use. That's testability.

Anyway, this sort of seems very college-freshman-dorm-discussion-level when I write it out like this, but I think there are a surprising number of people who don't seem to agree with it...

Exactly.

Science is not a thing but an activity. It's what scientists do. There are three major areas in doing science: 1) theoretical science (aka pure science) focusing chiefly on formalization, 2) experimental science, heavily involving design of experiments, and 3) applied science, e.g., engineering and technology, medicine, policy science, which apply general principles to specific conditions. These are separate fields, notably physics, where theoretical physics is advanced math, and experimental and applied physics are about not only formal knowledge but also practical skill. The three fields require development and use of different knowledge and skills. All are essential the scientific enterprise that results not only in expansion of the knowledge base but also in technological innovation.

What is so difficult to understand about this?

Noahpinion
Should theories be testable?
Noah Smith | Assistant Professor of Finance, Stony Brook University