Andrew Gelman — Yes, you can do statistical inference from nonrandom samples. Which is a good thing, considering that nonrandom samples are pretty much all we’ve got.

To put it another way: Sure, it’s fine to say that you “cannot reach external validity” from your sample alone. But in the meantime you still need to make decisions. We don’t throw away the entire polling industry just cos their response rates are below 10%; we work on doing better. Our samples are never perfect but we can make them closer to the population.

Remember the Chestertonian principle that extreme skepticism is a form of credulity.

Making assumptions is necessary. However, it is also necessary to recognize and acknowledge limitations. Formal modeling is never more accurate for the math than the assumptions permit.

Reasoning is a tool of intelligence. It is not a magic wand. Taking reasoning for a magic wand because it is highly formalized is magical thinking.

It is important to distinguish necessity from contingency. Necessity is based on logic necessity (tautology) and logical impossibility (contradiction). These are purely syntactical, that is, based on applying rules to signs. Logical necessity is probability one; contradiction is probability zero. All description is contingent on observation.

Statistics is a reasoning tool for dealing with contingency. The formal aspect of the tool does not vary, but its application is dependent on assumption and measurement. Thinking that the results will be the same owing to the invariant formal aspect is a mistake. Results can never be more precise than measurements or more accurate than assumptions permit, no matter how rigorous the formal methods applied.

Statistical Modeling, Causal Inference, and Social Science
Yes, you can do statistical inference from nonrandom samples. Which is a good thing, considering that nonrandom samples are pretty much all we’ve got.
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University