It’s not just p=0.048 vs. p=0.052 — Andrew Gelman

Numbers may be eternal and unchanging, but they are not gods. Andrew Gelman observes that we also have to step back and use our common sense regarding what the numbers actually say, instead of drawing arbitrary lines based on self-imposed criteria like "significance" and then take them as "messages from the gods." It doesn't work like that. Formalism only goes so far.

So. Yes, it seems goofy to draw a bright line between p = 0.048 and p = 0.052. But it’s also goofy to draw a bright line between p = 0.2 and p = 0.005. There’s a lot less information in these p-values than people seem to think. [Just about everyone would say that p = 0.2 is insignificant and p = 0.005 is significant.]

So, when we say that the difference between “significant” and “not significant” is not itself statistically significant, “we are not merely making the commonplace observation that any particular threshold is arbitrary—for example, only a small change is required to move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical significance. Rather, we are pointing out that even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities.”

Moral of the story: Math and statistics are tools for human use, not messengers from heaven.

Statistical Modeling, Causal Inference, and Social Science
It’s not just p = 0.048 vs. p = 0.052
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University