Noah Smith — Why 'Statistical Significance' Is Often Insignificant

The knives are out for the p-value. This statistical quantity is the Holy Grail for empirical researchers across the world -- if your study finds the right p-value, you can get published in a credible journal, and possibly get a good university tenure-track job and research funding. Now a growing chorus of voices wants to de-emphasize or even ban this magic number. But the crusade against p-values is likely to be a distraction from the real problems afflicting scientific inquiry....

The real danger is that when each study represents only a very weak signal of scientific truth, science gets less and less productive. Ever more researchers and ever more studies are needed to confirm each result. This process might be one reason new ideas seem to be getting more expensive to find.

If we want to fix science, p-values are the least of our problems. We need to change the incentive for researchers to prove themselves by publishing questionable studies that just end up wasting a lot of time and effort.

There is a difference in proving that one has the ability to use the tools of one's trade and using the tools to produce authentic, useful and elegant output.

Bloomberg View
Why 'Statistical Significance' Is Often Insignificant
Noah Smith

Andrew Gelman — My favorite definition of statistical significance

From my 2009 paper with Weakliem:

Throughout, we use the term statistically significant in the conventional way, to mean that an estimate is at least two standard errors away from some “null hypothesis” or prespecified value that would indicate no effect present. An estimate is statistically insignificant if the observed value could reasonably be explained by simple chance variation, much in the way that a sequence of 20 coin tosses might happen to come up 8 heads and 12 tails; we would say that this result is not statistically significantly different from chance. More precisely, the observed proportion of heads is 40 percent but with a standard error of 11 percent—thus, the data are less than two standard errors away from the null hypothesis of 50 percent, and the outcome could clearly have occurred by chance. Standard error is a measure of the variation in an estimate and gets smaller as a sample size gets larger, converging on zero as the sample increases in size.

Statistical Modeling, Causal Inference, and Social Science
My favorite definition of statistical significance

Andrew Gelman | Professor of Statistics and Political Science, and director of the Applied Statistics Center at Columbia University

Andrew Gelman — When considering proposals for redefining or abandoning statistical significance, remember that their effects on science will only be indirect!

Summary: The end-in-view is doing good science and avoiding junk science, which is proliferating. Adjusting standards, etc. are only means to an end. There are no silver bullets or magic wands. Doing good science depends on good design, accurate measurement, and replication.

Statistical Modeling, Causal Inference, and Social Science
When considering proposals for redefining or abandoning statistical significance, remember that their effects on science will only be indirect!
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University

Andrew Gelman — Alan Sokal’s comments on “Abandon Statistical Significance”

If you are keeping up with this. Some finer points.

From the epistemological point of view, criticism of statistical significance here is based on questioning a criterion that is stipulated, i.e., defined arbitrarily.

Doing so gives formalization and modeling a greater importance than advancing understanding. That is unscientific.

A pragmatic approach is more appropriate than a strictly formal one, especially as an institutional norm.

Formal rigor is a necessary condition but not a sufficient one. Don't lose the forest for the trees.

The quest for knowledge is the quest for a consensual world view based on reasoning and evidence. This is based on many inputs and their consilience within the framework. This is an ongoing enterprise and science is not the only contributor to it. But science based on naturalism is a very significant contributor, tethering knowledge to logical pedigree and empirical warrant.

Statistical Modeling, Causal Inference, and Social Science
Alan Sokal’s comments on “Abandon Statistical Significance”
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University

Also

true economics
Ambiguity Fun: Perceptions of Rationality?Constantin Gurdgiev | chairman of the Ireland-Russia Business Association, contributor and former editor of Business & Finance Magazine, and lecturer in Finance with Trinity College, Dublin

Lars P. Syll — Time to abandon statistical significance

As shown over and over again when significance tests are applied, people have a tendency to read ‘not disconfirmed’ as ‘probably confirmed.’ Standard scientific methodology tells us that when there is only say a 10 % probability that pure sampling error could account for the observed difference between the data and the null hypothesis, it would be more ‘reasonable’ to conclude that we have a case of disconfirmation. Especially if we perform many independent tests of our hypothesis and they all give about the same 10 % result as our reported one, I guess most researchers would count the hypothesis as even more disconfirmed.

We should never forget that the underlying parameters we use when performing significance tests are model constructions. Our p-values mean nothing if the model is wrong. And most importantly — statistical significance tests DO NOT validate models!

Lars P. Syll’s Blog
Time to abandon statistical significance
Lars P. Syll | Professor, Malmo University

Abandon Statistical Significance — Blakeley B. McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L. Tacket

Abstract

In science publishing and many areas of research, the status quo is a lexicographic decision rule in which any result is first required to have a p-value that surpasses the 0.05 threshold and only then is consideration—often scant—given to such factors as prior and related evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain. There have been recent proposals to change the p-value threshold, but instead we recommend abandoning the null hypothesis significance testing paradigm entirely, leaving p-values as just one of many pieces of information with no privileged role in scientific publication and decision making. We argue that this radical approach is both practical and sensible.

Uncritically adopting universal rules and criteria is a sign of lazy thinking and likely ideological thinking aka dogmatism as well.

This move would overturn the existing scientific publishing model, it is unlikely to happen without considerable opposition. This model is key in establishing reputational credibility and advancement in the profession. Players like set rules. This is especially true in formal subjects, where training focuses on producing "the right answer" based on customary application of formal methods. The downside is group think and imposition of a consensus reality.

Abandon Statistical Significance

Blakeley B. McShane, David Gal, Andrew Gelman, Christian Robert, and Jennifer L. Tackett