So. Yes, it seems goofy to draw a bright line between p = 0.048 and p = 0.052. But it’s also goofy to draw a bright line between p = 0.2 and p = 0.005. There’s a lot less information in these p-values than people seem to think. [Just about everyone would say that p = 0.2 is insignificant and p = 0.005 is significant.]
So, when we say that the difference between “significant” and “not significant” is not itself statistically significant, “we are not merely making the commonplace observation that any particular threshold is arbitrary—for example, only a small change is required to move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical significance. Rather, we are pointing out that even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities.”Moral of the story: Math and statistics are tools for human use, not messengers from heaven.
Over the twenty years that I have been pursuing an Islamic approach — focusing on the production of USEFUL knowledge, I have managed to heal all three of these divides. This happens naturally, when you focus on solution of real world problems. You automatically need to combine information coming from many different specialization areas. You need to use reasoning and also intuition. You also need to use both theory and its applications to the real world experiences. This leads to substantial changes in the subject matter itself. I have applied this approach with great success to Econometrics, Statistics, Microeconomics, Macroeconomics, Experimental Economics, and even Mathematics itself. I am in process of creating textbooks and teaching materials in all of these areas. Because my work is most advanced in the area of Statistics, I am working on putting it all together in a new course on Real Statistics: An Islamic Approach. There is a large amount of pre-existing material – lectures, texts, exercises, references – that I have created over the past decade on working on this course. However, as I progress, I keep learning new things, and this time I want to put together a polished new version of this course for public use. My primary target audience is teachers of statistics — I would like to persuade them to use this new approach to teach statistics. Those who would like to follow my progress as I construct a new website on a lecture by lecture basis gradually are encourged to fill in the following Registration form. I will use emails to notify them when I complete a new lecture, and also invite feedback on what is there, so that we can build it up with clarity and consensus....All thinking, since humans think in language, is based on context, meaning being determined by context. The shaper of context is the worldview in which the group is functioning. In the West, the contemporary worldview was shaped by the Western history, chiefly Greek thought, Judaeo-Christian religion, Roman law, and modern science. Its intellectual products were shaped by the Western intellectual tradition that culminated most recently in the rise of science, which new supervenes over what preceded it. The basic assumption of the Western scientific world view is methodological naturalism, which many if not most of the foremost exponents equate with metaphysical materialism.
Sometime during this process of switching from teaching theory to teaching how to solve real world problems, I came across the “Statistics” textbook of David Freedman. This textbook actually implemented exactly this idea that I had come to believe in — do statistics in context of solving real world problems. One amazing characteristic of this textbook is that it has no mathematical formula – ZERO. Freedman explained that students use formulae as crutches to prevent them from thinking. So he explains all concepts in words only, exactly the same insight that I had learnt on my own. Formulas teach you techniques for calculation. We don’t need these techniques — leave them to the computer. We need to UNDERSTAND what these calculations mean. That is a VERY DIFFERENT process. I got involved in an email correspondence with David Freedman, who had very similar experience to mine. He had started out as a very heavily mathematically oriented researchers. His early papers are all very heavy mathematically. Later, when he got involved in doing some testimony in real world court cases, he realized that all of the theory he had learnt was useless in the real world. This is because the assumptions we make in theory are almost always false in the real world. Then he had to learn how to do real world statistics, exactly as I have had to do. Since most fancy assumptions we make in statistics and econometrics are wrong, we need to learn how to do simple and basic inferences, which actually makes life much easier for students of the subject — we need to teach them basic and intuitive things, not complex models and math....An Islamic Worldview
"Pareidolia" refers to the common human practice of looking at random outcomes but trying to impose patterns on them. For example, we all know in the logical part of our brain that there are a roughly a kajillion different variables in the world, and so if we look through the possibilities, we will will have a 100% chance of finding some variables that are highly correlated with each other. These correlations will be a matter of pure chance, and they carry no meaning. But when my own brain, and perhaps yours, sees one of these correlations, I can feel my thoughts start searching for a story to explain what looks to my eyes like a connected pattern.…
Classes in statistics emphasize that "correlation doesn't mean causation." The lesson here is even stronger. Correlation doesn't necessarily mean anything at all.Classes in statistics emphasize that "correlation doesn't mean causation." The lesson here is even stronger. Correlation doesn't necessarily mean anything at all.
And, no, I don’t think it’s in general a good idea to pit theories against each other in competing hypothesis tests. Instead I’d prefer to embed the two theories into a larger model that includes both of them.
… a nonrandom sample tells us about a population, but we don’t know how precisely: we can’t determine a margin of error or a confidence level.A lot of mistakes occur from generalizing special cases. This tendency to overgeneralize, along with the tendency to absolutize, often infects formulation of assumptions in "scientific" modeling.
Stripping a phemenon of its social context, normalizing a base rate to 50%, and seeking an on-off decision: all of these can give the feel of scientific objectivity—but the very steps taken to ensure objectivity can remove social context and relevance.Statistical Modeling, Causal Inference, and Social Science
To put it another way: Sure, it’s fine to say that you “cannot reach external validity” from your sample alone. But in the meantime you still need to make decisions. We don’t throw away the entire polling industry just cos their response rates are below 10%; we work on doing better. Our samples are never perfect but we can make them closer to the population.
Remember the Chestertonian principle that extreme skepticism is a form of credulity.Making assumptions is necessary. However, it is also necessary to recognize and acknowledge limitations. Formal modeling is never more accurate for the math than the assumptions permit.
The knives are out for the p-value. This statistical quantity is the Holy Grail for empirical researchers across the world -- if your study finds the right p-value, you can get published in a credible journal, and possibly get a good university tenure-track job and research funding. Now a growing chorus of voices wants to de-emphasize or even ban this magic number. But the crusade against p-values is likely to be a distraction from the real problems afflicting scientific inquiry....
The real danger is that when each study represents only a very weak signal of scientific truth, science gets less and less productive. Ever more researchers and ever more studies are needed to confirm each result. This process might be one reason new ideas seem to be getting more expensive to find.
If we want to fix science, p-values are the least of our problems. We need to change the incentive for researchers to prove themselves by publishing questionable studies that just end up wasting a lot of time and effort.There is a difference in proving that one has the ability to use the tools of one's trade and using the tools to produce authentic, useful and elegant output.
From my 2009 paper with Weakliem:Throughout, we use the term statistically significant in the conventional way, to mean that an estimate is at least two standard errors away from some “null hypothesis” or prespecified value that would indicate no effect present. An estimate is statistically insignificant if the observed value could reasonably be explained by simple chance variation, much in the way that a sequence of 20 coin tosses might happen to come up 8 heads and 12 tails; we would say that this result is not statistically significantly different from chance. More precisely, the observed proportion of heads is 40 percent but with a standard error of 11 percent—thus, the data are less than two standard errors away from the null hypothesis of 50 percent, and the outcome could clearly have occurred by chance. Standard error is a measure of the variation in an estimate and gets smaller as a sample size gets larger, converging on zero as the sample increases in size.
As shown over and over again when significance tests are applied, people have a tendency to read ‘not disconfirmed’ as ‘probably confirmed.’ Standard scientific methodology tells us that when there is only say a 10 % probability that pure sampling error could account for the observed difference between the data and the null hypothesis, it would be more ‘reasonable’ to conclude that we have a case of disconfirmation. Especially if we perform many independent tests of our hypothesis and they all give about the same 10 % result as our reported one, I guess most researchers would count the hypothesis as even more disconfirmed.
We should never forget that the underlying parameters we use when performing significance tests are model constructions. Our p-values mean nothing if the model is wrong. And most importantly — statistical significance tests DO NOT validate models!Lars P. Syll’s Blog
AbstractUncritically adopting universal rules and criteria is a sign of lazy thinking and likely ideological thinking aka dogmatism as well.
In science publishing and many areas of research, the status quo is a lexicographic decision rule in which any result is first required to have a p-value that surpasses the 0.05 threshold and only then is consideration—often scant—given to such factors as prior and related evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain. There have been recent proposals to change the p-value threshold, but instead we recommend abandoning the null hypothesis significance testing paradigm entirely, leaving p-values as just one of many pieces of information with no privileged role in scientific publication and decision making. We argue that this radical approach is both practical and sensible.
Paul Rosenbaum’s 1999 paper “Choice as an Alternative to Control in Observational Studies” is really thoughtful and well-written. The comments and rejoinder include an interesting exchange between Manski and Rosenbaum on external validity and the role of theories....Important in the most studies in social science, including economics, are necessarily observational rather than experimental. The question is how to design observational studies to make them as close as possible to experimental studies where tight control of variables is available.
There seems to be an expectation in science that the people who gather a dataset should also be the ones who analyze it. But often that doesn’t make sense: what it takes to gather relevant data has little to do with what it takes to perform a reasonable analysis. Indeed, the imperatives of analysis can even impede data-gathering, if people have confused ideas of what they can and can’t do with their data.
I’d like us to move to a world in which gathering and analysis of data are separated, in which researchers can get full credit for putting together a useful dataset, without the expectation that they perform a serious analyses. I think that could get around some research bottlenecks.
It’s my impression that this is already done in many areas of science—for example, there are public datasets on genes, and climate, and astronomy, and all sorts of areas in which many teams of researchers are studying common datasets. And in social science we have the NES, GSS, NLSY, etc. Even silly things like the Electoral Integrity Project—I don’t think these data are so great, but I appreciate the open spirit under which these data are shared....Statistical Modeling, Causal Inference, and Social Science
In the Posterior Analytics, Aristotle places the following crucial condition on proper knowledge: we think we have knowledge of a thing only when we have grasped its cause (APost. 71 b 9–11. Cf. APost. 94 a 20). That proper knowledge is knowledge of the cause is repeated in the Physics: we think we do not have knowledge of a thing until we have grasped its why, that is to say, its cause (Phys. 194 b 17–20). Since Aristotle obviously conceives of a causal investigation as the search for an answer to the question “why?”, and a why-question is a request for an explanation, it can be useful to think of a cause as a certain type of explanation. (My hesitation is ultimately due to the fact that not all why-questions are requests for an explanation that identifies a cause, let alone a cause in the particular sense envisioned by Aristotle.) — Stanford Encyclopedia of Philosophy
