Andrew Gelman — Gaydar and the fallacy of objective measurement

Stripping a phemenon of its social context, normalizing a base rate to 50%, and seeking an on-off decision: all of these can give the feel of scientific objectivity—but the very steps taken to ensure objectivity can remove social context and relevance.

Statistical Modeling, Causal Inference, and Social Science
Gaydar and the fallacy of objective measurement
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University

Lars P. Syll — Randomization — a philosophical device gone astray

When giving courses in the philosophy of science yours truly has often had David Papineau’s book Philosophical Devices (OUP 2012) on the reading list. Overall it is a good introduction to many of the instruments used when performing methodological and science theoretical analyses of economic and other social sciences issues.

Unfortunately, the book has also fallen prey to the randomization hype that scourges sciences nowadays....

Lars P. Syll’s Blog
Randomization — a philosophical device gone astray
Lars P. Syll | Professor, Malmo University

Lars P. Syll — P-hacking and data dredging

I think there are two separate issues here that depend on intent. "P-hacking" likely implies intent, and that is not necessarily a factor in all cases, and it may well not be in many if not most cases.

In some cases there may be intent to persuade by playing loose, or even to deceive. I recall that How to Lie with Statistics was required reading in the Stat 101 course I took over fifty years ago. But this is not the only issue.

As Richard Feynman famously observed, science is about not fooling ourselves. This applies to each of us individually owing to cognitive bias. Humans are smart, but we mare still primates. 

Nobody is entirely free of cognitive-affective bias. So we have to take steps to counter this tendency. Science was developed as an instrument to address this.

The reason we use rigorous method is to avoid, or at least minimize, our tendency toward being shaped by cognitive biases such as confirmation bias and anchoring. 

Methodology is about using instruments on good data in a rigorous fashion that reduces not only error in application but also bias.

There is no method that completely eliminates error and bias. On one hand, GIGO, and being human, on the other.

A lot of the problems in doing science as well as applying other rigorous instruments lies in measurement. The highest level of formality does nothing to affect errors in measurement. I happened to be thinking about the issues around measurement just prior to reading this post.

And lot of the most interesting things are difficult to measure when humans are involved and psychology enters into the data significantly. History also present issues regarding not only data quality and availability but also changing context that affects the data.

It's good we are having a debate about p-values, since there are issues there than do seem to be influential in a negative way.  And it is not only the stat, but also the data that the method is being applied to.

There are essentially three areas of interest. The first is the method, in this case probability and statistics  as formal method. The second is data and its reliability and precision, along with data collection. The third is data processing and selection. All of these are subject to error and manipulation. This is especially a problem when data sets are proprietary and are not transparent.

But the debate should not stop there. The methodological debate is not over, as some would have it. Science is always tentative on discovery and it is a work in progress. Science is often viewed as a fixed body of true knowledge. That is not a good approach to doing science. The fundamental principle of science is questioning authority, especially that of received belief, intuition and common sense.

Humans are fallible, and it is doubtful that we can ever finally work out all the kinks epistemologically and methodologically. We are a work in progress, too, just as is science.

As a discipline becomes more formalized, there is a greater tendency to emphasize formal rigor at the expense of data and evidence, especially when there are issues around data and evidence. Such tendencies are fertile ground for cognitive-affective bias.

Epistemology, logic, and methodology are foundational to gaining reliable knowledge. We need to keep this in mind.

On one hand, the search for absolute knowledge is a chimera since no criteria can be established as absolute. Criteria are stipulated. This realization should make us humble — and careful.

On the other hand, humans are not lost in a sea of relativity either. History has shown that it is possible to arrive at knowledge that is reliable and practical if intelligence is applied and bias reduced.

Lars P. Syll’s Blog
P-hacking and data dredging
Lars P. Syll | Professor, Malmo University

G.A. Barnard: The “catch-all” factor: probability vs likelihood — Debate between G. A. Barnard and Leonard Jimmie Savage

Similar to there Bayesian versus frequentist debate in statistical reasoning.

Likelihood Principle

My epistemological view on this is that the border between them is fuzzy and needs to be approached on a case by case basis, along with acknowledging a cognitive bias toward greater certainty than is attainable from the given and the reasoning about it.

Humans don't like uncertainty and have a strong bias toward minimizing it at the risk of fooling themselves. Even statisticians.

Error Statistics

G.A. Barnard: The “catch-all” factor: probability vs likelihood

Debate between G. A. Barnard and Leonard Jimmie Savage
Posted by Deborah Mayo, professor in the Department of Philosophy at Virginia Tech and visiting professor at the Center for the Philosophy of Natural and Social Science of the London School of Economics.

Diane Coyle — Statistics vs truthiness

[Howard Wainer's] Truth or Truthiness a collection of essays in effect, published as a response to this brave new world of truthiness (ie. lies that people believe because they want to) in politics and public debate. Wainer writes very clearly about statistics in general, and his main theme here, causal inference. This is of course dear to the heart of economists, and gratifyingly Wainer recognises that the profession is more scrupulous than most disciplines about causation. The book starts by underlining the importance of having a clear counterfactual in mind and thinking – thinking! – about how it might be possible to estimate the size of any causal effect. As Wainer puts it, “The real world is hopelessly multivariate,” so untangling the causality is never going to happen without careful thought.

I also discovered that one aspect of something that’s bugged me since my thesis days – when I started disaggregating macro data – namely the pitfalls of aggregation, has a name elsewhere in the scholarly forest: “The ecological fallacy, in which apparent structure exists in grouped (eg average) data that disappaears or even reverses on the individual level.” It seems it’s a commonplace in statistics – here’s one clear explanation I found. Actually, I think the aggregation issues are more extensive in economics; for example I once heard Dave Giles do a brilliant lecture on how time aggregation can lead to spurious autocorrelation results....

Any competent logician can explain how it is illogical to proceed necessarily from individual to general owing to the fallacies of composition and hasty generalization, or to proceed from the general to the individual without regard for synergy, that is, the whole is greater than the sum of the parts.

Consequently, assuming methodological individualism and microfoundations is fraught with pitfalls. Getting the causality right is difficult in social science, even in specific cases, as the difficulty in replicating studies shows.

The Enlightened Economist
Statistics vs truthiness
Diane Coyle | freelance economist and a former advisor to the UK Treasury. member of the UK Competition Commission, and acting Chairman of the BBC Trust, the governing body of the British Broadcasting Corporation

Bloomberg View — The Value of That P-Value

P-values get some notice.

Bloomberg View
The Value of That P-Value
Editorial Board

Nassim Taleb — Skin in the game – how to make economic models contact reality

Why businesses don't employ econometricians, how Wall Street was taken in by the quants, and why you shouldn't listen to them either.

Lars Syll's Blog
Skin in the game — how to make economic models contact reality
Nassim Taleb

Lars P. Syll — Probabilistic reasoning and its limits (wonkish)

Keynes.

Lars P. Syll's Blog
Probabilistic reasoning and its limits (wonkish)
Lars P. Syll | Professor of Social Studies and Associate Professor Economic History, Malmo University

Nate Silver — Seven Ways to Evaluate a Poll

1. How does the poll compare to other recent surveys of the state?

2. How does the poll compare to the polling firm’s previous surveys in the state?

3. How does the survey compare with the polling firm’s surveys in other states?

4. How does the poll compare with the national trend?

5. How does the poll compare with the historical trend in a state?

6. How does the poll relate to the electoral calendar 

7. How does the number of undecided voters compare with prior renditions of the survey?

The New York Times | Five Thirty Eight
Seven Ways to Evaluate a Poll

Nate Silver

Nate Silver — Models, Models, Everywhere

We assign a lot of bandwidth to the FiveThirtyEight presidential forecast model, but that doesn’t mean it’s the only thing you should look at to get a sense for where the election is headed. So it’s time for a quick gut-check. How does the FiveThirtyEight model — where Barack Obama’s probability of winning the Electoral College recently ticked above 70 percent — compare against alternative means of forecasting the election?

There are a number of other statistical forecasting systems, most of which rely on polls, economic variables or some combination thereof.

I tracked down about every one of these models that I could find, subject to the condition that it couched its forecast in probabilistic terms (or that it was well-documented enough to allow this to be inferred with relative ease, like from the standard error that the model stated).

In my view, it’s in estimating the uncertainty in a forecast where most of the challenge and intrigue lies. To paraphrase Charles Barkley, any knucklehead can make a point prediction — but it takes brains to calculate a confidence interval.

Read it at The New York Times | Five Thirty Eight
Models, Models, Everywhere
Nate Silver

Lars P. Syll — The unknown knowns of modern macroeconomics

More on the folly of drawing sweeping conclusions from DSGE modeling. This time based on nature of statistical reasoning.

If one follows über-statistician Nate Silver's blog, Five Thirty Eight, at The New York Times, one has grasp of how difficult statistical modeling is socially. Politics is one of the most researched and and polled fields in existence, with huge amounts spent on data gathering and processing. Nate's record has been good so far, but he explains the difficulty of coming to any tight conclusions owing to a variety of factors and always qualifies his statements.

No one trying to figure out political outcomes would dream of assuming a representational agent or perfect knowledge to simplify the design problem. Where do economist get the pass to do this and not get called out for it, especially when predictions turn out to be wildly off the mark. After all, this flawed reasoning goes into the grinder of policy making.

Lars sums it up reagrding DSGE:

The root of our problem ultimately goes back to how we look upon the data we are handling. In modern neoclassical macroeconomics – Dynamic Stochastic General Equilibrium (DSGE), New Synthesis, New Classical and “New Keynesian” – variables are treated as if drawn from a known “data-generating process” that unfolds over time and on which we therefore have access to heaps of historical time-series. If we do not assume that we know the “data-generating process” – if we do not have the “true” model – the whole edifice collapses.

Economic actors behave no more predictably than voters in that they are the same people performing a similar function — choosing. The actors are just as diverse and volatile. A big difference is that voters have very limited choice well-recognized motives, and yet prediction is still dicey. On the other hand, economic actors have a wide range of choices and motives — and prediction can be relatively precise?

Read it at Lars P. Syll's Blog
The unknown knowns of modern macroeconomics
by Lars P. Syll

Sean Carmody — The power and peril of FRED

“FRED” is the St.Louis Federal Reserve Economic Database. It is an excellent repository of economic data, currently boasting 45,000 time-series from 42 data sources. The web-site offers a powerful interface for creating charts of FRED data. Unfortunately, it is a little too powerful, offering a rather dangerous feature: the secondary axis.

I have railed against secondary axes before. They tend to lure the viewer into seeing spurious correlations....

Read it at Stubborn Mule
The power and peril of FRED
by Stubborn Mule

Lars P. Syll — Keynes’s critique of econometrics (wonky)

Unfortunately, economists often hold the view that Keynes’s criticisms of econometrics is the conclusions of a sadly misinformed and misguided intellectual who disliked and did not understand much of it. This is really a gross misapprehension. To be careful and cautious is not the same as to dislike. And as any perusal of the mathematical-statistical and philosophical works of people like for example Nancy Cartwright, Chris Chatfield, Hugo Keuzenkamp or Arios Spanos would show, the same critique is more or less put forward by respected authorities.

I would argue, against “common knowledge”, that Keynes did not misunderstand the crucial issues at stake in the development of econometrics. Quite the contrary. He knew them all too well – and was not satisfied with the validity and philosophical underpinning of the assumptions made for applying its methods.

Read it at Lars P. Syll' Blog
Keynes’s critique of econometrics
by Lars P. Syll | Professor, Malmo University