Showing posts with label scientific method. Show all posts
Showing posts with label scientific method. Show all posts

Thursday, January 9, 2020

Lars P. Syl — Is economics — really — predictable?


There is a big difference between predicting and forecasting. Scientific theory is about causal explanation and prediction through formulating testable hypothesis that challenge the theory rigorously based on experimental evidence. Forecasting is making educated guesses based on limited and variable information information. The former applies chiefly to ergodic systems and the latter to non-ergodic, or if the system is actually ergodic, not enough it known about it to construct a rigorous causal explanation.

The ideal causal explanation is in terms of deterministic functions in which a rule applied to a single measurable input results invariably in a single measurable output. The debate over whether statistics can deliver on causal explanation is still raging, in light of the principle that correlation is not causation. For example, Einstein rejected that it could and continued to seek for a set of deterministic functions as the basis for causal explanation in physics, viewing QM as an admission of lingering ignorance about the laws of nature owing to QM being stochastic.

Libraries are full of tomes debating the details of this, but this is a rough outline to which most agree. Thus, forecasting can be "scientific" and even based on causal explanation, but it fails the test of prediction strictly speaking based on performance. The subject matter of the social sciences is more like the weather than planetary motion, and so the results are mixed. There is no ephemeris for economic cycles.

The question how sharp the line dividing prediction and forecasting may be, and this is a matter of argument since no set of criteria are universally agreed upon. Some conventional economists seek to categorize economics with the natural sciences rather than the social sciences, for example. Is that justifiable?

To understand Keynes, it is necessary to take his Treatise on Probability as a starting point.

Why is this important other than philosophically? Because humans are ideological and affected by presumptions as hidden assumptions. We tend to overestimate our level of knowledge, on one hand, and other the other, we inflate our degree of confidence.

To paraphrase Richard Feynman the purpose of science is to prevent us from fooling ourselves and we ourselves are the easiest people to fool (owing to cognitive-affective bias).

Lars P. Syll’s Blog
Is economics — really — predictable?
Lars P. Syll | Professor, Malmo University

Wednesday, August 21, 2019

Econometrics and the problem of unjustified assumptions — Lars P. Syll


This is important but may be too wonkish for those who are not intimately familiar with econometrics. So let me try to simplify it and universalize it.

The basic idea in logical reasoning is that an argument is sound if and only if the premises are true and the logical form is valid.  Then the conclusion follows as necessarily true.

This is the basis of scientific reasoning.

In modeling, a set of assumptions, both substantive and procedural, is stipulated, that is, assumed to be true. In a well-founded model all the assumptions that make substantive claims are known to be true empirically on the basis of evidence. This is called semantic truth. The logical truth of logical form is formal proof. This is called syntactical truth. Only the former contains substance. The latter is purely procedural.

A key methodological assumption of the scientific method is naturalism. Being "scientific" signifies being based on observation, rather than say, intuition or "common sense," that is, self-evidence. No self-evident first principles — that's doing philosophy, not science. Not that such speculation is not useful. It's just not science and should not be conflated with science. There is often a tendency to do so.

This presents two major difficulties with scientific modeling versus philosophical speculation. The first is the empirical warrant of the starting points, the stipulations that are assumed to be true and serve as the premises of the argument. The second is knowing that all relevant information is included in the assumptions. This is called identification.

Paraphrasing Richard Feynman, we do science in order to avoid fooling ourselves and we are the easiest ones to fool (owing to confirmation bias, for example). This requires following scientific method scrupulously when substantial claims are made.

Keynes pointed out to Roy Harrod that econometrics did not conform to this strict procedure and that owing to the nature of the subject matter, economics was "moral science," which at the time signified what we would now call "philosophy." The social sciences and much of psychology fall into this category. They are basically speculative exercises that employ some formal methods that may be scientific, or not. 

Accounting is a formal method that is proto-scientific in the sense that double entry it is made up of tautologies. But the entries can be checked for substance against journals and inventories. It is a method to prevent fooling ourselves on one hand, and to prevent cheating on the other.

When accounting tautologies (identities) are interpreted causally, then causal explanation demands empirical corroboration through data, e.g., measurable changes in stocks and flows.

Lars P. Syll’s Blog
Econometrics and the problem of unjustified assumptions
Lars P. Syll | Professor, Malmo University

See also

Bond Economics
Comments On "Business Cycle Anatomy"
Brian Romanchuk

Tuesday, May 21, 2019

Michelle Starr — Tomorrow The Definition of The Kilogram Will Change Forever. Here's What That Really Means

This is a big deal even though it won't be noticed by most people. However, precise measurement essential to science and measurement involves application of metrics defined by criteria. The units and their criteria are arbitrary. There was no such thing as a kilogram prior to the development and introduction of the metric system. Same with other measurement systems. The "trick" is to establish a constant criterion in a relative universe. That is as close to an absolute as human can construct. This post explains how the issue has been approached in physics.

Of course, a great deal more precision can be arrived at through physics than other science, which strive to use the measurements developed by physics in so far as possible, but physical measurement is applicable only to quantity. Measuring quality presents greater challenges. So do psychological "dimensions."

Science Alert
Tomorrow The Definition of The Kilogram Will Change Forever. Here's What That Really Means
Michelle Starr

Saturday, February 9, 2019

Peter Cooper — MMT is Politically Open and Applicable to Both Capitalism and Socialism

Modern Monetary Theory (MMT) offers an understanding of sovereign (and non-sovereign) currencies that is applicable to a wide range of economic systems, including capitalist and socialist ones. Irrespective of the personal political preferences of its proponents, the theoretical framework in itself is neutral on the appropriate balance between public sector and private sector activity, or the relative merits of capitalism and socialism. In contrast to neoclassical theory, which starts from a general presumption in favor of private market-based activity except where the existence of market failure in excess of government failure can be explicitly established, MMT as a theory characterizes the appropriate mix of public and private activity as a social (or political) choice....
This is important because the substantive and procedural assumptions of an approach to inquiry, coupled with presumptions that are often unstated assumptions, determine the framework and therefore bias the outcome of analysis toward the assumptions, both stated and unstated. If assumptions and presumptions contain a normative element in addition to a positive (descriptive) one, then the approach is values-based, which in scientific terms implies subjective rather than objective.

It is difficult to impossible to formulate a theory involving social, political or economic data that is not normative to some degree owing to cultural and subcultural bias in that cognitive biases are endemic. For example, cognitive science reveals that reason cannot be completely disentangled from feeling in brain. Scientists recognize that attempt to minimize the subjective factors in the interest of approaching objectivity as closely as possible.

heteconomist
MMT is Politically Open and Applicable to Both Capitalism and Socialism
Peter Cooper

Sunday, April 29, 2018

Brian Romanchuk — Can We Falsify Models With Time-Varying Parameters?

In a previous article, I argued that having unknown fixed parameters within many economic models does not create much in the way of uncertainty: just extend the range of historical data available, and we can pin down the parameter values. This article covers a related case: what if we allow parameters to vary with time? This possibility will make it impossible to make reliable forecasts with the model. However, such models have another defect: they can be fitted to practically any data set, making the model non-falsifiable. This can be illustrated by thinking about the simplest model of stock index returns. My argument that the apparent success of mainstream macro modelling techniques relies on the use of such non-falsifiable models….

Bond Economics
Can We Falsify Models With Time-Varying Parameters?
Brian Romanchuk

Saturday, March 17, 2018

Thursday, November 2, 2017

Noah Smith — Why 'Statistical Significance' Is Often Insignificant

The knives are out for the p-value. This statistical quantity is the Holy Grail for empirical researchers across the world -- if your study finds the right p-value, you can get published in a credible journal, and possibly get a good university tenure-track job and research funding. Now a growing chorus of voices wants to de-emphasize or even ban this magic number. But the crusade against p-values is likely to be a distraction from the real problems afflicting scientific inquiry....
The real danger is that when each study represents only a very weak signal of scientific truth, science gets less and less productive. Ever more researchers and ever more studies are needed to confirm each result. This process might be one reason new ideas seem to be getting more expensive to find.
If we want to fix science, p-values are the least of our problems. We need to change the incentive for researchers to prove themselves by publishing questionable studies that just end up wasting a lot of time and effort.
There is a difference in proving that one has the ability to use the tools of one's trade and using the tools to produce authentic, useful and elegant output.

Tuesday, October 3, 2017

Andrew Gelman — When considering proposals for redefining or abandoning statistical significance, remember that their effects on science will only be indirect!


Summary: The end-in-view is doing good science and avoiding junk science, which is proliferating. Adjusting standards, etc. are only means to an end. There are no silver bullets or magic wands. Doing good science depends on good design, accurate measurement, and replication.

Statistical Modeling, Causal Inference, and Social Science
When considering proposals for redefining or abandoning statistical significance, remember that their effects on science will only be indirect!
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University

Andrew Gelman — Alan Sokal’s comments on “Abandon Statistical Significance”


If you are keeping up with this. Some finer points.

From the epistemological point of view, criticism of statistical significance here is based on questioning a criterion that is stipulated, i.e., defined arbitrarily.

Doing so gives formalization and modeling a greater importance than advancing understanding. That is unscientific.

A pragmatic approach is more appropriate than a strictly formal one, especially as an institutional norm.

Formal rigor is a necessary condition but not a sufficient one. Don't lose the forest for the trees.

The quest for knowledge is the quest for a consensual world view based on reasoning and evidence. This is based on many inputs and their consilience within the framework. This is an ongoing enterprise and science is not the only contributor to it. But science based on naturalism is a very significant contributor, tethering knowledge to logical pedigree and empirical warrant.

Statistical Modeling, Causal Inference, and Social Science
Alan Sokal’s comments on “Abandon Statistical Significance”
Andrew Gelman | Professor of Statistics and Political Science and Director of the Applied Statistics Center, Columbia University

Also

true economics
Ambiguity Fun: Perceptions of Rationality?Constantin Gurdgiev | chairman of the Ireland-Russia Business Association, contributor and former editor of Business & Finance Magazine, and lecturer in Finance with Trinity College, Dublin

Wednesday, September 27, 2017

Lars P. Syll — Time to abandon statistical significance

As shown over and over again when significance tests are applied, people have a tendency to read ‘not disconfirmed’ as ‘probably confirmed.’ Standard scientific methodology tells us that when there is only say a 10 % probability that pure sampling error could account for the observed difference between the data and the null hypothesis, it would be more ‘reasonable’ to conclude that we have a case of disconfirmation. Especially if we perform many independent tests of our hypothesis and they all give about the same 10 % result as our reported one, I guess most researchers would count the hypothesis as even more disconfirmed.
We should never forget that the underlying parameters we use when performing significance tests are model constructions. Our p-values mean nothing if the model is wrong. And most importantly — statistical significance tests DO NOT validate models!
Lars P. Syll’s Blog
Time to abandon statistical significance
Lars P. Syll | Professor, Malmo University

Sunday, September 3, 2017

Adam Shaw — Why economic forecasting has always been a flawed science

So how do you make good predictions? I met “superforecaster” Michael Story, who was ranked 18th best among the 20,000 people who formed the Good Judgment team. The team took part in a competition conducted by the US intelligence community to find the world’s best forecasters. Launched in 2011, the four-year contest required the group to provide forecasts on 500 questions ranging from the future for oil prices to the financial outlook. The Good Judgment team won the tournament, reportedly outperforming even professional intelligence analysts with access to classified data.
The grading of their volunteers’ forecasting abilities was key to why Good Judgment did so well. People knew how well they were performing and were driven to improve. They were also encouraged to correct their biases and alter their world view amid changing circumstances. This self-analysis and willingness to adapt, they believe, was crucial to the team’s success.…
Conclusion.
The Good Judgment team believes part of the problem is that we misunderstand the science of forecasting and look to the wrong people for predictions. If we want to know what’s happening to the economy, we think the obvious thing to do is ask an economist. But Storey says that may be the wrong approach. Forecasting is an art that is separate from the need to have specific subject knowledge. The people who were best at predicting the Arab spring, he said, were not Middle East experts. They were people who studied eastern Europe and had seen similar patterns develop there. We don’t need subject experts, we need people who are great at forecasting anything.
Adam Shaw

Thursday, August 17, 2017

Noah Smith — "Theory vs. Data" in statistics too


Important.

I think Noah has this right. Fit the tool to the job, rather than the job to the tool.

Aristotle defined speculative knowledge in terms of causal explanation. This definition stuck although Aristotle's analysis of causality did not.
In the Posterior Analytics, Aristotle places the following crucial condition on proper knowledge: we think we have knowledge of a thing only when we have grasped its cause (APost. 71 b 9–11. Cf. APost. 94 a 20). That proper knowledge is knowledge of the cause is repeated in the Physics: we think we do not have knowledge of a thing until we have grasped its why, that is to say, its cause (Phys. 194 b 17–20). Since Aristotle obviously conceives of a causal investigation as the search for an answer to the question “why?”, and a why-question is a request for an explanation, it can be useful to think of a cause as a certain type of explanation. (My hesitation is ultimately due to the fact that not all why-questions are requests for an explanation that identifies a cause, let alone a cause in the particular sense envisioned by Aristotle.) — Stanford Encyclopedia of Philosophy
There is a distinction between reasons and causes. Some types of explanation seek only reasons, while other seek causes. Causation subsequently came to be viewed in terms of articulating mechanisms or lines of transmission (models) that are substantiated in evidence.

Explanation by reasons is different since the strict criterion of articulating mechanisms or lines of transmission that can be checked against evidence is not required.

Explanation by reasons rather than strictly by establishing causation is based on the principle of sufficient reason, which is usually credited to Spinoza and Leibnitz.

In philosophical logic, two negative criteria are foundational. Valid reasoning is vitiated by 1) arguing in a circle and 2) infinite regress.

Without recourse to checking against evidence there is no stopping point in assigning causes other than stipulation, e.g. of a first cause.

However, there may be a reason for a stopping point that doesn't involve causality based on evidence from observation or only stipulation, for example, principles that are "self-evident" based on intuition such as Aristotle's conception of intellectual intuition, or Kant's synthetic a priori propositions as articulated in the Critique of Pure Reason

On the other hand, Hume argued that causality is merely over-interpretation of constant correlation, there being no knowledge of the world other than that based on sense data. There is no observable causal link.

Cutting to the chase, scientific explanation based on causality is grounded in models that articulate causal mechanisms or lines of transmission that show how things change invariantly, which is the basis for deterministic functions. Where this is not possible, then there are two other avenues. The first is explanation by giving reasons, which is the domain of speculative philosophy. The second is employing statistics to explore patters of correlation. The question then is to what degree causal models can be gained from statistical methods, or whether it is possible at all. 

This is the issue that Noah Smith's post is getting at.

Noahpinion
"Theory vs. Data" in statistics too
Noah Smith | Bloomberg View columnist

Tuesday, July 18, 2017

Chris Dillow — Facts, frictions & "mainstream" economics


A major problem with conventional economics is that it is like doing physics without taking friction into account. OK for creating simple teaching models to illustrate fundamentals, maybe. But disastrous in doing advanced theory and especially in applications like engineering.

There are a lot of inefficiencies in economic behavior that are difficult to measure and very difficult to reduce in a cost-effective way, such as transaction cost. Ignoring these factors or pretending that they don't contribute substantially to results can render modeling quite non-representational when predictions are compared to evidence.

And this is in addition to "theonomic" assumptions!

Stumbling and Mumbling
Facts, frictions & "mainstream" economics
Chris Dillow | Investors Chronicle

Saturday, July 15, 2017

Lars P. Syll — Why testing axioms is necessary in economics

Where do axioms come from and how are they tested?

Axioms are starting points of deductive systems. They are stipulations.

Axioms must avoid the traps of illogic, circularity, and infinite regress.

Axioms in scientific theories are assumptions derived either from induction as generalization from experience (data) or abduction in C. S. Peirce's sense as discovery through "educated guessing."

The objective of scientific inquiry is not to "prove" axioms, since axioms are the basis of proof in a logical system. Axioms function as criteria for syntactical or logical truth, also called "necessity."

Axioms in hypothetical-deductive systems provide the basis for the scientific method. 

Assumptions of representational models stand or fall with the testing of models they are used to construct. 

Scientific models are used to generate hypotheses that can be tested empirically by deriving hypotheses as theorems from the assumptions that serve as axioms for the system.

Failure of a hypothesis as a theorem of deductive system reveals semantic inconsistency or incompleteness and calls the system into question as a coherent explanation of the data. 

This also calls the explanatory model into question as providing a causal explanation based on a representational model purporting to show causal transmission.

Correlation is not causation. Without a theory explicating transmission, there is no properly scientific explanation.

Lars P. Syll’s Blog
Why testing axioms is necessary in economics
Lars P. Syll | Professor, Malmo University

Friday, June 16, 2017

Lars P. Syll — What is it that DSGE models — really — explain?

‘Rigorous’ and ‘precise’ DSGE models cannot be considered anything else than unsubstantiated conjectures as long as they aren’t supported by evidence from outside the theory or model. To my knowledge no in any way decisive empirical evidence has been presented.
No matter how precise and rigorous the analysis, and no matter how hard one tries to cast the argument in modern mathematical form, they do not push economic science forwards one single millimeter if they do not stand the acid test of relevance to the target. No matter how clear, precise, rigorous or certain the inferences delivered inside these models are, they do not say anything about real world economies.
Proving things ‘rigorously’ in DSGE models is at most a starting-point for doing an interesting and relevant economic analysis. Forgetting to supply export warrants to the real world makes the analysis an empty exercise in formalism without real scientific value.
Mainstream economists think there is a gain from the DSGE style of modeling in its capacity to offer some kind of structure around which to organise discussions. To me that sounds more like a religious theoretical-methodological dogma, where one paradigm rules in divine hegemony. That’s not progress. That’s the death of economics as a science.
Lars P. Syll’s Blog
What is it that DSGE models — really — explain?
Lars P. Syll | Professor, Malmo University

Wednesday, June 14, 2017

Heiner Flassbeck — Are Keynesianism and Neoclassical economics antipodes?

Over time, it has become abundantly clear that Keynesians made a major strategic mistake to consider neoclassical economics as a scientific counterpart that has to be opposed. Instead of characterizing neoclassical economics had from the very outset as a normative structure, which serves no scientific purpose, they took it on as a science. But no science can ever win the confrontation with an ideological superstructure. The neoclassical attempt is not directed to make genuine scientific progress, but to defend its own position at all costs and even if this cost comes with the high price of inconsistency no one cares. The “general theory,” which Keynes attempted to write, was not ‚general‘ because neoclassicism was not a special brand of economic theory, but a normative structure.
Flassbeck argues that neoclassical economics has an ideological bias toward a government-free market based economy and creates a narrative that pictures a "natural" economy based on supply and demand in markets as one without government. Their solution to market failure is therefore to blame government intrusion and to seek to reduce the role of government. Flassbeck observes that this has no empirical basis and is completely ideological.

A problem with many "Keynesians, " e.g., New Keynesians, is that they have acknowledged the scientific basis of neoclassical economics, where there is none, and tried to shape their views in reaction but within the same flawed framework. This, too, results in pseudoscience.

flassbeck economics international
Are Keynesianism and Neoclassical economics antipodes?
Heiner Flassbeck | Director of Flassbeck-Economics

Tuesday, March 14, 2017

Jason Smith — How do you know if you're researching in bad faith? A handy checklist.


10 points checklist.

If you don't follow it, then you either don't know what  you are doing as a scientist, or you are pushing an agenda and not disclosing it, or you are fooling yourself.

Compare Richard Feynman:
 The first principle is that you must not fool yourself — and you are the easiest person to fool. —  adapted from a 1974 Caltech commencement address; also published in Surely You're Joking, Mr. Feynman!, p. 343
We've learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature's phenomena will agree or they'll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven't tried to be very careful in this kind of work. And it's this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.  "Cargo Cult Science", adapted from a 1974 Caltech commencement address; also published in Surely You're Joking, Mr. Feynman!, p. 342
All experiments in psychology are not of this [cargo cult] type, however. For example there have been many experiments running rats through all kinds of mazes, and so on — with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train rats to go to the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.
The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell.
Then he thought maybe they were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell.
He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go to the third door. If he relaxed any of his conditions, the rats could tell.
Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers the clues that the rat is really using — not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running.
I looked into the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or of being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of cargo cult science. — "Cargo Cult Science", adapted from a 1974 Caltech commencement address; also published in Surely You're Joking, Mr. Feynman!, p. 345

Monday, October 12, 2015

Jason Smith — Noah is stealing my material


More on econ as science.

Noah says to prove the assumptions are wrong. Lars says show that they are correct. Who has the better case?

The issue is how well a theoretical model works. Assumptions are always simplifications for economy and tractability of explanation. 

There is nothing inherently wrong about assumptions not being precise. They only need to be precise enough to yield results within an acceptable degree of tolerance.

The proof of the pudding is though hypothesis testing more than verifying assumptions, although if assumptions are not reasonably correct, then the model is questionable as an explanation that is generalizable. 

It is possible that a dodgy model can sometime yield positive results (pace Friedman's instrumentalism), as broken clock is correct twice a day. The test of theory is how well the model performs as an explanation of how things stand over time, that is, taking change into account. Theoretical models that don't reliably predict as not generalizable explanations. They are only generalizable in terms of restrictive assumptions that may or not hold in specific cases. That is to say they are models of special cases. 

I would say that scientific method is applicable in econ as it is in other social sciences and also in philosophy, since even speculation must take established truths into account. The question really is whether econ is a natural science like physics, a hybrid science positioned between natural science and life and social sciences, a narrative explanation of occurrences like history, or speculation based on principles grounded in intuition, like speculative philosophy. 

I would say that econ as practiced is some of each, and all approaches make their own contributions to the field. Defenders of econ as science can cite examples to make their case, and opponents can do likewise.

Jason Smith has written on this previously, to which I have linked here at MNE.






There are more posts there, but these are representative.

The title,  Is human agency Noah's big unchallenged assumption?,  hits the nail on the head. The social sciences are about human agency, and so are behavioral psychology and motivational psychology and some other branches of psychology. So is theory of history. And in philosophy, so are theory of man, ethics, theory of action, and social and political philosophy.

Economics is based on a theory of man and theory of action. The theory of man and of human action are not subjects of study in the field of economics, or at least not exclusively so (pace Ludwig von Mises).

There is no agreed upon general theory of man or of human action in either the sciences or philosophy. Why? Foundational disagreement is often due to lack of criteria that are agreed upon, or failure of agreed upon criteria to determine a definitive result. It is also possible that data or method are insufficient. 

As a result, general agreement is usually limited to quite specific cases under particular conditions that are not generalizable, that is, special cases. As a result, the case method is generally used in business schools, for instance, that than theoretical economics.

Information Transfer Economics
Noah is stealing my material
Jason Smith