Thursday, December 29, 2016

Kira Radinsky and Yoni Acriche — How to Make Better Predictions When You Don’t Have Enough Data


"Transfer learning. "
.. in order to stay relevant, statisticians will have to get out of the purist position of fitting models that are based solely on direct historical data, and to enrich their models with recent data from similar domains that could better capture current trends. 
This is known as Transfer Learning, a field that helps to solve these problems by offering a set of algorithms that identify the areas of knowledge which are “transferable” to the target domain. This broader set of data can then be used to help “train” the model. These algorithms identify the commonalities between the target task, recent tasks, previous tasks, and similar-but-not-the-same tasks. Thus, they help guide the algorithm to learn only from the relevant parts of the data.
This has been used by military strategist from time immemorial. No battle is the same but familiarity with past operations and outcomes is needed to provide a structure for thinking about present cases by identifying similar patterns, for example terrain. Military strategists have incredibly detailed knowledge of historical operations extending back millennia.

There is a similar field in logic called fuzzy logic.

Bayesianism is a similar technique in statistics.

These are rigorous approaches to heuristics that are appropriate when it is neither practical or possible to do more rigorous analysis. 

Heuristics may be superior to rigorous analysis is time-critical situations, for example. There may also be transaction costs that make more rigorous analysis impractical.

As always, it is a matter of choosing the right tool for the situation and using the tool skillfully. The executives that can do this efficiently and effectively are the most highly paid, and to a great degree it accounts for outsized CEO compensation as long as results corroborate.

Richard Feynman and Enrico Fermi were known as exceptionally good at heuristics. At the time of the testing of the first atomic bomb, the physicists present were, of course, intent on determining the yield and had in fact made bets on it. Enrico Fermi threw some pieces of paper in the air simultaneously with the blast and watched their behavior to estimate the yield. He was remarkably close using this heuristic device. Obviously, a lot of background went into that.

These proto-algorithms are held in the brain as what Michael Polanyi, Karl's equally brilliant brother, called "tacit knowledge." This is a basis for "intuition" as "educated guessing."

The take-away is that there is no sharp distinction between heuristic thinking and rigorous thinking. They generally overlap. Focusing on the contrast between them leads to a false dichotomy. They are complementary modes of thought that blend as they approach each other and more data becomes available.

For example, engineers regularly begin with heuristics, first to estimate the problem and its solution, and secondly, to provide a reference against which to judge the results of more rigorous thinking. If rigor thinking produces a significantly different result from previous estimates, then the solution has to be checked closely to see which went off the rails and how.

Harvard Business Review
How to Make Better Predictions When You Don’t Have Enough Data
Kira Radinsky and Yoni Acriche

17 comments:

Peter Pan said...

You see the gypsy lady in a tent with a crystal ball.

Alternatively, you can give lots of money to a "consultant". When a prediction is correct, they're lauded as brilliant. When a prediction is wrong, there are no refunds. Great business if you can get it.

I don't think the methods described in this article should be conflated with predictions, when their scope is limited. A cost-benefit analysis that is limited in scope can be practical. A more in depth CB analysis can be a waste of time and resources. The time frame from conception to result is also important.

In a word, actual "predictions" are a crapshoot and should be defined as such.

Tom Hickey said...

"Prediction" is a fuzzy word. The soothsayer analogy is apt. Many people prefer "forecast" to "prediction" for this reason.

But again, I don't think it is either-or. "Prediction" has a variety of meanings based on the context. Obviously there is a lot of range between fortune telling and rigorous analysis based on ample data.

Decision-making under uncertainty is often necessary but obviously limited.

Some would say that without rigorous analysis, it's better just to forget it. But that is not practical when there's a lot on the line, as in military ops and business decisions.

Peter Pan said...

A weather forecast is different to a climate forecast. A farmer can make an investment based on the relatively short term weather forecast. A country concerned about it's agricultural production might choose a contingency plan based on the longer term climate forecast.

To invest money on a prediction is equivalent to spending it on entertainment. This HBS article may be attempting to sell something under false pretenses.

Military ops and business decisions are exceptions to the rule of blithely ignoring the future. You'd think that more important issues would receive the same consideration and analysis. But no, our priorities are focused on optimizing our ability to wage war and make money. And so that is where the hucksters pitch their wares.

Matt Franko said...

If you study derivative action of a known mathematical function you can make predictions .....

Peter Pan said...

.... and confirm the result every time or some of the time?

Matt Franko said...

If its a known empirical function that work has already been done and you can expect it...

Matt Franko said...

Apply a known force to a known mass for a known time and you can predict its velocity, etc...

Its the only way we can make predictions...

Peter Pan said...

Predictions involve uncertainty. You're describing theory, acquired through the process of experimentation and observation.

Peter Pan said...

Why do high frequency circuits have potentiometers in them?

Matt Franko said...

Variations in the earths magnetic fields? Creating variations in the impedance faced by the source in that frequency band?

We dont have authority over the earths magnetic fields... they are somewhat stochastic in appearance... so we have to plan accordingly in that case...

Matt Franko said...

"Predictions involve uncertainty."

If the uncertainty is very small we can disregard it... this is "tolerance" ...

https://en.wikipedia.org/wiki/Engineering_tolerance

Matt Franko said...

And where we have absolute authority, uncertainty is ZERO...

Some might look at NAIRU as a form of "tolerance"... this is a BIIIIIG mistake... we have absolute authority over our economics....

Matt Franko said...

Prediction: govt provides $1T xfer payments as taxable income at 25% >>>> taxes will be $250B...

Lock it....

Peter Pan said...

Because the component model breaks down with HF applications. RLC elements are distributed throughout the circuit and interact with each other.

Where the uncertainty is small, tolerance is acceptable. If not, pots can be added and adjusted until an acceptable tolerance is achieved.

Unpredictability = bad

A theory that accurately describes reality can provide authority. This is one of the holy grails of science. Put the tea leave readers out of business. Cut down or eliminate the need for potentiometers. It means more money in my pocket.

Tom Hickey said...

Causal explanation and predication (predictability) are about regularity. Science is the search for invariance as strict regularity. A function is an expression of invariance. Functions are defined as Inputs yielding single outputs. The input and output are expressed as variables, and the arguments (value of independent variable) are as precise as measurement allows. This determines the precision of the value of the output.

At the classical level of natural science equations are deterministic, but at the quantum level stochastic (wave function).

But even at the classical level, the level of determinism is limited by the ability to measure data. Variability as lack of regularity can be introduced by limitations on measurement.

So there is a range of predictability with "fortune telling" or guessing at one end and determinism at the other. A lot of what is interesting and important in life lies along the range. So human do their best to deal with this limitation on foreknowledge using different stratagems and tools, some better than others.

Less risk-appetitive people rush in where angels fear to tread, while risk-adverse people hold out for certainty. This attitudes result in different contexts for the application of "prediction."

Matt Franko said...

yeah but Tom these people are going all around saying shit like "we're out of money!" so they are miles from any of this type thinking...

they dont see any deterministic aspects to any of this at all... its a big problem...

and another thing there is this scripture:

"The poor you will always have with you, and you can help them any time you want." Mark 14:7

I'll make another guaranteed prediction here from this... call it NAIRU or call it wtf... we are going to be stuck with this "poor" cohort among us as long as these morons keep looking at all of this as non-deterministic.... lock it...

Peter Pan said...

Unfortunately the NAIRU isn't locked. It can be at whatever rate you want it to be.