Chamber Connection

EIGHT WAYS TO CRITICALLY EVALUATE A FORECAST

Anticipating future conditions and requirements are keys to future success. At the same time, forecasts are at best a patchy guide to the world of tomorrow. They are often wide of the mark, sometimes absurdly so. It's never hard to get a view of the future. What's hard is knowing whether it's correct. Given that forecasts are necessary but usually of low quality, managers need to assert a strong independence of mind to probe whether any forecast is valid, and if so, how valid? By asking tough questions we can get to the value in a forecast and avoid being duped by bias, cowed by expert credentials or bamboozled by fancy software.

Some key questions to ask include:

1. What is the purpose of the forecast? What can be gleaned about why its exists, who put it out or what the intention of the forecaster was? Is the forecast upfront about its purpose? All forecasting is done for benefit. By recognizing the interests at work behind a forecast, assessing what benefit or benefits are sought by the forecaster or whoever commissioned it, one can make a better judgment as to potential strengths and weaknesses. We may ask: What action or concerns is the forecast trying to arouse? How is it legitimating a view that the forecaster or forecast organization holds?

2. Is the forecast telling you what will happen or trying to make something happen? Forecasts fall into two main categories: future-aligning, where forecasters anticipate change in order to adapt early and successfully to it; or future-influencing, where forecasters are trying to influence events. Future-aligning approaches aim to be objective. They mail fail, but the intention is there. So on balance, this approach will be more accurate. Future-influencing forecasts aim to succeed on other terms -- alerting and shaping opinion, changing minds, and harnessing action. Forecasts that are trying to lobby or change industry conditions make themselves known by seeking publicity and often being a forecast of extreme optimistic or pessimistic outcomes (to be aspired to or negated).

3. Is there too much certainty? Certainty is a warning sign. In short-term situations or closed systems with few variables, the attempt to pinpoint outcomes is reasonable. But the forecast consumer should consider claims to medium- and long-term accuracy with acute skepticism. Long-range detailed point predictions are almost always worthless. Beyond the short term the world is complex and effectively unpredictable. Anyone who says they can predict it is announcing themselves as a charlatan.

4. Does the forecast rely on "experts?" Experts are necessary in a specialized world, and expertise and credentials are important in forecasting, but experts are wrong as much as anyone. This is because a field's experts are particularly likely to be heavily invested in the status quo and have expertise in its existing procedures, attitudes and prejudices. Change often comes from outside, and experts -- blinkered by their knowledge of today -- are often the last to see it.

5. Are blocking forces identified and fully accounted for? All drivers of change work against the frictional resistance of the status quo -- the systems and solutions that people are currently invested in and comfortable with. They also face direct "blockers" and "turners," which are forces that have a vested interest in the status quo and don't want to see change, or that have an interest in another type of change. A good forecast will assess the strength of resistance to change and anticipate specifically if and how this resistance will be overcome, if indeed it will be, and account for the resources required to achieve this. Rather than running with the breathless wow-of-the-new, the forecast will display a measured pragmatism in the face of constraints, and adjust the forecast directions and/or timing accordingly.

6. Is a machine doing the thinking? A forecast takes us from present conditions to future outcomes. In every case there is a method for getting from the present to the future. A good forecast will state its primary methods, including the limitation and biases thereof. The author will show his or her working, revealing a train of logic that one can follow. This does not imply that highly methodological forecasts are better. Often formal methods give the illusion of process when there is none or, worse, spurious process. Many forecasts are overloaded with method and acrobatic computation, but short on basic insight and common sense. The existence or claim of complex methods and fancy analysis is in itself a warning sign of a bad forecast. The forecaster may be oblivious to -- or intentionally obscuring -- the vulnerability of the model's key assumptions with bewitching graphics and fancy math.

7. Is the data real or a projection? Data is never as solid as it seems. Among the problems are validity of definitions, validity of sampling, how research is skewed by the form of questioning, and so on. A particular problem in forecasting is that sometimes data points used in discussion are not real recorded figures but "future" data points that have been projected from past data, which raises obvious questions about how this projections has been done and how valid the process is. A good forecast will carefully distinguish real data from projected data.

8. Is the forecast jumping on the bandwagon? Generally, it is better if many sources are saying the same thing or making the same forecast. Corroboration is reassuring. However, while consensus is good, it is not infallible, and consensus-based forecasts are particularly vulnerable to a bandwagon effect or "groupthink," where everyone predicts the same thing because everyone else is. A good forecast will not easily be sucked into the prevailing wisdom and will question general consensus before buying into it.

Source: "Industry Week" Magazine

April 7, 2010

article by Adam Fordon, the author of "FutureSavvy"