|
Process Excellence News
Tips, tricks, and advice from the Pyzdek Institute |
|
10 Steps to Build a Crystal Ball 
Building regression models is a tricky business, filled with traps and pitfalls for the unwary. Read on to learn how to navigate around the hazards and build crystal ball models that will help your employer or client see their future of process excellence.
|
10 Steps to a Crystal Ball
"Forecasting is not a respectable human activity, and not worthwhile beyond the shortest periods."
Peter Drucker
The gold standard for modeling the future in a business environment is the designed experiment. Design of Experiments (DoE) is a well developed approach to planning and executing controlled manipulations of the world, collecting data, and using the data to build models that we can use to make changes that improve some aspect of performance. DoEs yield nice, "clean" statistical estimates of model parameters. I love DoEs.
Somewhat less respectable are models derived from historical data. The logic is: hey, we spent a lot of money to get this data, let's get as much information as possible from it. It makes sense, but using historical data requires caution. Here are a few of the problems one might encounter:
- Measurement error. Historical data are often recorded by a variety of people with varying levels of training, experience, and commitment to data accuracy. Or the precision required for day to day use of the data may be wide compared to what you need for modeling. Data errors that may be of little importance when the data are used for their originally intended purpose, may wreak havoc on your model building activity.
- Range restriction. Operational systems are deliberately controlled to minimize the impact of system variation on results. This means that the amount by which system parameters are allowed to vary is limited to a very small range. It is very possible that the response we are modeling will not be effected by variation of inputs in this range, but that doesn't mean that the responses wouldn't change if the inputs were varied over a larger range. The result is a model that gives misleading results by excluding important parameters.
- Failure to observe lurking variables. It is possible that an important variable that is driving one of the inputs is not recorded because it doesn't matter to the process operator. But that doesn't mean that it doesn't matter to you. Unfortunately, with historical data it is often impossible to know what this variable might be.
- Colinearity. Historical data often contain many different variables that actually measure the same underlying driver. For example, a call center might have data on queue size, abandon rate, wait time, etc. which all move in the same direction as arrivals, utilization and other drivers vary. If these variables are included in the same model, the model's parameter estimates will vary wildly. This can cause a variety of serious problems, including models that are patently silly. The embarrassment can be memorable.
I could go on, but you get the idea. Still, despite these concerns, people will continue to use historical data to build predictive models. This includes my clients and the Six Sigma Belts that I train. When I'm asked to help build a linear regression model from historical data I recommend DoEs if possible. If that's not possible I judiciously point out the pitfalls, then suggest the following:
- Select a continuous response variable (Y) to be modeled (we'll leave responses on other measurement scales for another column.)
- Choose a set of predictor variables. Choose Xs that subject matter expertise suggests might cause a change in Y.
- Create a regression model that includes all Xs. Be sure the analysis includes the Variance Inflation Factors. VIFs measure the correlation between each X in the model and the other Xs in the model.
- Eliminate any variables with a VIF greater than 10. A VIF of 10 means that 90% of the X is already explained by the other Xs in the model.
- Calculate a new regression with the remaining Xs and repeat the process of dropping Xs with high VIFs. Repeat the process until all remaining VIFs are less than 10.
- Using software such as Minitab, perform a best subsets regression with the remaining variables.
- Select the best subset model using the following criteria: Cp ≤ p+1, where p is the number of predictors; Cp minimized is a criterion that is often used; the standard error should be small, and R-square should be large. Any variable with a P-value greater than 0.05, or with the wrong sign, should be discarded.
- Occam's razor is the philosophy that one should not increase, beyond what is necessary, the number of entities required to explain anything. In this case, applying Occam's razor means that we will use as few predictors as possible. We won't add variables just to get a small improvement in R-square or the standard error.
- Assess the quality of the fitted model. Look at the residuals to see if there are patterns, lack of normality, or outliers. If justified, remove the cases responsible for the problems. Use a procedure to identify influential observations. Minitab's DFITS metric is a good way to do this. DFITS represents roughly the number of estimated standard deviations that the fitted value changes when the ith observation is removed from the data. An easy way to compare DFITS is to graph the DFITS values using boxplots, then look for extreme values on the boxplot chart. Brush these values to identify the cases responsible and consider dropping them.
- If a transformation was used, convert the predictions back to the original units and compare them to the actual values.
Models built with this procedure won't be perfect, but as G.E.P. Box says, "All models are wrong. Some models are useful." Perhaps, that's the true gold standard.
|
|
|
|
|
Featured Article
What would your employer or clients think if you told them that you could create a crystal ball that would show them precisely what to change to improve any metric they chose? If you apply regression properly, that's precisely what you can do!
|
|
