| Comptlimentary eBook Featuring Predictive Evaluation |  | |
Learn more about PE and how it will improve your training results. This complimentary
|
|
|
 | Evaluating Interventions with No Clear Beginning and End | |
Recently I received the following question: How do you approach evaluating the impact of interventions that are not necessarily an 'event' with a clear beginning and end? For example, ongoing coaching; or an internal online community designed for a group of employees (e.g., project managers, or regional sales managers, or research engineers, or... etc.) to help them connect and collaborate with others and share expertise and resources to improve performance; or self-serve online resources or tools or tutorials or learnlets (e.g., video clips, podcasts, etc.) that folks might use on their own as needed for just in time performance support?
Answer:I have evaluated learning systems like this before that do not have a clear start/stop points. What I did was evaluate adoption and impact in a "slice of time" such as the 1st quarter. I set my predictions around:
- What activity was hoped for from the participants
- What adoptive behaviors would manifest themselves as a result of the activities
- What results (impact) should the company expect
I then collected activity during the time slice and adoption/impact data immediately after the time slice. I was then able to report on activity (what employees were using, what if any had they employed at work, and what if any impact was realized). I then was able to forecast what future value along with the cost could be realized. In one case, the support system was cancelled due to lack of activity, adoption, and impact. |
 | The Predictive Evaluation Framework for Existing Courses | |
The Predictive Evaluation approach is very attractive to those executives who want to know specifically what business results and the organizational impact training can be expected to provide. Predictive Evaluation is a four-step model: Predicting, Intention Evaluation, Adoption Evaluation, and Impact Evaluation.
The Predicting element of the model is a form of business forecast that predicts the future as accurately as possible, given all the information available, including historical data and knowledge of any future events that might impact the forecast. PE not only forecasts value, but it is also a monitor and control mechanism that involves three stages. The first stage is focusing on the right metrics for evaluating training. You could monitor many things, but you want to focus on those that impact the business. I suggest measuring Intention, Adoption, and Impact.
The second stage is implementing steering control. When you're thinking about performance from training and your delivery period starts on January 1 and ends on December 31, you don't want to wait until December 31 to figure out that you've got a problem and then make some changes; you want to solve that problem as early as you can, by using PE results. As steering controls, Intention data are leading indicators to Adoption, and Adoption data are leading indicators to Impact. It's just like steering: we make changes when we know we've got a problem.
The third stage is using the right performance measures. We need to make sure that we've got appropriate measures by predicting training's Intention, Adoption, and Impact before we train. |
| How is Predictive Evaluation different from other approaches? (part 3 of 4) | |
Typical Approach
ROI and/or Cost-Benefit evaluation. ROI evaluation, when it is attempted by trainers, often over-relies on subjective estimates of the percentage of return. An ROI figure has little value for making decisions about the program. Cost-benefit evaluation requires significant use of statistics to provide useful data. Both types of evaluation are conducted well after training has concluded and do not provide data that can be used to improve the program in real time.
Predictive Evaluation (PE) Provides concrete, business-focused and evidence-based data on return on investment. Data collection is robust and rigorous, and management finds the data far more compelling, convincing, and useful. |
| Advance Praise for Predictive Evaluation
|  "Dave provides clear and easy-to-follow step-by-step instructions on how to make sure that all your good Needs Analysis and effective Instructional Design are put to use in service of having the desired impact on your organization. He describes specific processes to be sure that the right data is uncovered, the details are specified and external factors impacting value are recognized and considered. He provides detailed guidelines on how to collect and analyze data related to adoption of new skills, knowledge and behavior. This book will help anyone struggling to show the value of training, before the training has been implemented and after."
Christy Pines, Manager, Learning Services Cannon USA
Corporate Human Resources Division
|
|
|
|
|
