learnability matters

Evaluation of training means measuring the effectiveness of a training program. Evaluation helps in defining the learning outcomes more sharply, remove unnecessary training content, and ensure that the training method meets the training needs of the learners and consequently of the business.

This effectiveness is as much about short-term retention as about the long-term retention and application by learners. Today most organizations talk about measuring the effectiveness of the training programs. While most traditional institutions have mechanisms in place where they measure the effectiveness of the classroom training programs, most modern day training programs are still not evaluated objectively or accurately.

There are several approaches used to measure the effectiveness of training. One popular approach is Kirkpatrick’s Four Levels of Evaluation. In this model, each successive evaluation level is built on information provided by the earlier lower level.

Let’s look at a brief description of each of the four levels and the respective criticisms.
Level 1 Evaluations: Reactions
As the name suggests, evaluation at this level captures the immediate reaction of the participants about the training. It assesses learners’ satisfaction with the training. While this is not an accurate measure of effectiveness, it certainly can be a filter for bad training. Positive reactions may or may not result in good learning but negative reactions can certainly be roadblocks to learning.

Criticism: Research shows that reactions should be observed and not asked. Kirkpatrick’s model has not defined the mechanisms of capturing the reactions. The model depends on the participants to provide this information, which is not a measure of the effectiveness. At best, it can be an indicator.

Level 2 Evaluations: Learning
At this level, evaluation moves beyond learner satisfaction to assess the learning that has happened in terms of acquisition of skills, knowledge, or attitude. Common methods used to test this are pretest and post test assessments to determine the amount of learning that has occurred.

Criticisms: Pre-assessment and post-assessment can capture knowledge and comprehension level skills. Also this is a measure of how effective the training is in transferring this information to the learner’s short-term memory. It does not measure the effectiveness of training on the learner’s cognitive skill, which will ultimately determine whether the learning will be applied at the workplace or not. In short, this phase does not measure learning; it only measures “rememberability”.

The other point of criticism is that at no point is there a measurement against the specified learning goals and outcome. There is no mechanism to compare the % of fulfillment of the specified learning goals and outcome.

Level 3 Evaluations: Transfer
This level is a critical measure of the amount of transfer of learning or amount of application of learning at the workplace. Evaluating at this level attempts to answer the question - Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner? According to Kirkpatrick, measuring at this level is difficult as it is often impossible to predict when the change in behavior will occur, and thus requires important decisions in terms of when to evaluate, how often to evaluate, and how to evaluate.

Criticisms: This is the most critical level of evaluation because it helps measure the most primary purpose of training. However, Krikpatrick’s model does not specify the mechanisms of capturing and analyzing the information obtained in this level. While it is impossible to predict a change in behavior post training; information about learner behavior should have been captured during learner analysis (pre-training) so that the training is designed keeping in mind the said behavior. Once, this variable is factored in the evaluation metrics, evaluating the change in behavior or acquisition or transfer of skill will be much more effective and accurate.

Krikpatrick’s model also ignores the other dependencies at this level of evaluation, such as “who will measure this information, what are the interventions needed, when to intervene, and what happens if intervention is delayed or absent…” The actual measurement of training effectiveness will depend on all these dependencies that are not defined in the Kirkpatrick’s model.

Level 4 Evaluations: Results
At this level, evaluation is done to measure the success of the program in terms that managers and executives can understand - increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, and even higher profits or return on investment. However, most organizations do not address this form of evaluation because they think determining results in financial terms is difficult to measure, and is hard to link directly with training.

Criticism: This evaluation level is the most critical test for business and it’s the least explained in Kirkpatrick’s. The major criticism against Kirkpatrick is that its fourth level does not actually estimate the business impact of the training. It does not provide the outcome as ROI. While it makes an effort to measure the success of the program from the perspectives of managers and executives, it actually does not evaluate how the program relates to the measures of business success.

There is an interesting article Alternatives for Measuring Learning Success by Kaliym Islam that talks about how with the advent of the electronic delivery of training programs and the increased capital investment required to develop these programs, the Kirkpatrick model has become obsolete. Some interesting read about criticisms of Kirkpatrick’s Models:
1) Developing a quantitative model to measure training effectiveness
2) Beyond Kirkpatrick: Measuring the Financial Returns of e-Learning
3) Alternatives for Measuring Learning Success

The next article in this series will talk about an evaluation methodology evolved by Kern. This methodology not only overcomes some of Kirkpatrick’s deficiencies but also talks about a paradigm shift in the way eLearning should look at evaluation.

Share this