learnability matters

Kern has evolved a three-step learning evaluation methodology. Unlike Kirkpatrick’s model which considers evaluation as the final step in the training development process, Kern’s evaluation model is a continuous and ongoing process. This ensures that training is continuously revised, refined, and the results of evaluation are ploughed back to make the training program more robust. The objectives of evaluation at Kern are as follows:

• How well does the training map to the needs of learners?
• Does the training meet its goals, objectives, and learning outcomes?
• How well is the training working?
• How does the training impact the skills and behaviors of learners at their workplace?

A quick look at the phases in Kern’s development process that has the evaluation mechanisms built in:

Phases in the Development Process

Levels of Evaluation

• Level 1: Expert Evaluation of Training Methods and Cognitive Load
• Level 2: Training Evaluation with Learner Participation
• Level 3: Impact Evaluation with Learner and Stakeholder Participation

Let us discuss each level in detail.

Level 1: Expert Evaluation of Training Methods and Cognitive Load

Objective of Evaluation: The objective of evaluation at this level is to map the in-progress program with the findings of learner observation done earlier. At this level, specific issues like teaching methodologies and learning strategies are evaluated, the content scope and coverage is validated, and navigation and other usability issues are tested to ensure that they map to the profiles and learning objectives. It evaluates the overall effectiveness of the training program with respect to the defined learning objectives. The gaps are filled in to ensure that the program maps to the learners’ profiles accurately.

Evaluation Mechanism: This is also called the expert review or cognitive walkthrough. The expert team uses heuristics-based evaluation or scenario-based evaluation to evaluate the course. The development team and the subject matter experts are also part of the evaluation process. They evaluate the course independently and then have open discussions mapping the training methods/learning strategies used to the personas created for the program. The issues marked out are debated and collated so that the program design maps best to the learner requirements. Stakeholder participation at this stage also adds valuable inputs to the discussion.

Level 2: Training Evaluation with Learner Participation
Objective of Evaluation: Typically, learners feel that they are the subject of evaluation rather than the curriculum. However, we believe that learners can be active evaluators of the training programs.

Evaluation Mechanism: At this level, a minimum of 7 learners are recruited to go through the Alpha version of the program. The learners are asked to go through the program as they would in real life. The tester sits behind the learner and observes the movement of the learner thorough the program. The tester notes down the observation and also makes contextual inquiry in to understand the reason for certain actions taken by the learners. At this stage, testers distinguish between learners’ opinions and actions. The observations made for all learners are analyzed and the changes incorporated in the program.

Level 3: Impact Evaluation with Learner and Stakeholder Participation

Objective of Evaluation: The objective of this evaluation is to measure the achievement of specific learning outcomes, knowledge, skills, and attitudes/behaviors by the learners.
According to the findings of Research Institute of America, 33 minutes after completion of a course, students retain only 58 percent of the material covered in the class. By the second day, 33 percent is retained, and three weeks after the course, only 15 percent of the knowledge delivered is retained. See Figure 1 below.

Further studies by Neil Rackam also support these findings. They have proved that 87 percent of the learning from any given classroom workshop is lost within 30 days if not followed by either a coaching intervention or a refresher intervention with the participants’ manager.

However, research on blended learning have more encouraging results. If the learning is aligned to the learner’s real-world performance, there is a 55 percent improvement in the performance of learners. Similarly, if learning is supported by retrieval practice and assessments, learners show more than 100% improvement in performance. See the table below.

To improve the performance of learners further in the long run, the frequency of the refresher course can be determined. For example, it can be recommended that learners must go through the online modules twice in two months followed by once every three months.

Evaluation Mechanism: To evaluate the actual impact or learning or training on the knowledge, skills and behavior of learners, we design a metrics where we assign weightage to various learning parameters. Some of the tools that we use to record data for this metrics are as follows:

a. Pre-test and post-test mechanism integrated in the training program to check the knowledge levels of learners
b. Knowledge check questions (quiz) immediately at the end of the program and after a defined frequency (such as fifteen days) after taking the program
c. Checklists that project managers/supervisors can use daily or weekly to monitor the activities of learners
d. Skill analysis pre-training and post training for learners
e. Bimonthly and yearly analysis of performance appraisals of learners

Typically, the evaluation metrics is evolved in discussion with the stakeholders because evaluation happens in close coordination with the stakeholders.

If you follow an evaluation methodology in your organization, do share your thoughts.

Share this