learnability matters

Geeta Bose presented this paper on Learner Centered Methodology - A New Approach to Effective Learning at the 3rd National Seminar on ‘eLearning and eLEarning Technologies”, (ELELTECH India) November 5-6th, 2009. This seminar was organized by Center for Development of Advanced Computing (C-DAC), Hyderabad and JNT University Hyderabad in association with Department of Information Technology, Ministry of Communications and Information Technology Government of India.

Abstract— According to Learner Centered Methodology (LCM), all courses are learner-focused. They are designed to enable learners to achieve their learning outcomes and increase ROI of stakeholders. This methodology is based on proven research in human factors engineering, which has techniques, processes, and procedures to design user-centered products. This methodology has been successfully put to practice in many live projects.

Index Terms—Learner-centered methodology, learner testing, contextual inquiry, and evaluation.

I. INTRODUCTION

Most eLearning courses are content-centered rather than learner-centered. While instructional designers go through the various phases of audience analysis, content analysis, and requirements analysis, they seldom incorporate the outcomes of the analysis into the course design. In most cases, the motivations of stakeholders, technology constraints, and project deadlines take precedence over the learners’ needs in an eLearning program. Often learning strategies and pedagogy are driven by the type of content rather than by the needs of learners. The decisions to use case studies, themes, or particular instructional strategies are based on the type of content covered rather than the profile of learners. Consequently, the pedagogy is far removed from the learner’s motivation. In this scenario, an entry-level manager as well as a CEO may be subjected to the same eLearning program with neither benefiting from it.

Very few eLearning programs are actually aligned to the motivations of the learner from the analysis phase down to the micro design of the course. Let us understand the methodology in detail. To overcome this expectation mismatch, we have evolved the LCM methodology. We design and develop courseware based on the philosophy that the learner is at the center of the eLearning process. In this process, the development life cycle (DLC) has the learner as the focal point. Every step in the DLC derives from the study of learners.

II. PRINCIPLES OF LCM

A. Key Principles of LCM
LCM is an evolutionary approach. The final courseware is shaped over time. The instructional designers arrive at the optimum design through an iterative process of validating with the learning outcomes at all phases and steps in the eLearning development phase.

1. Learner is at the center of the development process.
2. LCM is an iterative process. Every phase involves iterations within the phase before becoming input for the next phase.
3. This is a diagnostic approach to eLearning, which is in contrast to a quick-fix approach.

B. Benefits to Stakeholders and Clients
Typical eLearning approach promises clients ROI by saving on the costs of classroom training, increasing productivity with just-in-time training, and saving on delivery time. This is only 25% of the potential gains that the client can derive from the LCM approach. An eLearning courseware designed with the LCM approach offers the following benefits:

1. Definite increase in the productivity of employees from day one
2. Reduction in peer-review time, rework, and supervision
3. Reduction in learning time on the job
4. Increase in first-time correct deliverables
5. Supervisors have time for other things 6. Longer and better retention

C. Phases in LCM Approach
LCM is an evolutionary approach. The final courseware is shaped over time. The instructional designers arrive at the optimum design through an iterative process of validating with the learning outcomes at all phases and steps in the eLearning development phase.

Various tests, checks, and validations at each phase of the DLC provide inputs that help designers understand learners’ goals, motivations, expectations, and learning behavior. Each test is a scientific approach that elicits information from learner. This information is ploughed back into the DLC to ensure that the focus of the courseware is always on the learner.

D. Influence of LCM on the Micro Design of the Course

Visual – The visuals, images, video, audio, and other media used are based on information derived about learners in the learner analysis phase. The use of these elements is evaluated while creating the high-level design document. The use of each visual element is derived from research about learner’s goal.

Writing Style – This is perhaps the single most element that is derived directly from learner analysis by most instructional designers. Written information that is part of elearning should be concise and to the point.

Interactive – An advantage of LCM is that design factors in the learners preferences and motivations. Learners can interact with the courseware through quizzes and multimedia activities that let them practice skills, demonstrate knowledge, discover relationships and new information, and reinforce learning.

Engaging – Appeal to the learner’s professional experience and their emotions. Use case-based learning and scenarios that get the learner to analyze and synthesize information and put it within a frame of reference. Ask the learner to make decisions or judgments relevant to the case-based learning and scenarios.

Relevant – E-learning should address a learner’s current needs or learning gaps. In addition, developers must maintain the accuracy of the content. Revisit elearning content on a regular basis to ensure that it is still relevant and accurate for your audience.

Feasible – Make sure that the technologies you choose for enabling elearning are feasible for your learners and their learning environments. If your audience is using 56K modems, be mindful of that limitation in choosing media and tools for creating your e-learning experience.

Empowering – Provide access to additional resources for more information so that the self-directed learner can explore content relevant to their interests and learn more.

III.    LCM – A CASE STUDY

Let’s understand the phases of LCM using one of our live project case studies.

Kern designed a training program on Grooming and Personality Development for the customer service executives (CSEs) of Godrej & Boyce for their Godrej Lifespace Stores across India. This was a program designed and developed using LCM. The crux of Kern’s LCM methodology is that the learner is at the centre of the development process. The training solutions are proposed only after analyzing learners’ needs and requirements. This course was a 2008 Brandon Hall Excellence in Learning Award winner.

Phase I: Learner Research
Kern began the project with a detailed learner analysis across 15 stores using contextual inquiry and ethnographic research methods. Kern visited Godrej’s Lifespace stores across 15 locations to observe and speak to the customer service executives (CSEs) – our eventual learners. We spent time observing them at work, gathering information about their demographics, speaking to their store chiefs, and gathering information about their work environment, their motivations, their aspirations, and expectations. This method of gathering data about learners is called contextual inquiry.

Contextual inquiry findings helped Kern arrive at the following:

1. Learning objectives of the program
2. Persona creation–a male CSE and a female CSE
3. Constraints in G&B’s existing training scenario (refer Appendix 1)
4. Understanding of the learners (graduates with average English speaking skills, proficient in local lanauges, possess good product knowledge but poor dressing and presentation skills)
5. Identify the learning challenges of learners (refer Appendix 2)
6. Define the evaluation metrics to map the learning outcome later
7. Define the ID review parameters based on the learner analysis

Phase 2: Design and Prototyping
Based on the contextual inquiry findings, Kern proposed the following blended solution to G&B:

1. A self-paced eLearning program for CSEs in their respective stores
2. A two-day workshop for storechiefs to help them:

• Implement the eLearning program in their stores
• Mentor the CSEs
• Monitor their performance
• Evaluate their performance

Supporting learning interventions in the stores such as one-on-one mentoring, posters and checklists, and the audio system in the store to help CSEs recall and remember information over longer duration.

An analysis of the contextual inquiry findings helped the instructional design team arrive at the following design directions for the self-paced program.

• The course has to be highly visual for quick comprehension.
• Clear demarcations for men and women as certain topics are only for women and men need not go through it and vice versa.
• The course content must be modular. A quick gain attention screens for every learning nugget to give the CSE a positive and optimistic feel.
• Option for audio in different languages e) Every nugget should be short and end with quizzes f) Two categories of content: Understand (knowledge and comprehension level) and Demonstrate (application level). Design two basic strategies for each.
• Indicate “current viewing” and “already viewed.” This needs to be checked with regard to development constraints.
• The learning nuggets on the Menu have to be self-explanatory so that the learner can quickly decide which topic they want to go through.

An instructional audit of the design document helped us map the instructional approaches to the learner analysis findings from the contextual inquiry. The training development team used review checklists to map the design with the findings of contextual inquiry. The development team and the subject matter experts (SME) mapped the training methods to the learner personae. The issues marked out in the checklist were debated and a consensus was reached so that the program design mapped most to the learner requirements. The design was then presented to the stakeholders who provided valuable inputs to the discussion and signed-off the design document.

Benefits: Training gaps were spotted well in advance. This helped reduce 80% of rework that typically happen when training is evaluated post delivery.

Phase 3: Development
Based on these design inputs, Kern designed a self-paced CD-based module for G&B. To meet the learning objectives, the course has 14 learning nuggets and each nugget does not exceed 10 minutes. Since both male and female CSEs had different learning needs, the course offered options to the learners to chose “Male” or “Female” to see their respective nuggets.

At every step in the process, QA of the storyboards involved instructional design reviews (ID) and SME reviews. ID reviews ensured that the content mapped to the learner’s needs, wants, and motivations. The SME review helped validate content accuracy and the language reviews ensured that the language was lucid and mapped to learner’s level of comprehension.

Keeping in mind the learning style of learners, the screens are more visual than textual. Also, most concepts are visually explained supplemented by video and a transcript of the audio. The course design was such that it could be easily translated into multiple languages.

Learner Testing
14 CSEs from the stores across 4 cities were recruited to go through the “learnability” testing (similar to usability testing) of the self-paced training program.

In learnability testing, the CSEs are asked to go through the program as they would in real life. The tester observes the learners as they go thorough the program. The tester notes down observations and also questions to understand certain actions taken by the learners. At this stage, testers clearly distinguish between learner’s opinions and learner’s actions.

The observations made are analyzed and changes incorporated in the self-paced program for a final release.

Benefits: Development team could fix 99% of the problems, bugs, and value-adds that typically emerge post final release. This saved the company valuable time and money that goes into fixing the courses post final release.

Since learner testing happens in a “real-like” environment, it helped the company test the course implementation mechanisms. Consequently, there were no hiccups during the final release.

Apart from course-related findings, learnability testing also threw up interesting results about the company’s processes and practices. These were valuable inputs for the company. For example, most learners were unhappy with the quality of uniforms, while some learners felt that uniforms violated their cultural sensitivity.

Phase 4: Implementation & Evaluation
The program had two components, online program and workshop. The workshop was designed to familiarize the store chiefs with the eLearning program so that later in the stores they could act as mentors for CSEs. During the workshop, store chiefs were provided handouts and checklists for evaluation and intervention. It trained the store chiefs on ways to implement training, observe CSEs, and intervene with corrective actions. During the workshop, the store chiefs learned how to use the evaluation checklist, conduct quiz and weekly discussion sessions. They also learned to use supporting training tools such as one-on-one mentoring, posters, checklists, and the audio system in the stores.

IV. LEARNABILITY EVALUATION & MEASURING LEARNING EFFECTIVENESS

Kern offered a payment-linked guarantee of effectiveness of the training program to G&B. Therefore, Kern and G&B agreed upon a set of evaluation parameters at the beginning of the program. The evaluation methodology adopted here attempts to measure the success of the program in terms of increased business benefits such as:

• Minimize rework
• Improve productivity
• Save time
• Save money

The tools used to evaluate the workshop, the self-paced training, and the complete training program, are listed below.

A. Workshop Evaluation
The trainer conducts written tests at the end of the workshop for store chiefs attending the workshop. This evaluates the knowledge-level skills of store chiefs.
The trainer conducts role-play competitions and group presentations at the end of the workshop for store chiefs to evaluate their behavioral skills.

B. Self-Paced Training Evaluation

• Pre-assessments included in the self-paced eLearning program
• Post-assessments included in the self-paced eLearning program

C. Complete Training Evaluation

• The store chief conducts quiz contests among the CSEs in the respective stores to test their immediate recall and retention of information one week after they have completed the program.
• The store chiefs conduct another quiz contest 15 days after they complete the training program to test their recall and retention of knowledge over a ittle longer duration.
• The store chiefs monitor and rate the behavioral skills of learners using an evaluation checklist (refer Appendix-3)
• One week after the implementation of training program in the respective stores, the store chiefs start monitoring the CSEs’ attitudes and skills. They rate the CSEs on the parameters defined using the evaluation checklist, which is online on their intranet.
• This evaluation happens again after 2 weeks. The store chiefs rate the CSEs and monitor their progress.
• They conduct one-on-one monitoring sessions with laggards and group sessions to felicitate high performers. This motivates other CSEs to perform better.
• The store chiefs again rate the CSEs after 25 days. This rating reflects the long-term retention of skills and behavior by learners.
• The final scores of each CSE is the summation of the store chief’s ratings and the individual quiz scores.

V. RESULTS OF THE TRAINING PROGRAM

Grooming

Customer Management

1) Benefits from the Training Program
The two Mystery Shoppers Audits conducted post-training implementation revealed that stores who had undergone the training had better groomed people. These stores sold better than the stores who had not undergone the training.

The Management Review Reports showed lower attrition in the stores that had undergone the Grooming and Personality Development training. The CSEs in these stores had higher job satisfaction and more pride in their work than the stores that had not undergone the training.

2) Change in Mindset towards Training
The workshop training helped store chiefs rehearse their mentoring role that they would play in their stores. It also helped store chiefs buy in to the actual benefits of the training program. The involvement of store chiefs in the actual training evaluation gave them a greater sense of ownership towards the training program. This helped them overcome their initial resistance to training.

3) Effective Learning
Before G&B followed the blended learning approach, 87% of learning was lost within 30 days of the classroom training conducted. With the blended learning approach, 80% of learners showed 90% improvement in retaining knowledge- level    information    about    Grooming    and    Personality Development skills. Post training, 78% learners showed 80% improvement in personality development skills.

4)  Increased Training Coverage
Earlier, G&B had 8-10 classroom training for new joiners in batch sizes of 12-15. With the blended learning approach, the training coverage increased manifold:

Now    it    had    to    conduct    only    one    classroom training/workshop for store chiefs at a regional level. Since each region has around 15 stores, one workshop helped train 15 store chiefs simultaneously.

Each store chief then implements the self-paced eLearning program in his/her respective stores. Therefore, about 75 CSEs in 15 stores are trained simultaneously with the training effort of one workshop.
Earlier, there was no scope for refresher training. With the self-paced training option, each store chief can recommend the CSEs for refresher training without depending on the corporate training calendar or sanction.

5)    Cost Saving
Earlier G&B incurred INR 300,000 per workshop. It could conduct at the most 10 such workshops for 150 CSEs in a year. CSEs had to travel from their base locations to the head office to attend these training. The stores had to incur an opportunity cost in their absence. While earlier workshops could train only a maximum of 150 CSEs in a year, the need was to train at least 700 CSEs per year.

With the blended learning approach, G&B incurs a one-time cost of INR 500,000 on the self-paced module and INR 300,000 on the workshop for store chiefs. Now the only recurring cost is the single workshop at a regional level for store chiefs and ZERO cost on training CSEs in stores using the self-paced training module.

VII. CONCLUSION
The Learner Centered Methodology can be successfully used to design learning solutions for various learning needs. In fact, using this methodology it is easy to evolve the most appropriate mode of delivery of training as well as the relevant evaluation mechanisms to be used to evaluate the training. We have used this methodology in designing pure eLeanring courses, in designing pure classroom training courses, as well as in designing blended learning solutions.

VIII. APPENDIX

A. Contextual Inquiry Findings - Constraints in G&B’s Old Training Scenario
During contextual inquiry, we realized that the biggest constraint in the existing training scenario was to train 1000s of CSEs simultaneously across the length and breadth of the country.

Not all new CSEs are able to attend the induction program as soon as they join. They can attend the induction program as per the training schedule and subject to vacancies in the batches. At times, CSEs find a slot in the training sessions after one year of joining the organization.

Due to the training constraint, most CSEs come to the shop floor without training. They take 3-4 months to become productive. However, the company ideally wants trained people on the shop floor to handle customers better.

1. Since the training calendar is finalized at the beginning of the financial year, there is no opportunity for need-based training or refresher training. However, most stores need refresher training.
2. G&B has aggressive plans to set up new stores across the country. However, the current training plan cannot support more than 8-10 training programs in a year.
3. Since there is no follow-up training, retention of learning is very low among the CSEs.

Apart from post assessments, there is no other evaluation mechanism. Hence, the business benefits from these training sessions are not measured.

B.  Contextual Inquiry Findings – Learning Challenges
Based on the contextual inquiry findings, the instructional design team discovered the following challenges:

1. Learner’ s    grasp    of    English    was    poor .    English    was needed to interact with the customers.
2. The training would be localized in the regional languages later.
3. Learners could only devote approximately 10 minutes learning time per sitting.
4. The training was supposed to bring about a behavioral change among learners. The team had to handle sensitive issues such as personal dress sense and personal hygiene in a multi- cultural scenario.
5. Motivation among learners was low as most of them are on contract basis and attrition rates are very high.
6. The team had to ensure that learners understand the important of grooming and personality development.
7. The team had to ensure that the store chiefs could play the role of a mentor.

C. Interventions by Store Chief

1. Rating each person as per the evaluation checklist between 5 to 7 days of training, 10-15 days, and 20-25 days
2. Group session within 1-2 days after the rating
3. Individual feedback from each person on whether they have understood the training, what are the major learning, and if there are any questions - through a group meeting
4. Recognize high scorers in the meeting , help others to learn more by demonstrating role play or asking questions
5. Give sensitive feedback individually - especially for areas where the rating is 2 or less
6. Chalk out individual action plan for improvement for these areas

REFERENCES
[1] Jeffery Rubin, “Handbook of Usability Testing”

[2] W. Allen Michael “Michael Allen’s Guide to e-Learning”