learnability matters

As promised, here’s the follow-up article on ‘Do you test your elearning courses?’. Before I proceed to explain how we conduct learnability testing at Kern, let me briefly explain the origins of this testing methodology. eLearning at Kern is highly influenced by usability. We apply design logic, contextual inquiry, usability testing, design principles typography, accessibility, and lots more to our eLearning courses. Learnability testing is inspired by usability testing. In simple words, usability testing is a controlled experiment that tests how well people use a particular product. The user performs a series of tasks while the observer watches him and takes notes. Usability testing is conducted at several stages and feedback implemented to refine the design. At Kern, we modified this testing methodology to test elearning courses. Where usability testing checks whether a product is ‘usable’, learnability testing checks whether an elearning application is ‘learnable’. The main objective of learnability testing is to remove all obstacles that hinder the learning experience.

Let us understand what ‘learnability’ is. A course is ‘learnable’ if it fulfills the following criteria:

• Is the learner motivated to finish the course? Is the course engaging?
• Do the instructional strategies interest the learner? Simply put, does the learner ‘get’ the theme?
• Is the instructional flow clear to the learner? Is the learner comfortable with the content?
• Is the course free of obstacles (too much audio, too much text, bugs, navigational errors and so on) that may hinder learning?

At Kern, we conduct a detailed ‘learnability test’ before the release of our final course. The process has three parts:

1. Testing Phase
2. Analysis and Results
3. Revision of Course

Testing Phase

During the testing phase, we decide the number of learners who need to go through the course (sample size) and recruit these learners. The learners go through the course in their actual work environment. Let us learn more about how this phase is carried out.

Introduction: We introduce ourselves to the learner and brief him/her about the objective of the inquiry process. We clearly state the expectations. This part is extremely important as it sets the stage for the rest of the evaluation process. A sample introduction is given below.

“My name is Archana and I work with Kern Communications Pvt. Ltd. I will show you the draft design of an elearning program that will help you learn grooming skills and personality development. I need your help to evaluate how well the course is designed.
We really appreciate your participation. As a token of appreciation, here is a small gift.

I will ask you to go through the course as if you are going through it as a part of your work. If you are able to go through it without hitches, we know that the course is designed well. However, if you face any trouble, we will know that design needs more work.

This is a test of the COURSE. And, it is NOT a test of YOUR abilities. So if you can’t proceed at any point of time, don’t worry. It just means the course can be designed better and needs more work.

I will just watch you from behind and I want you to complete the course. As we’re learning, please think out loud. You can take a short break in between if you want. Remember to tell me what you’re looking at, why you choose a button or link, and what you expect. Any questions? Let’s start.”

It is extremely important to let the learners know that we are not testing their abilities and that there is no pressure to take the test. If the learner is not interested in participating, he/she is free to leave. This helps put them at ease and they share their feedback freely. Our aim is to build a rapport with the learners and let them understand the importance of their feedback. We brief them on how the testing will be conducted and inform them that the entire session will be recorded (the camera also captures the learner’s facial expressions as they attempt the course)

Observation: We then observe the learner as he/she takes the course. We interrupt to ask questions pertaining to the current page, such as ‘what are you looking for?’ We also note down our individual observations. We sit on the right of learner and slightly behind the learner. We note observations for each screen. This includes time spent on each screen, actions performed, comments made, and so on. We try and encourage the learner to voice out what he/she is thinking. To give you a better understanding of how we got about categorizing this information, I have displayed the elements that needs to be observes and the observation criteria.

While conducting the test, we ensure that we keep the following list of don’ts in our mind.

• Avoid jotting down their personal comments or opinions of a behavior. Note down observations in terms of ‘The learner is looking around and changing positions constantly’ rather than ‘The learner is bored.’
• Do not prompt or assist the learner. Allow the learner to identify the way out. If you feel that they are lost, ask them to voice their concerns out aloud.
• If the learner criticizes something, don’t justify or argue. Note this down.
• Don’t ask leading questions to the learner.
• Don’t ask the learner for suggestions for instructional strategies and content presentation.
• Don’t sit too close to the learner.

Posttest Questionnaire: This exercise takes about ten to fifteen minutes. We ask the learner to reflect on his/her learning experience. We ask a series of questions using a posttest questionnaire. The questions asked are neutral in nature. We note the learner’s feedback. Learner feedback is valuable as it can be used as a yardstick to assess the course. Some of the questions included in this questionnaire are:

• What is your overall impression about the course?
• Do you think the way the concepts were explained helped you understand the concepts better?
• Were you able to relate to the scenarios and characters in the course?
• How would you rate the course on the basis of its look and feel?

Ensure that you have a list of crisp, relevant questions to ask your learners. Avoid asking for their suggestions, especially in terms of content accuracy and strategies used. The more comfortable the learner is, the more honest he/she will be while answering these questions.

Analysis and Results

In this step, we collate all the data recorded during the testing phase in an excel sheet. We then look for common trends and patterns across the sample size. We identify the common problems that learners faced during the observation. Remember, all issues mentioned by the learners are not fixed. Only those issues that are common across the sample size are fixed. This ensures that the major hitches in the course are tackled before its final release. We categorize the list of issues and suggest changes based on priority. We then send the final list of fixes to the technical team.

Revision of Course

The technical team starts fixing the issues in the course as suggested in the final list. The issues with the highest priority are fixed. The value-adds are fixed keeping in mind client requirements, alpha release, feasibility, effort involved, and so on. Remember, the objective of this step is not to beautify the course. The main goal is to remove or minimize all obstacles that hinder the learning process.

Conclusion

Testing an elearning course is the most humbling experience. We see things that we haven’t seen before. It is a complete eye-opener. It is very satisfying to know that we have tried 100% to ensure that our learner enjoys the learning experience. We should stop assuming things on the part of the learner and give the learner an opportunity to have his/her say. The end product will then be extremely high in quality, relevant to the learner, interesting, and motivating. I highly recommend you try this and see the difference for yourself. Remember, learnability matters!

Archana learnability, learner testing