learnability matters

Geeta Bose presented this paper written by Archana Narayan on Learnability Testing of Online Courses at the 3rd National Seminar on ‘eLearning and eLEarning Technologies”, (ELELTECH India) November 5-6th, 2009. This seminar was organized by Center for Development of Advanced Computing (C-DAC), Hyderabad and JNT University Hyderabad in association with Department of Information Technology, Ministry of Communications and Information Technology Government of India.

Abstract—

At Kern Learning Solutions, we have a unique testing process to evaluate an online course before its final release. We call this “learnability testing”. The aim is to ensure that all obstacles that hinder learnability are spotted and removed. In this paper, we will discuss how this testing process is different from testing conducted by other eLearning companies. Learnabilty testing is a comprehensive process that tests the instructional strategies used and the usability of the online course. We will look at how this testing is conducted.
Index Terms— Learnability, usability, concept, instructional strategy

I. INTRODUCTION

Let’s take a minute to understand how testing happens in typical elearning companies. Testing of a software application helps spot technical errors, programming bugs, and other functionality issues. A clean chit from the testing department ensures that a bug-free course is released. However, is this sufficient to certify that the course is effective? Shouldn’t you test the effectiveness of the instructional approaches used? How about evaluating whether the instructional design maps to the needs of the learners? How do you check whether the course ensures effective learning or not? Today, most elearning companies believe that mimicking the testing mechanism used by software companies is the most effective testing methodology. What they fail to consider is that the testing objectives for a software application and an elearning application are very different. The main objective of testing a software application is to ensure smooth functionality, efficiency, and usability. However, the main objective of testing an elearning application is to check how “learnable” the course is. Learnability testing is designed specifically to evaluate the effectiveness of a learning program. Most IDs resist concept testing of their courses by learners.

What are the common misconceptions of instructional designers, graphic designers, and/or programmers?

1. The course is extremely intuitive. This is probably the most common misconception. There are interfaces that look simple but are highly unintuitive. On the other hand, there are complex interfaces that only confuse the learner. Learnability testing checks to see whether the course is truly intuitive.
2. There are no bugs in my course. During a quality assurance check (QA), we look at the course sequentially with a specific goal. Even if you think you have tried everything, learnability testing shows how real learners will use the course. This may open your eyes to a few bugs.
3. We have provided every feature that the learner needs. Let the learners confirm this. After learnability testing, you may either discover that there are certain features that the learner will never use or that he/she is looking for a feature that you have overlooked. You may also discover that the learner simply cannot find certain features!
4. The graphics are attractive and pleasing to the eye. Learnability testing helps us understand if graphic image are self-explanatory or not. You may realize that there are a few images that do not aid learning.
5. The content has been double checked for accuracy by the SME and client. The subject matter expert (SME), client, and all other others involved in the development life cycle (DLC) may look at the course from the learner’s point of view. However, during learnability testing, you may realize that the learner’s experiences are different and they are unable to relate to the content. SME and client at times give you information as they ought to exist and not how they are in reality.
6. The instructional strategies used are engaging and motivating. Let the learner tell you this. There are certain strategies such as games, simulations, branching stories that overwhelm the learner, while others leave them bored. It is important to understand how they respond to the strategy used.
7. The course has covered the organizational and learning objectives. It is equally important to know whether you are able to meet these objectives. Does your course have the desired effect?

Learnability testing is inspired by usability testing. In simple words, usability testing is a controlled experiment that tests how well people use a particular product. The user performs a series of tasks while the observer watches him and takes notes. Usability testing is conducted at several stages and feedback is implemented to refine the design. At Kern, we modified this testing methodology to test elearning courses. Where usability testing checks whether a product is ‘usable’, learnability testing checks whether an elearning application is‘learnable’. A course is ‘learnable’ if it fulfills the following criteria:

• Is the learner motivated to finish the course? Is the course engaging?
• Do the instructional strategies interest the learner? Simply put, does the learner ‘get’ the theme?
• Is the instructional flow clear to the learner? Is the learner comfortable with the content?
• Is the course free of obstacles (too much audio, too much text, bugs, navigational errors and so on) that may hinder learning?
• Is the course usable?

II.    DETAILED LEARNABILITY TESTING PROCESS

Learnability can be divided into two broad categories: concept testing and usability testing.

Fig. 1 Learnability Testing as a process

A.    Concept Testing

Concept testing can be carried out during the design phase and the development phase. At Kern, we carry out concept testing on wire frames to ensure that we are in sync with the learner’s needs. During the design phase, we recruit learners and ask them to take the course in the wire frame format. We do not check functionality at this phase. You can conduct concept as many times as you feel necessary during the design phase. If you have done your research accurately, one round of concept testing will suffice. During the development phase, concept testing is conducted as part of the learnability process.
i. Instructional Strategy: The questions we ask ourselves are:

1. Is the learner showing interest in taking the course?
2. Is the learner motivated to read information?
3. Does the learner ‘understand’ and ‘relate’ to the theme used?

Given below is a screen shot of a wire frame. The learners were software developers and the main objective of the course was to help the learner understand the importance of usability and provide tips that he can use in his day-to-day work.

Fig. 2 Example of Instructional Strategy during Concept Testing

After storyboarding, Kern arranged a concept testing using the wire frame. We recruited software developers and asked them to take the course. The concept testing opened our eyes to several things. If we had not conducted a test at this phase and gone ahead with development, we would have had an ineffective product. The main learning from the concept testing were:

• The learners did not use Web 2.0 tools to collaborate. They were not confident about sharing their knowledge and did not see value in it.
• Most exercises in this course were open ended to ensure that the learner had the space to think his answers out. He had to type in their response in most cases. The learner was not sufficiently motivated to type out his answers.
• The learner found a particular module very basic and wished to have more examples from his day-to-day life.
• The learner was still not able to grasp the importance of usability.

ii. Content: It is important to check what the learner thinks about the content and its coverage.

1. Is she looking more information?
2. Does she feel that there is too much information?
3. We must find out whether the learner is able to relate to the scenarios, examples, and so on used in the course.
4. Is the learner able to understand the content? Is the learner comfortable with the language used?

We had designed a course for a leading bank. The learners were call center executives who primarily took calls and used a software application to enter data. During concept testing, we realized that we had used the term ‘tagging a call’ which the learners were unfamiliar with. Even though, this was provided by the SME, this phrase confused the learners. We learnt that the common lingo used at the workplace was ‘marking a call’.

iii. Audio: Learnability testing has shown us that different learner profiles react differently to audio. Therefore, audio is always an element we are eager to test. Testing helps us identify the right dose of audio. It helps us find answers for the following:

1. Is the audio sufficient on each screen?
2. Is the learner distracted by audio?
3. Does the learner avoid reading text because of audio?
4. What is the learner’s reaction to the pace of the audio?
5. Is the transcript (if used) useful?
6. Does the learner point out any discrepancies in language?

During concept testing for the course on Dealing with Conflict at the Workplace, we realized that the learners read the on screen text much faster than the audio. Therefore, they found the audio very distracting. Several learners start pointing out discrepancies in the audio and the on screen text. Therefore, learning was affected as they were distracted.

iv. Exercises: It is important to check whether the learner is applying the knowledge they have gained. How do they respond to exercises? Do the exercises challenge them? Do the learners find the exercises motivating? Is the learner comfortable with the feedback received? Concept testing helps us find the answers for this.

Given below is a screen shot from a course we designed for a leading IT company. The learners were middle level managers. This course was meant as a refresher and the main objective was to standardize the performance engagement process across the organization.

Fig. 3 Example of exercise during Concept Testing

During concept testing, we realized that while attempting the exercises, the learners were very confused with the feedback. We had not used the standard ‘That’s correct’ and ‘That’s incorrect!’ format. Therefore, the learners were unable to tell whether they had got the answer right or not and were dissatisfied with the exercises.

B.    Usability Testing
Usability testing can be conducted during the development phase and at the end of the DLC, i.e. when the alpha version is rolled out. The main objective is to test the usability of the eLearning course. Are there any usability issues in the course? The categories covered under this are discussed below.

i. Navigation:

• Is the learner able to find his way through the course?
• Are the Next and Back buttons positioned at the right place?
• Is the navigation intuitive?

Given below is a screen shot from a course we designed for a leading IT company. The learners were middle level managers. This course was meant as a refresher and the main objective was to standardize the performance engagement process across the organization. In this particular screen, each element is clickable. If you click a particular topic, you are taken to the corresponding sub screens related to this topic. After viewing the sub-screens of the topic, you will be brought back to the same screen with the topic previously visited dulled out.

Fig. 4 Example of navigation during Usability Testing

During usability testing, as expected learners selected the topics in a random fashion. What had missed our eyes during our QA crept up during learnability testing. The learners realized that the page numbering was jumbled up. This was because they were viewing the topics in random while the numbering had been done in a linear fashion. The numbering therefore, confused the learner.

ii. Learner Interface: It is important to find out if all elements (icons, buttons, and so on) on the interface are intuitive. Does the learner use all the elements provided on the interface? Is the learner searching for a particular feature such as Pause or Mute? Does the learner find the interface confusing or easy to use? How often does the learner use the features such as Mute, Pause, References and so on? We get answers to these questions through usability testing.

Given below is a screen shot from a course Dealing With Conflict at the Workplace. The learners were middle level managers. This course had audio that read the text off the screen.

Fig. 5 Example of lerner interface during Usability Testing

During usability testing, we realized that the learners immediately looked for Mute to turn off the audio. Since the audio read the text off the screen, they were distracted by the audio and preferred to read the text at their own pace. We also realized that the learner never used Resources and the Glossary.

iii. Affordance: Does the learner click all the clickable icons? Did she click any element that was not clickable? Does the learner rely on instruction text to carry out a particular action? Did the learner complete all tasks on a screen before moving ahead?

Given below is a screen shot from a course we designed for Godrej Lifespace. The learners were customer service executives who required training in grooming skills and personality development.

Fig. 6 Example of affordance during usability testing

During usability testing, we realized that even though the Menu was default open, the learner did not know where she needed to click to begin the course. Most learners asked the observers what they had to do to proceed further. We had toinclude an instruction text to ensure that the learner knew where to begin.

III.    CONDUCTING LEARNABILITY TESTING

As discussed earlier, learnablity testing includes both concept testing and usability testing. Therefore, this test is typically conducted towards the end of the development phase or on the alpha version of the course.
We decide the number of learners who need to go through the course (sample size) and recruit these learners. The sample size depends on the overall size of the target audience. If it is a smaller group of say, 50 learners, a sample size of seven should be sufficient. We request the clients to give us access to their employees. We then identify the learners based on the target audience defined at the beginning of the project. If the learners are remote or inaccessible, we recruit individuals who have a similar profile. While doing this, we keep in mind cultural diversities.

A.    Set up
At the learner’s workplace: We, typically, seek permission to conduct learnability testing in the workplace at the learner’s workstation. Therefore, we get to test the eLearning course in the learner’s real work environment. This requires two observers, a video camera, notebooks/pens, and audio recorder. The observers must be fluent in the learner’s preferred language.
Research Lab: Alternatively, we also set up the test in our research lab. We recruit learners and ask them to take the eLearning course in a controlled environment. We have a one- way mirror, a video camera, TV, two or more observers, (if required) and notebooks/pens. We opt for this when we recruit individuals who resemble the learner profiles. We also suggest you use this technique for high end courses that requires higher degree of observation.
The observer sits on the right of the learner and slightly behind the learner. This gives the learner his space and ensures that his personal space is not invaded. It is important that he feels comfortable during the testing process.

B. Introduction
We begin the learnability testing by introducing ourselves to the learners. We brief him about the objective of process. We clearly state the expectations. This part is extremely important as it sets stage for the rest of the process. A sample introduction is given below.

“My name is Archana and I work with Kern Learning Solutions Pvt. Ltd. I will show you the draft design of an elearning program that will help you learn grooming skills and personality development. I need your help to evaluate how well the course is designed. We really appreciate your participation. As a token of appreciation, here is a small gift.(pause) Please go through the course as if you are going through it as a part of your work. If you are able to go through it without hitches, we know that the course is designed well. However, if you face any trouble, we will know that design needs more work. This is a test of the COURSE. And, it is NOT a test of YOUR abilities. So if you can’t proceed at any point of time, don’t worry. It just means the course can be designed better and needs more work.
I will just observe you from behind and I want you to complete the course. As we’re learning, please think out loud. You can take a short break in between if you want. Remember to tell me what you’re looking at, why you choose a button or link, and what you expect. Any questions? (pause) Let’s start.”

It is extremely important to let the learner know that we are not testing his abilities and that there is no pressure to take the test. If the learner is not interested in participating, he is free to leave. This helps put him at ease and he will share feedback freely. Our aim is to build a rapport with the learner and let him understand the importance of his feedback. We brief him on how the testing will be conducted and inform him that the entire session will be recorded (audio/video).

C. Observation
We then observe the learner as she takes the course. We encourage the learner to voice her thoughts by asking them ‘what are you looking for?’ or ‘is something wrong?’ We also note down our individual observations. We record time spent on each screen, actions performed, comments made, and so on. We design a protocol to ensure that we keep the crucial aspects of concept testing and usability testing in our minds while observing the learner’s actions.

D.    Post-test Questionnaire
This exercise takes about ten to fifteen minutes. After taking the course, we ask the learner to reflect on his/her learning experience. We ask a series of questions using a post-test questionnaire. The questions asked are neutral in nature. We note the learner’s feedback. Learner feedback is valuable as it can be used as a yardstick to assess the course. Some of the questions included in this questionnaire are:

• What is your overall impression about the course?
• Do you think the way the concepts were explained helped you understand the concepts better?
• Were you able to relate to the scenarios and characters in the course?
• How would you rate the course on the basis of its look and feel?

Fig. 7 Example of post test questionnaire

Ensure that you have a list of crisp, relevant questions to ask your learners. Avoid asking for their suggestions, especially in terms of content accuracy and strategies used. The more comfortable the learner is, the more honest she will be while answering these questions.

IV.    ANALYSIS AND RESULTS

In this step, we collate all the data recorded during the testing phase in an excel sheet. We then look for common trends and patterns across the sample size. We identify the common problems that learner faced during the observation. Remember, all issues mentioned by the learner are not fixed. Only those issues that are common across the sample size are fixed. This ensures that the major hitches in the course are tackled before its final release. We categorize the list of issues and suggest changes based on priority. We then send the final list of fixes to the technical team.

Fig. 8 Example of a report generated after analysis

V. REVISION OF COURSES

The technical team starts fixing the issues in the course as suggested in the final list. The issues with the highest priority are fixed. The value-adds are fixed keeping in mind client requirements, beta release date, feasibility, effort involved, and so on. Remember, the objective of this step is not to beautify the course. The main goal is to remove or minimize all obstacles that hinder the learning process. If you do a concept testing using wire frames, it will reduce the rework that may be involved at a later stage. You will be able to identify obstacles that hinder learning much earlier in the process.

VI.    POINTS TO REMEMBER

While conducting learnability testing, keep the following in mind.

• Avoid jotting down the your personal comments or opinions. Note down observations in terms of ‘The learner is looking around and changing positions constantly’ rather than ‘The learner is bored.’
• Do not prompt or assist the learner. Allow the learner to identify the way out. If you feel that he are lost, ask him to voice his concerns out aloud.
• If the learner criticizes something, don’t justify or argue. Simply, note it down.
• Don’t ask leading questions to the learner. •    Don’t ask the learner for suggestions on instructional strategies and content presentation.
• Don’t sit too close to the learner.

VII. CONCLUSION

Typically, the client goes through the alpha version of the online course and provides feedback based on what he thinks is good for the learners. This is plugged into the course and the beta version of the course is rolled out. After implementation, the learners are asked to take the course. As the learner takes the course, she encounters certain difficulties. The feedback from learners is poor and it is too late to make changes to the program. The client is upset with elearning companies for not doing their jobs. Learnability testing allows the learner to take the complete course before the final release. Learnability testing of an online course has the following benefits:

• This testing is conducted in an objective and scientific manner.
• It helps identify major road blocks and obstacles to learning.
• It gives a quick trailer of how a learner will respond to your course.
• The feedback is based on learner behavior and not client’s interpretation of learner behavior.
• We can guarantee learning effectiveness and learner motivation.
• It acts as a last and final check to ensure that your course is on the right track and no surprises spring up later.
• Learners and clients are involved much earlier in the process. Therefore, the chance of getting it right increases drastically.

Learnability testing ensures that we, as learning companies, take responsibility for effectiveness of a course. We should be able to give a guarantee of effectiveness rather than deliver a course and then, forget about it. Testing an elearning course is the most humbling experience. We see things that we haven’t seen before. I would suggest that instructional designers conduct the learnability testing as they will be more tuned to the learner’s needs. It is a complete eye-opener. It is very satisfying to know that we have tried 100% to ensure that our learner enjoys the learning experience. We should stop assuming things on the part of the learner and give the learner an opportunity to have her say. The end product will then be extremely high in quality, relevant to the learner, interesting, and motivating.

Share this