What is testing in a typical eLearning environment?
In a typical eLearning company, testing involves several rounds of QA. To ensure that the course is error-free, these QA rounds generate numerous defect reports. But is that all that you need to test the course for?
Shouldn’t you test the effectiveness of the instructional approaches? How about evaluating whether the instructional design maps to the needs of the learners or testing whether the course ensures effective learning? And finally, how about testing whether the course is effective or not?
Why is this important you may ask?
Let’s take a minute to understand how testing happens in typical elearning companies. Testing for technical and programming errors and for the entire functionality of the course is done. A clean chit from the testing department ensures that a bug-free course is released. But there is a huge difference in a software application and an elearning program. Today, most elearning companies believe that mimicking the testing mechanism used by software companies is the most effective testing methodology. What they fail to consider is that the testing objectives for a software application and an elearning application is very different. The main objective of testing a software application is to ensure smooth functionality, efficiency, and usability. What is the main objective of testing an elearning application? The ultimate test of an elearning program is its “learnability” – how “learnable” is the course?
You can do a simple check at your end to figure out the initial learnability issues. Just bring in 5 learners and ask them to go through the course as they would in real life. You will be surprised to find that your water-tight, defect-free course does have some hic ups!
Coming back to the original question, why is it important to test the effectiveness of your course, let’s list the common misconceptions of eLearning designers:
- The course is extremely intuitive. (Well, you can never know till you have evidence to support this. You may observe that things that are intuitive to us are extremely confusing to the learner.)
- There are no bugs in my course. (The learner tries things differently. He/she may open your eyes to a few bugs that you missed.)
- We have provided every feature that the learner needs. (Now, you may either discover that there are certain features that the learner will never use or that he/she is looking for a feature that you have overlooked. You may also discover that the learner simply cannot find your features!)
- The graphics are attractive and pleasing to the eye. (You may find out that certain graphic images are not as self-explanatory as you thought them to be. The learner just doesn’t get it.)
- The content has been double checked for accuracy by the SME and client. (You may observe that the learner’s experiences are different and they are unable to relate to the content.)
- The instructional strategies used are engaging and motivating. (Well, that’s what you think! There is a chance that the learner would find the strategies boring and frustrating.)
- The course has covered the organizational and learning objectives. (Is covering the objectives sufficient? Shouldn’t you see whether they actually work?)
So, how do you test the effectiveness of your course? At Kern, we developed a process to evaluate a course before its final release. The aim was to ensure that all obstacles that hinder the learnability are spotted and removed.
To know more about the process that we follow and how we do it, await our next blog. In the meantime, if you have any thoughts, would love to hear from you.
This post has been viewed 1493 times.
10th October, 2007 at 5:39 am
Why do we have to wait for another article? I can’t wait…
10th October, 2007 at 6:03 am
So that you get some time to ponder over this….
Will post the next part soon!
10th October, 2007 at 7:30 am
Very interesting and so well-written. From what I understand, are you talking about testing as we understand usability testing? Where you make users go through the course and observe their experiences? If yes, path-breaking stuff. Cant wait to read more
Definitely something no eLearning company does but something which every company needs to do.
10th October, 2007 at 9:19 am
Glad you brought this up… learnability testing has been derived from usability testing. Learnability testing has been designed keeping in mind how usability testing can be adopted to the elearning scenario. The objective of which is to ensure that all obstacles are removed to ensure a pleasant learning experience. It test whether the course is ‘learnable’ and helps in identifying the road blocks.
You are right…. no elearning company takes this aspect seriously. Guess most are just content with a bug-free course. Guess it will be clearer with the second part of this article- Learnability testing
10th October, 2007 at 9:23 am
Hi
I had written an article on this topic sometime back :http://writersgateway.wordpress.com/2007/09/26/whoa-a-bug-the-importance-of-testing-and-evaluating-an-e-learning-product/
From my experience, I would say most companies do not actually measure the effectiveness of the course before it goes out to the target audience. Also testing is not done properly.
I guess testing and evaluating an e-learning course is very crucial and instructional designers need to be aware of its importance.
Thanks for the post.
Waiting to read the next one.
10th October, 2007 at 9:32 am
[...] There is another interesting post on this topic here. [...]
11th October, 2007 at 7:00 am
Hi Rupa,
Read your post. Very interesting…
As you would have noticed, I have restricted myself to only testing the course and not to evaluating its effectiveness. Evaluation is a huge area in itself. We are hoping to post a case study on that soon! Stay in the loop.
13th October, 2007 at 3:07 pm
Well written article, Archana. In most companies, the typical testing round before the final release is conducted in isolation. Here “testers” test the course and “learners” are nowhere in the loop. There is a huge difference in findings when testers test the course in isolation versus when testers test the course in the “presence of learners”. As we have often seen, testing the course in the presence of learners can be quite a humbling but a truly learning experience for the designers.
It will be really interesting if you can post some real life examples and findings from our learnability testing rounds - especially about the list of common misconceptions that you have mentioned here.
15th October, 2007 at 4:13 am
Hi Geeta,
Sure will post some examples and findings with respect to Kern’s experience as soon as I post the article on how learnability testing is actually conducted at Kern.
Cheers,
Archana
17th October, 2007 at 3:20 pm
My background is in testing mostly within the web and (e)learning sector.
What you mention here cannot possibly be down to one person or team.
When you mention that people test things to ensure it is *bug free* (though there are always bugs in software) this is usually because this is what they have been asked to do. I call this the *mission*.
When someone asks me to test something, I specifically ask them to tell me what the mission is, ie: what am I supposed to be testing? what areas? why? what’s the timeframe? etc…
If they wanted me to test the learnability I probably could, however the reality is it will take longer and cost more. It is very rare that there is a big enough budget to do everything they ask. In an ideal world a good project team would take testing, evaluation and QA to a greater level, but most of us don’t live in that ideal world.
The proof is in the many poor quality elearning courses I have come across out there.
18th October, 2007 at 10:00 am
Hi Rosie,
This article is written exclusively for testing in the elearning sector. The main objective of this article was bring to the forefront that removing bugs is not sufficient to ensure that your eleanring course is learnable. Removing bugs is important no doubt…… But what we are trying to stress on is that elearning application testing should be adapted and it is not sufficient to test it like a software application.
When I talk about learnability, what I mean is removing all obstacles that hinder learning. Learnability testing ensures that all obstacles are removes or minimized to ensure a pleasant learning experience. We do this by allowing the learner to interact with application while we observe their experience. Now again, this is very different from evaluating the actual effectiveness of a course.
Trust me, this does not involve a big budget. Lots of poor eleanring is churned out under excuse of low budget. We, at Kern, factor in time for testing (QA and learnability testing happens separately also).
Cheers,
Archana
26th October, 2007 at 6:32 pm
Completely agree Archana. Learnability testing does not require a big budget. It only involves planning the project well to factor in this critical phase that most elearning companies do not treat seriously.
On the contrary, the benefits of learnability testing extend beyond the existing course. Learnability testing throws up many interesting results that can be used as best practices while designing future courses - thus ensuring better courses in future.
Rosie said: “When someone asks me to test something, I specifically ask them to tell me what the mission is, ie: what am I supposed to be testing? what areas? why? what’s the timeframe? etc…”
Rosie, it is good to know that you specifically ask for these details. To obtain accurate results, it is very important to ask relevant questions. Rosie, if you add some questions about your learners, their needs and requirements to your queries, you will be amazed at the focus your testing phase will have.
6th April, 2010 at 2:18 am
[...] learnability matters » Do you test your eLearning Courses? [...]
6th July, 2010 at 4:43 am
[...] There is another interesting post on this topic here. [...]
9th July, 2010 at 6:23 pm
[...] effectiveness of your own courseware. I’m not talking about testing the usability of the work – that should always be done – I’m talking about testing on the material. How else can you know if you’ve managed to [...]