Could Your LMS Use A Little User Testing Love?

So you are in the planning team of a new brand LMS. Exciting! Whether it’s a brand new LMS, a radical upgrade, or an inevitable switch, it’s always a good idea to consider how regular users are using and perceiving it.

This article first appeared on LMSPulse.

This is the point of “User Testing,” a dependable tool in the User Experience (UX) Design process. Also one that becomes an afterthought, if it’s considered at all. This is particularly striking in the design process of LMS and elearning apps and systems.

Sometimes you just can’t help but wonder: Has a student ever been consulted?

Part of the problem is that the current literature on usability is the limited supply of studies focusing on students, online learners and similar roles. What little academic output exist, it focuses on theoretical frameworks and plans rather than evidence. Not that theory isn’t valuable, it is perhaps the most needed part of the equation in elearning usability testing. Without a strong candidate, the testing has no better choice than to adhere to the original goals of a usability test: Higher clicks, lower bounce rates, more conversions, more purchases, more valuable shopping carts. Are those the kind of things we’re looking to optimize in a learning experience? We are still at a primitive stage where no commonly accepted framework exists, let alone a systematic gathering of comparable evidence.

For the “edtechpreneurial” minds out there, this may scream opportunity.

What is User Testing? A quick overview of the process

Before you start, it is recommended to keep your “Core Value Proposition” front and center. Aside from elearning, Usability Tests are fairly standardized by now. It is still a good idea to make sure the goals embedded in your system have some degree of coordination with your core message.

As an EdTech company, your Core Value Proposition (CVP) should help you become a part of the learner’s journey towards educational achievement. Keep this in mind when you determine the goals and expected outcomes of your test.

The Process

1. Define your desired outcomes. It can be a little tricky to differentiate between your elearning goals and those of your learners. Often they intersect. Likely, both learner and elearning designers want to increase skills.
2. Have a hypothesis, i.e. a guiding “theory of change.” I cannot stress this enough. Set an expectation, however basic, of how you expect the process to play out, how long it will take, levels of achievement and satisfaction from the experience, etc. Otherwise you risk to see everything as a victory. This is also important for low-budget scenarios: Unless your results show a significant deviation from your expectations, you might want to find the testing results successful, and your experience requiring no further changes. (In other words: Pick your battles.)
3. Design the test. In general terms, there are 3 phases in a test, listed below. You don’t require a literal laboratory as long as you keep close attention to the factors (that is, the variables) that you want to take into account. Just as important is to define a preliminary idea (that is, a hypothesis) of how the outcomes will turn out.
4. Perform the test: Here are the 3 phases mentioned:
   • “User Story“: It will help you analyze your experience in terms of your CVP. By solving the user problem directly or indirectly, not only you can get insight about the usability of your system, but whether or not it provides an experience aligned to your values. The richer the amount of details from the user, all the better; but too many means a cumbersome form to fill out.
   • “Task Center“: Define key tasks commonly performed by the respective user. There are different types, but for now you can focus on two kinds of tasks: Action and Information gathering. Set clear conditions that define the accomplishment of a task. Experts recommend taking advantage of screen and voice recording during the task, and encourage the tester to “think out loud” as they figure the system out. While modern devices make recording increasingly plausible, this could lead to user behaving in ordinary ways.
   • “Assessment“: In the final phase, the ultimate goal is to stack up outcomes and opinions against your hypothesis. As earlier, there is a balance between data volume and drudge. In principle, don’t ask for information for which you don’t have a hypothesis.
5. Test more than once, and as soon as possible after each change. Strongly related to step 2. First rounds of testing should really be considered as baseline, and subsequent ones should be performed after key changes. It is recommended that, if at all possible, first and second training take place between simple and major changes. (Ideally only one change.)

Did the wave collapsed into a particle already? Why and what to consider in an LMS User Testing

They say you modify what you measure. But things may change regardless of whether you measure them or not. Human being do anyway. Furthermore, in learning contexts, it is in the learner’s nature to be continuously measured, either by another human, a technology, or themselves. (Most often, by a mix of everything.) While lab settings try to “isolate” external forces in order to understand the “variables” that are the focus of the research, it can often be not only expensive, but the resulting data is likely to produce inapplicable results.

Fortunately, unlike an online store or a news website, testing and measurement are explicit part of the deal in elearning systems and applications. The right experimental framework can produce data from naturally occurring interactions, in ways that allow for analysis through controlled variables. For statistics and data science enthusiast, this is the field of Bayesian inference. With ample and diverse enough data it is possible to examine the effects of “treatments” (changes on the application) in everyday use, no need of lab work.

In short, user testing would be best thought of as an extension of the regular assessment components of an online experience. It does involve removing binary ideas of “pass” or “fail,” as the focus shifts towards a successful system, rather than a successful student. Of course, student success cannot be guaranteed by a system alone.

This article first appeared on LMSPulse.

Featured Image: Taras Shypka, Unsplash.