Dashboards for learners don’t always motivate them

Domain-specific learner-facing learning analytics dashboards (LADs)

Did you ever feel nervous when taking a stats test? Did you ever dread meeting with your statistics professor? Did you ever feel overwhelmed by the number of formulas and computations covered in a stats course?

If you answered "YES!" to any of those questions, you are not alone. In fact, this uneasiness and negative feelings when dealing with statistics characterize a well-known phenomenon called statistics anxiety. To mitigate statistics anxiety and improve the learning experience in an online statistics course, we conceptualized and tested theory-driven learning analytics dashboards (LADs). Below, I provide a brief description of this effort, including the primary inquiry behind the research project, the current stage of the field (unexplored areas in LAD research), and the outcomes related to the two experimental design studies conducted as part of this research.

Why LADs?

The popularization of learning analytics dashboards (LADs) has prompted researchers and designers to consider specific target audiences and their needs. However, even when we focus on a specific audience — such as graduate students taking an online statistics course — the possibilities of how to design, develop, implement, and evaluate LADs can still be overwhelming. While the organization of the data in manageable “chunks” of information is what makes LADs useful to learners, deciding what and how to present information is the key challenge for designers, researchers, and practitioners. In other words,

Can theory-driven LADs be used as effective instructional tools to support learning and motivation?

Are we missing the target?

To tackle the challenges associated with the development and evaluation of LADs intended to support learners, several researchers and practitioners have adopted self-regulated learning models to guide their efforts. One important benefit of using self-regulated learning models is that it encompasses cognitive, behavioral, and affective/motivational constructs. Still, there has been a misalignment between proposed goals and measured outcomes of learner-facing LADs. While most LADs have been developed to support learners’ cognitive outcomes, their evaluations are usually centered around affective or motivational constructs. Ironically, motivation is underrepresented as a target outcome in LADs.

Why should we care about motivation?

The lack of interventions using LADs to support learners' motivation is particularly striking because not only is motivation an important element of self-regulated learning, it can promote learners’ interest and performance in STEM. But how can we design dashboards to support learners’ motivation? The wealth of knowledge and theories of motivation are essential to help us systematically conceive, design, and test LADs. The following two sections describe two experimental studies that have done exactly that. Implemented in the context of an online statistics course, these studies illustrate how the operationalization of theories into LADs to support learners’ motivation is not as straightforward as one would imagine. In fact, these empirical studies have shown some mixed results that highlight how the design of LADs can interact with learners’ specific characteristics (e.g., initial levels of motivation) to generate unexpected outcomes.

Predictive vs. Descriptive LADs

Which do you think is more effective at boosting student motivation? Showing them how they’re doing in relation to the class average, or showing them what grade they’re likely to get?

Informed by the multiple-goal orientation framework, which assumes that mastery and performance goal orientations can co-exist within learners and can optimize motivation and learning outcomes, we designed two LADs in a research study. The first one is a descriptive LAD that features students’ performance in terms of percentages as shown below:

This descriptive learner-facing LAD shows that this student got 94% of responses correct in the module about t-test.

The second example is a predictive LAD that features students’ probability of receiving a specific letter grade at the end of the course

This  predictive learner-facing LAD shows that this student has a 77% probability of receiving a letter grade “B” (blue line) by the end of semester.

We found that:

• both the descriptive and predictive dashboards reduced learners’ negative attitudes toward statistics
• but only the predictive dashboard influenced learners’ motivation.

Still, that influence depended on learners’ initial levels of motivation. More specifically:

• the predictive dashboard increased motivation for learners with initially higher levels of motivation,
• but reduced motivation for learners with initially lower levels of motivation.

Moreover,  interviews indicated that learners who adopted a more performance-avoidance goal orientation (fear of performing poorly in the course) presented higher levels of anxiety regardless of the dashboard used.

Can we scaffold for task value?

Which do you think is more effective at boosting student motivation? Simply teaching them topics, or explaining why each topic is important?

Based on the results above, we changed the design of the predictive dashboard to include prompts about the value of learning specific topics and skills in the statistics course. Different prompts were used for each module and emphasized the utility of learning the topics/skills covered in that module as shown below. This design was informed by the situated expectancy-value theory, which assumes that self-efficacy (expectancy for performing the task well) and interest in the task (subjective task value) are key components to support learners’ motivation. Based on this theory, we were relatively confident that the inclusion of these prompts would result in improved outcomes.