Struggling to understand your learning analytics? Give online formative assessment a shot!

How can online formative assessment be used in predicting students' final course performance? What are the predictors of final learning outcomes?

June 26, 2023

Keywords: Learning Analytics, Online Formative Assessment, Prediction
Target readers: educators, researchers, learners

Authors: Okan Bulut & Tarid Wongvorachan

Okan Bulut is an Associate Professor in the Measurement, Evaluation, and Data Science program and a researcher at the Centre for Research in Applied Measurement and Evaluation at the University of Alberta. He conducts research on educational data mining methods, learning analytics, and digital assessments.

Follow Dr. Bulut’s research at https://sites.google.com/ualberta.ca/okanbulut

Tarid Wongvorachan is a doctoral student in the Measurement, Evaluation, and Data Science program at the University of Alberta. His primary area of interest is in the usage of quantitative methods to address research problems in educational measurement and other issues pertaining to social science.

Follow Tarid’s research at https://taridwong.github.io/about.html

 

Picture generated with Bing Image Creator

STRUGGLING TO UNDERSTAND YOUR LEARNING ANALYTICS? GIVE ONLINE FORMATIVE ASSESSMENT A SHOT!

Revolutionizing modern education with learning analytics

Today, we will discuss the benefits of learning analytics in improving our overall learning experience. Learning analytics refers to collecting, analyzing, and interpreting learning-related data to better understand and enhance learning outcomes. Some useful applications of learning analytics include learning analytics dashboards (Roberts et al., 2017), early warning systems (Jokhan et al., 2019), and personalized feedback (Knight et al., 2020).

However, there is a catch.

Sometimes, the tools we use to measure students’ learning progress may not be very beneficial. Although they might give us a score representing the student’s achievement level, they may not tell us exactly why or how the student scored that way. We need models based on theory that can explain how students learn and what factors affect their learning.

What do we do then?

We believe that the secret ingredient is formative assessment—a wide range of methods that teachers use to conduct evaluations of students’ learning during instruction such as ungraded quizzes, reflective course activities, and collaborative problem-solving tasks. Formative assessments help teachers gauge the understanding of their students during instruction and generate valuable feedback that can help students identify their strengths and weaknesses.

Figure 1. The role of online formative assessment within a learning analytics ecosystem.

Formative assessments are aligned with the following elements of learning analytics: theory, learning design, and predictive modelling (see Figure 1).

  • Theory is the use of different ideas and frameworks, such as self-regulated learning and expectancy-value theory, to understand and explain how students learn.
  • Learning design means creating online learning environments, tasks, and social interactions that have a clear purpose and goal for learning.
  • Predictive modelling refers to using statistical methods and techniques to predict student performance based on data collected from their learning activities.

Online formative assessment and learning analytics

In today’s world of learning analytics, online formative assessment tools are essential for enhancing the educational experience. By using these tools, we can obtain valuable data on student progress, which helps us adjust our teaching style to cater to their needs. Also, online formative assessment enables us to develop better predictive models of student learning by providing insights into what they know and can do, as well as their motivation and engagement. These characteristics of online formative assessment make it an effective tool for building accurate, meaningful, and actionable models of student learning based on learning analytics.

An empirical study on online formative assessment

We recently conducted an empirical study (Bulut et al., 2023) on the role of online formative assessment in building predictive models of student learning. Our goal was to investigate which variables (i.e., those extracted from online formative assessments or other types of student actions stored in the learning management system) were strong predictors of learning outcomes.

Using data from several online courses, we found that:

  1. the more quizzes students completed during the course, the better they performed on the final course exam;
  2. students who completed the formative weekly quizzes earlier outperformed those who completed the quizzes right before the course exams to test their knowledge; and
  3. the number of clicks in the learning management system (indicating student activity in the course) was another significant predictor.

Overall, we learned that the features extracted from online formative assessments (e.g., completion, timestamps, and scores) were strong and significant predictors of students’ final course performance.

Our take-home messages

  1. Our research has revealed that online formative assessments are highly practical for learning analytics. These assessments enable educators to collect meaningful data on students’ learning progress.
  2. By utilizing the information gathered from online assessments, educators can gain deeper insights into students’ strengths and weaknesses and provide tailored feedback that caters to their individual learning needs.
  3. Rather than relying on complex data from learning management systems that may not accurately reflect the actual learning outcomes, we recommend that higher education institutions prioritize the use of online formative assessments to optimize the efficiency of their learning analytics systems.

References

Bulut, O., Gorgun, G., Yildirim‐Erbasli, S. N., Wongvorachan, T., Daniels, L. M., Gao, Y., Lai, K. W., & Shin, J. (2022). Standing on the shoulders of giants: Online formative assessments as the foundation for predictive learning analytics models. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13276

Jokhan, A., Sharma, B., & Singh, S. (2019). Early warning system as a predictor for student performance in higher education blended courses. Studies in Higher Education, 44(11), 1900–1911. https://doi.org/10.1080/03075079.2018.1466872

Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á., Kitto, K., Liu, M., Mogarkar, R. V., & Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141–186. https://doi.org/10.17239/jowr-2020.12.01.06

Roberts, L. D., Howell, J. A., & Seaman, K. (2017). Give me a customizable dashboard: Personalized learning analytics dashboards in higher education. Technology, Knowledge and Learning, 22(3), 317–333. https://doi.org/10.1007/s10758-017-9316-1