AI-powered educational technology that is designed to support teachers in providing personalized instruction can enhance their ability to address the needs of individual students, hopefully leading to better learning gains. This paper presents results from a participatory research aimed at co-designing with science teachers a learning analytics tool that will assist them in implementing a personalized pedagogy in blended learning contexts. The development process included three stages. In the first, we interviewed a group of teachers to identify where and how personalized instruction may be integrated into their teaching practices. This yielded a clustering-based personalization strategy. Next, we designed a mock-up of a learning analytics tool that supports this strategy and worked with another group of teachers to define an ‘explainable learning analytics’ scheme that explains each cluster in a way that is both pedagogically meaningful and can be generated automatically. Third, we developed an AI algorithm that supports this ‘explainable clusters’ pedagogy and conducted a controlled experiment that evaluated its contribution to teachers’ ability to plan personalized learning sequences. The planned sequences were evaluated in a blinded fashion by an expert, and the results demonstrated that the experimental group – teachers who received the clusters with the explanations – designed sequences that addressed the difficulties exhibited by different groups of students better than those designed by teachers who received the clusters without explanations. The main contribution of this study is twofold. First, it presents an effective personalization approach that fits blended learning in the science classroom, which combines a real-time clustering algorithm with an explainable-AI scheme that can automatically build pedagogically meaningful explanations from item-level meta-data (Q Matrix). Second, it demonstrates how such an end-to-end learning analytics solution can be built with teachers through a co-design process and highlights the types of knowledge that teachers add to system-provided analytics in order to apply them to their local context. As a practical contribution, this process informed the design of a new learning analytics tool that was integrated into a free online learning platform that is being used by more than 1000 science teachers.

Multimodal Learning Analytics (MMLA) innovations are commonly aimed at supporting learners in physical learning spaces through state-of-the-art sensing technologies and analysis techniques. Although a growing body of MMLA research has demonstrated the potential benefits of sensor-based technologies in education, whether their use can be scalable, sustainable, and ethical remains questionable. Such uncertainty can limit future research and the potential adoption of MMLA by educational stakeholders in authentic learning situations. To address this, we systematically reviewed the methodological, operational, and ethical challenges faced by current MMLA works that can affect the scalability and sustainability of future MMLA innovations. A total of 96 peer-reviewed articles published after 2010 were included. The findings were summarised into three recommendations, including i) improving reporting standards by including sufficient details about sensors, analysis techniques, and the full disclosure of evaluation metrics, ii) fostering interdisciplinary collaborations among experts in learning analytics, software, and hardware engineering to develop affordable sensors and upgrade MMLA innovations that used discontinued technologies, and iii) developing ethical guidelines to address the potential risks of bias, privacy, and equality concerns with using MMLA innovations. Through these future research directions, MMLA can remain relevant and eventually have actual impacts on educational practices.

Several emotional theories that inform the design of Virtual Learning Environments (VLEs) categorize affect as either positive or negative. However, the relationship between affect and learning appears to be more complex than that. Despite several empirical investigations in the last fifteen years, including a few that have attempted to complexify the role of affect in students’ learning in VLE, there has not been an attempt to synthesize the evidence across them. To bridge this gap, we conducted a systematic review of empirical studies that examined the relationship between student outcomes and the affect that arises during their interaction with a VLE. Our synthesis of results across thirty-nine papers suggests that except engagement, all of the commonly studied affective states (confusion, frustration, and boredom) have mixed relationships with outcomes. We further explored the differences in student demographics and study context to explain the variation in the results. Some of our key findings include poorer learning outcomes arising for confusion in classrooms (versus lab studies), differences in brief versus prolonged confusion and resolved versus persistent confusion, more positive (versus null) results for engagement in learning games, and more significant results for rarer affective states like frustration with automated affect detectors (versus student self-reports). We conclude that more careful attention must be paid to contextual differences in affect's role in student learning. We discuss the implication of this review for VLE design and research.

Collaborative argumentation is key to promoting understanding of scientific issues. However, classroom structures may not always prepare students to engage in argumentation. To address this challenge, education researchers have examined the importance of social knowledge construction and managing uncertainty in group understanding. In this study, we explore these processes using data from /r/ChangeMyView, an online forum on Reddit where users present their opinions, engage others in critiquing ideas, and acknowledge when the discussion has modified their opinions. This unfacilitated environment can illuminate how argumentation evolves naturally towards refined opinions. We employ automated text analyses (LIWC) and discourse analyses to understand the features and discourse sequences of successful arguments. We find that argumentative threads are more likely to be successful if they focus on idea articulation, coherence, and semantic diversity. Findings highlight the role of uncertainty: threads with more certainty words are less likely to be successful. Furthermore, successful arguments are characterized by cycles of raising, managing, and reducing uncertainty, with more occurrences of evidence and idea incorporation. We discuss how learning environments can create norms for idea construction, coherence, and uncertainty, and the potential to provide adaptive prompts to maintain and reduce uncertainty when unproductive argumentative sequences are detected.

Collaborative problem solving has numerous benefits for learners, such as improving higher-level reasoning and developing critical thinking. While learners engage in collaborative activities, they often experience impasse, a potentially brief encounter with differing opinions or insufficient ideas to progress. Impasses provide valuable opportunities for learners to critically discuss the problem and re-evaluate their existing knowledge. Yet, despite the increasing research efforts on developing multimodal modeling techniques to analyze collaborative problem solving, there is limited research on detecting impasse in collaboration. This paper investigates multimodal detection of impasse by analyzing 46 middle school learners’ collaborative dialogue—including speech and facial behaviors—during a coding task. We found that the semantics and speaker information in the linguistic modality, the pitch variation in the audio modality, and the facial muscle movements in the video modality are the most significant unimodal indicators of impasse. We also trained several multimodal models and found that combining indicators from these three modalities provided the best impasse detection performance. To the best of our knowledge, this work is the first to explore multimodal modeling of impasse during the collaborative problem solving process. This line of research contributes to the development of real-time adaptive support for collaboration.

Evidence from various domains underlines the key role that human factors, and especially, trust, play in the adoption of technology by practitioners. In the case of Artificial Intelligence (AI) driven learning analytics tools, the issue is even more complex due to practitioners’ AI-specific misconceptions, myths, and fears (i.e., mass unemployment and ethical concerns). In recent years, artificial intelligence has been introduced increasingly into K-12 education. However, little research has been conducted on the trust and attitudes of K-12 teachers regarding the use and adoption of AI-based Educational Technology (EdTech).

The present study introduces a new instrument to measure teachers’ trust in AI-based EdTech, provides evidence of its internal structure validity, and uses it to portray secondary-level school teachers’ attitudes toward AI. First, we explain the instrument items creation process based on our preliminary research and review of existing tools in other domains. Second, using Exploratory Factor Analysis we analyze the results from 132 teachers’ input. The results reveal eight factors influencing teachers’ trust in adopting AI-based EdTech: Perceived Benefits of AI-based EdTech, AI-based EdTech’s Lack of Human Characteristics, AI-based EdTech’s Perceived Lack of Transparency, Anxieties Related to Using AI-based EdTech, Self-efficacy in Using AI-based EdTech, Required Shift in Pedagogy to Adopt AI-based EdTech, Preferred Means to Increase Trust in AI-based EdTech, and AI-based EdTech vs Human Advice/Recommendation. Finally, we use the instrument to discuss 132 high-school Biology teachers’ responses to the survey items and to what extent they align with the findings from the literature in relevant domains.

The contribution of this research is twofold. First, it introduces a reliable instrument to investigate the role of teachers’ trust in AI-based EdTech and the factors influencing it. Second, the findings from the teachers’ survey can guide creators of teacher professional development courses and policymakers on improving teachers’ trust in, and in turn their willingness to adopt, AI-based EdTech in K-12 education.

The introduction of math solving photo apps in late 2014 presented students with a tempting new way to solve math problems quickly and accurately. Despite widespread acknowledgement that students increasingly use these apps to complete their coursework, as well as growing concerns about cheating as more students learn online, the prevalence and impact of this technology remains largely unexplored. This study uses a large dataset consisting of 700 unique math exercises and over 82 million student submissions to investigate changes in exercise answering speeds during the last decade. Through a series of exploratory analyses, we identify dramatic shifts in exercise submission speed distributions in recent years, with increasing numbers of rapid responses suggesting growing student reliance on math solving photo technology to answer math problems on homework and exams. Our analyses also reveal that decreases in exercise answering speeds have occurred contemporaneously with the introduction and proliferation of math solving photo apps in education and we further substantiate the role of these tools by verifying that exercise susceptibility to math solving photo apps is associated with decreases in submission speeds. We discuss potential applications of our findings to improve math assessment design and support students in adopting better learning strategies.

This mixed-method study examined the impacts of a learning-analytics (LA) hints system on middle school students’ problem-solving performance and self-efficacy (SE). Students in condition A received the LA hint system, students in condition B received a static hint system that contains the same set of hints but without the LA mechanism, condition C was a control group that no hints were provided. The statistical results showed that the problem-solving SE for students who engaged with the LA hint system improved significantly. Student interviews revealed that real-time supports and in-time positive feedback played key roles in supporting their SE growth. Moreover, student-generated quantitative and qualitative log data were collected for interpreting the research outcomes. The quantitative logs provided an in-depth examination of problem-solving strategies across the conditions while the qualitative logs provided another perspective to understand students’ problem-solving status. Implications for future implementation of LA-hint system in virtual PBL environments were provided.

Teacher’s in-class positioning and interaction patterns (termed spatial pedagogy) are an essential part of their classroom management and orchestration strategies that can substantially impact students’ learning. Yet, effective management of teachers’ spatial pedagogy can become increasingly challenging as novel architectural designs, such as open learning spaces, aim to disrupt teaching conventions by promoting flexible pedagogical approaches and maximising student connectedness. Multimodal learning analytics and indoor positioning technologies may hold promises to support teachers in complex learning spaces by making salient aspects of their spatial pedagogy visible for provoking reflection. This paper explores how granular x-y positioning data can be modelled into socio-spatial metrics that can contain insights about teachers’ spatial pedagogy across various learning designs. A total of approximately 172.63 million position data points were collected during 101 classes over eight weeks. The results illustrate how indoor positioning analytics can help generate a deeper understanding of how teachers use their learning spaces, such as their 1) teaching responsibilities; 2) proactive or passive interactions with students; and 3) supervisory, interactional, collaborative, and authoritative teaching approaches. Implications of the current findings to future learning analytics research and educational practices were also discussed.

Despite theoretical benefits of replayability in educational games, empirical studies have found mixed evidence about the effects of replaying a previously passed game (i.e., elective replay) on students’ learning. Particularly, we know little about behavioral features of students’ elective replay process after experiencing failures (i.e., interruptive elective replay) and the relationships between these features and learning outcomes. In this study, we analyzed 5th graders’ log data from an educational game, ST Math, when they studied fractions—one of the most important but challenging math topics. We systematically constructed interruptive elective replay features by following students’ sequential behaviors after failing a game and investigated the relationships between these features and students’ post-test performance, after taking into account pretest performance and in-game performance. Descriptive statistics of the features we constructed revealed individual differences in the elective replay process after failures in terms of when to start replaying, what to replay, and how to replay. Moreover, a Bayesian multi-model linear regression showed that interruptive elective replay after failures might be beneficial for students if they chose to replay previously passed games when failing at a higher, more difficult level in the current game and if they passed the replayed games.

Email communication between instructors and students is ubiquitous, and it could be valuable to explore ways of testing out how to make email messages more impactful. This paper explores the design space of using emails to get students to plan and reflect on starting weekly homework earlier. We deployed a series of email reminders using randomized A/B comparisons to test alternative factors in the design of these emails, providing examples of an experimental paradigm and metrics for a broader range of interventions. We also surveyed and interviewed instructors and students to compare their predictions about the effectiveness of the reminders with their actual impact. We present our results on which seemingly obvious predictions about effective emails are not borne out, despite there being evidence for further exploring these interventions, as they can sometimes motivate students to attempt their homework more often. We also present qualitative evidence about student opinions and behaviours after receiving the emails, to guide further interventions. These findings provide insight into how to use randomized A/B comparisons in everyday channels such as emails, to provide empirical evidence to test our beliefs about the effectiveness of alternative design choices.

Skills analysis is an interdisciplinary area that studies labor market trends and provides recommendations for developing educational standards and re-skilling efforts. We leverage techniques in this area to develop a scalable approach that identifies and evaluates educational competencies. In this work, we developed a skills extraction algorithm that uses natural language processing and machine learning techniques. We evaluated our algorithm on a labeled dataset and found its performance to be competitive with state-of-the-art methods. Using this algorithm, we analyzed student skills, university course syllabi, and online job postings. Our cross-sector analysis provides an initial landscape of skill needs for specific job titles. Additionally, we conducted a within-sector analysis based on programming jobs, computer science curriculum, and undergraduate students. Our findings suggest that students have a variety of hard skills and soft skills, but they are not necessarily the ones that employers want. The data also suggests these courses teach skills that are somewhat different from industry needs, and there is a lack of emphasis on soft skills. These results provide an initial assessment of the program competencies for a computer science program. Future work includes more data gathering, improving the algorithm, and applying our method to assess additional educational programs.

Writing from multiple sources is a commonly administered learning task across educational levels and disciplines. In this task, learners are instructed to comprehend information from source documents and integrate it into a coherent written composition to fulfil the assignment requirements. Even though educationally potent, multi-source writing tasks are considered challenging to many learners, in particular because many learners underuse monitoring and control, critical metacognitive processes for productive engagement in multi-source writing. To understand these processes, we conducted a laboratory study involving 44 university students. They engaged in multi-source writing task hosted in digital learning environment. Adding to previous research, we unobtrusively measured metacognitive processes using learners’ trace data collected via multiple data channels and in both writing and reading space of the multi-source writing task. We further investigated how these processes affect the quality of a written product, i.e., essay score. In the analysis, we utilised both automatically and human-generated essay score. The rating performance of the essay scoring algorithm was comparable to that of human raters. Our results largely support the theoretical assumptions that engagement in metacognitive monitoring and control benefits the quality of written product. Moreover, our results can inform the development of analytics-based tools that support student writing by making use of trace data and automated essay scoring.

Despite the growing prevalence of Massive Open Online Courses (MOOCs) in the last decade, using them effectively is still challenging. Particularly, when MOOCs involve teaching programming, learners often struggle with writing code without sufficient support, which may increase frustration, attrition, and eventually dropout. In this study, we assess the pedagogical design of a fresh introductory computer science MOOC. Keeping in mind MOOC “end-user” instructors, our analyses are based merely on features easily accessible from code submissions, and methods that are relatively simple to apply and interpret. Using visual data mining we discover common patterns of behavior, provide insights on content that may require reevaluation and detect critical points of attrition in the course timeline. Additionally, we extract students’ code submission profiles that reflect various aspects of engagement and performance. Consequently, we predict disengagement towards programming using classic machine learning methods. To the best of our knowledge, our definition for attrition in terms of disengagement towards programming is novel as it suits the unique active hands-on nature of programming. To our perception, the results emphasize that more attention and further research should be aimed at the pedagogical design of hands-on experience, such as programming, in online learning systems.

While learning analytics frameworks precede the official launch of learning analytics in 2011, there has been a proliferation of learning analytics frameworks since. This systematic review of learning analytics frameworks between 2011 and 2021 in three databases resulted in an initial corpus of 268 articles and conference proceeding papers based on the occurrence of “learning analytics” and “framework” in titles, keywords and abstracts. The final corpus of 46 frameworks were analysed using a coding scheme derived from purposefully selected learning analytics frameworks. The results found that learning analytics frameworks share a number of elements and characteristics such as source, development and application focus, a form of representation, data sources and types, focus and context. Less than half of the frameworks consider student data privacy and ethics. Finally, while design and process elements of these frameworks may be transferable and scalable to other contexts, users in different contexts will be best-placed to determine their transferability/scalability.

With the rise of the gig economy, online language tutoring platforms are becoming increasingly popular. They provide temporary and flexible jobs for native speakers as tutors and allow language learners to have one-on-one speaking practices on demand. However, the lack of stable relationships hinders tutors and learners from building long-term trust. “Distributed tutorship”—temporally discontinuous learning experience with different tutors—has been underexplored yet has many implications for modern learning platforms. In this paper, we analyzed tutorship sequences of 15,959 learners and found that around 40% of learners change to new tutors every session; 44% learners change to new tutors while reverting to previous tutors sometimes; only 16% learners change to new tutors and then fix on one tutor. We also found suggestive evidence that higher distributedness—higher diversity and lower continuity in tutorship—is correlated to slower improvements in speaking performance scores with a similar number of sessions. We further surveyed 519 and interviewed 40 learners and found that more learners preferred fixed tutorship while some do not have it due to various reasons. Finally, we conducted semi-structured interviews with three tutors and one product manager to discuss the implications for improving the continuity in learning under distributed tutorship.

One of the ultimate goals of several learning analytics (LA) initiatives is to close the loop and support students’ and teachers’ reflective practices. Although there has been a proliferation of end-user interfaces (often in the form of dashboards), various limitations have already been identified in the literature such as key stakeholders not being involved in their design, little or no account for sense-making needs, and unclear effects on teaching and learning. There has been a recent call for human-centred design practices to create LA interfaces in close collaboration with educational stakeholders to consider the learning design, and their authentic needs and pedagogical intentions. This paper addresses the call by proposing a question-driven LA design approach to ensure that end-user LA interfaces explicitly address teachers’ questions. We illustrate the approach in the context of synchronous online activities, orchestrated by pairs of teachers using audio-visual and text-based tools (namely Zoom and Google Docs). This study led to the design and deployment of an open-source monitoring tool to be used in real-time by teachers when students work collaboratively in breakout rooms, and across learning spaces.

Stakeholder engagement is a key aspect for the successful implementation of Learning Analytics (LA) in Higher Education Institutions (HEIs). Studies in Europe and Latin America (LATAM) indicate that, overall, instructors and students have positive views on LA adoption, but there are differences between their ideal expectations and what they consider realistic in the context of their institutions. So far, very little has been found about stakeholders’ views on LA in Brazilian higher education. By replicating the survey conducted in other countries, in seven Brazilian HEIs, we found convergences both with Europe and LATAM, reinforcing the need for local diagnosis and indicating the risk of assuming a ”LATAM identity”. Our findings contribute to building a corpus of knowledge on stakeholders expectations with a contextualised comprehension of the gaps between ideal and predicted scenarios, which can inform institutional policies for LA implementation in Brazil.

This study introduces a new analysis scheme to analyze trace data and visualize students’ self-regulated learning strategies in a mastery-based online learning modules platform. The pedagogical design of the platform resulted in fewer event types and less variability in student trace data. The current analysis scheme overcomes those challenges by conducting three levels of clustering analysis. On the event level, mixture-model fitting is employed to distinguish between abnormally short and normal assessment attempts and study events. On the module level, trace level clustering is performed with three different methods for generating distance metrics between traces, with the best performing output used in the next step. On the sequence level, trace level clustering is performed on top of module-level clusters to reveal students’ change of learning strategy over time. We demonstrated that distance metrics generated based on learning theory produced better clustering results than pure data-driven or hybrid methods. The analysis showed that most students started the semester with productive learning strategies, but a significant fraction shifted to a multitude of less productive strategies in response to increasing content difficulty and stress. The observations could prompt instructors to rethink conventional course structure and implement interventions to improve self-regulation at optimal times.

We investigated the generalizability of language-based analytics models across two collaborative problem solving (CPS) tasks: an educational physics game and a block programming challenge. We analyzed a dataset of 95 triads (N=285) who used videoconferencing to collaborate on both tasks for an hour. We trained supervised natural language processing classifiers on automatic speech recognition transcripts to predict the human-coded CPS facets (skills) of constructing shared knowledge, negotiation / coordination, and maintaining team function. We tested three methods for representing collaborative discourse: (1) deep transfer learning (using BERT), (2) n-grams (counts of words/phrases), and (3) word categories (using the Linguistic Inquiry Word Count [LIWC] dictionary). We found that the BERT and LIWC methods generalized across tasks with only a small degradation in performance (Transfer Ratio of .93 with 1 indicating perfect transfer), while the n-grams had limited generalizability (Transfer Ratio of .86), suggesting overfitting to task-specific language. We discuss the implications of our findings for deploying language-based collaboration analytics in authentic educational environments.

Learning Analytics (LA) dashboards have become a popular medium for communicating to teachers analytical insights obtained from student data. However, recent research indicates that LA dashboards can be complex to interpret, are often not grounded in educational theory, and frequently provide little or no guidance on how to interpret them. Despite these acknowledged problems, few suggestions have been made as to how we might improve the visual design of LA tools to support richer and alternative ways to communicate student data insights. In this paper, we explore three design alternatives to represent student multimodal data insights by combining data visualisation, narratives and storytelling principles. Based on foundations in data storytelling, three visual-narrative interfaces were designed with teachers: i) visual data slices, ii) a tabular visualisation, and iii) a written report. These were validated as a part of an authentic study where teachers explored activity logs and physiological data from co-located collaborative learning classes in the context of healthcare education. Results suggest that alternatives to LA dashboards can be considered as effective tools to support teachers’ reflection, and that LA designers should identify the representation type that best fits teachers’ needs.

From carefully crafted messages to flippant remarks, warm expressions to unfriendly grunts, teachers’ behaviors set the tone, expectations, and attitudes of the classroom. Thus, it is prudent to identify the ways in which teachers foster motivation, positive identity, and a strong sense of belonging through inclusive messaging and other interactions. We leveraged a new coding of teacher supportive discourse in 156 video clips from 73 6th to 8th grade math teachers from the archival Measures of Effective Teaching (MET) project. We trained Random Forest classifiers using verbal (words used) and paraverbal (acoustic-prosodic cues, e.g., speech rate) features to detect seven features of teacher discourse (e.g., public admonishment, autonomy supportive messages) from transcripts and audio, respectively. While both modalities performed over chance guessing, the specific language content was more predictive than paraverbal cues (mean correlation = .546 vs. .276); combining the two yielded no improvement. We examined the most predictive cues in order to gain a deeper understanding of the underlying messages in teacher talk. We discuss implications of our work for teacher analytics tools that aim to provide educators and researchers with insight into supportive discourse.

Feedback is a critical element of student-instructor interaction: it provides a direct manner for students to learn from mistakes. However, with student to teacher ratios growing rapidly, challenges arise for instructors to provide quality feedback to individual students. While significant efforts have been directed at automating feedback generation, relatively little attention has been given to underlying feedback characteristics. We develop a methodology for analyzing instructor-provided feedback and determining how it correlates with changes in student grades using data from online higher education engineering classrooms. Specifically, we featurize written feedback on individual assignments using Natural Language Processing (NLP) techniques including sentiment analysis, bigram splitting, and Named Entity Recognition (NER) to quantify post-, sentence-, and word-dependent attributes of grader writing. We demonstrate that student grade improvement can be well approximated by a multivariate linear model with average fits across course sections between 67% and 83%. We determine several statistically significant contributors to and detractors from student success contained in instructor feedback. For example, our results reveal that inclusion of student name is significantly correlated with an improvement in post-feedback grades, as is inclusion of specific assignment-related keywords. Finally, we discuss how this methodology can be incorporated into educational technology systems to make recommendations for feedback content from observed student behavior.

Learning Analytics (LA) is a bricolage field that requires a concerted effort to ensure that all stakeholders it affects are able to contribute to its development in a meaningful manner. We need mechanisms that support collaborative sense-making. This paper argues that graphical causal models can help us to span the disciplinary divide, providing a new apparatus to help educators understand, and potentially challenge, the technical models developed by LA practitioners as they form. We briefly introduce causal modelling, highlighting its potential benefits in helping the field to move from associations to causal claims, and illustrate how graphical causal models can help us to reason about complex statistical models. The approach is illustrated by applying it to the well known problem of at-risk modelling.

Learning analytics dashboards (LADs) are becoming more prevalent in higher education to help students, faculty, and staff make data-informed decisions. Despite extensive research on the design and implementation of LADs, few studies have investigated their relation to justice, equity, diversity, and inclusion (JEDI). Excluding these issues in LAD research limits the potential benefits of LADs generally and risks reinforcing long-standing inequities in education. We conducted a critical literature review, identifying 45 relevant papers to answer three research questions: how is LAD research improving JEDI, ii. how might it maintain or exacerbate inequitable outcomes, and iii. what opportunities exist in this space to improve JEDI in higher education. Using thematic analysis, we identified four common themes: (1) participant identities and researcher positionality, (2) surveillance concerns, (3) implicit pedagogies, and (4) software development resources. While we found very few studies directly addressing or mentioning JEDI concepts, we used these themes to explore ways researchers could consider JEDI in their studies. Our investigation highlights several opportunities to intentionally incorporate JEDI into LAD research by sharing software resources and conducting cross-border collaborations, better incorporating user needs, and centering considerations of justice in LAD efforts to improve historical inequities.

Higher education institutions increasingly rely on machine learning models. However, a growing body of evidence shows that these algorithms may not serve underprivileged communities well and at times discriminate against them. This is all the more concerning in education as negative outcomes have long-term implications. We propose a systematic process for framing, detecting, documenting, and reporting unfairness risks. The systematic approach’s outcomes are merged into a framework named FairEd, which would help decision-makers to understand unfairness risks along the environmental and analytical fairness dimension. The tool allows to decide (i) whether the dataset contains risks of unfairness; (ii) how the models could perform along many fairness dimensions; (iii) whether potentially unfair outcomes can be mitigated without degrading performance. The systematic approach is applied to a Chilean University case study, where a predicting student dropout model is aimed to build. First, we capture the nuances of the Chilean context where unfairness emerges along income lines and demographic groups. Second, we highlight the benefit of reporting unfairness risks along a diverse set of metrics to shed light on potential discrimination. Third, we find that measuring the cost of fairness is an important quantity to report on when doing the model selection.

Existing research indicates that students prefer to work with tutors who express politely in online human-human tutoring, but excessive polite expressions might lower tutoring efficacy. However, there is a shortage of understanding about the use of politeness in online tutoring and the extent to which the politeness of instructional strategies can contribute to students’ achievement. To address these gaps, we conducted a study on a large-scale dataset (5,165 students and 116 qualified tutors in 18,203 online tutoring sessions) of both effective and ineffective human-human online tutorial dialogues. The study made use of a well-known dialogue act coding scheme to identify instructional strategies, relied on the linguistic politeness theory to analyse the politeness levels of the tutors’ instructional strategies, and utilised Gradient Tree Boosting to evaluate the predictive power of these politeness levels in revealing students’ problem-solving performance. The results demonstrated that human tutors used both polite and non-polite expressions in the instructional strategies. Tutors were inclined to express politely in the strategy of providing positive feedback but less politely while providing negative feedback and asking questions to evaluate students’ understanding. Compared to the students with prior progress, tutors provided more polite open questions to the students without prior progress but less polite corrective feedback. Importantly, we showed that, compared to previous research, the accuracy of predicting student problem-solving performance can be improved by incorporating politeness levels of instructional strategies with other documented predictors (e.g., the sentiment of the utterances).

This study presents a novel video recommendation system for an algebra virtual learning environment (VLE) that leverages ideas and methods from engagement measurement, item response theory, and reinforcement learning. Following Vygotsky's Zone of Proximal Development (ZPD) theory, but considering low affect and high affect students separately, we developed a system of five categories of video recommendations: 1) Watch new video; 2) Review current topic video with a new tutor; 3) Review segment of current video with current tutor; 4) Review segment of current video with a new tutor; 5) Watch next video in curriculum sequence. The category of recommendation was determined by student scores on a quiz and a sensor-free engagement detection model. New video recommendations (i.e., category 1) were selected based on a novel reinforcement learning algorithm that takes input from an item response theory model. The recommendation system was evaluated in a large field experiment, both before and after school closures due to the COVID-19 pandemic. The results show evidence of effectiveness of the video recommendation algorithm during the period of normal school operations, but the effect disappears after school closures. Implications for teacher orchestration of technology for normal classroom use and periods of school closure are discussed.

Spacing and procrastination are often thought of as opposites. It is possible, however, for a student to space their studying by doing something every day throughout the semester and still procrastinate by waiting until late in the semester to increase their amount of studying. To analyze the relationship between spacing and procrastination, we examined 674 students’ interactions with a course eBook over four semesters of an introductory programming course. We measured each student’s semester-level spacing as the number of days they interacted with the eBook, and each student’s semester-level procrastination as the average delay from the start of the semester for all their eBook interactions. Surprisingly, there was a small, yet positive, correlation between the two measures. Which, then, matters for course performance: studying over more days or studying earlier in the semester? When controlling for total amount of studying, as well as a number of academic and demographic characteristics in an SEM analysis, we find a strong positive effect of spacing but no significant effect of procrastination on final exam scores.

This research aimed to explore the relationship between different indicators of the depth and quality of participation in computer-mediated learning environments. By using network analyses and statistical tests, we discovered significant associations between the cognitive presence phases of the Community of Inquiry framework and speech acts, and examined the impact of two different instructional interventions on these associations. We found that there are strong associations between some speech acts and cognitive presence phases. In addition, the study revealed that the association between speech acts and cognitive presence is moderated by external facilitation, but not affected by user role assignment. The results suggest that speech acts can plausibly be used to provide feedback in relation to cognitive presence and can potentially be used to increase the generalizability of cognitive presence classification.

Temporal analysis has been demonstrated to be relevant in Learning Analytics research, and capturing time-on-task, i.e., the amount of time spent by students in quality learning, as a proxy to model learning behaviour, predict performance, and avoid drop-out has been the focus of a number of investigations. Nonetheless, most studies do not provide enough information on how their data were prepared for their findings to be easily replicated, even though data pre-processing decisions have an impact on the analysis’ outcomes and can lead to inaccurate predictions. One of the key aspects in the preparation of learning data for temporal analysis is the detection of anomalous values of temporal duration of students’ activities. Most of the works in the literature address this problem without taking into account the fact that different activities can have very different typical execution times. In this paper, we propose a methodology for estimating time-on-task that starts with a well-defined data consolidation and then applies an outlier detection strategy to the data based on a distinct study of each learning activity and its peculiarities. Our real-world data experiments show that the proposed methodology outperforms the current state of the art, providing more accurate time estimations for students’ learning tasks.

Twitter is a very popular microblogging platform that has been actively used by scientific communities to exchange scientific information and to promote scholarly discussions. The present study aimed to leverage the tweet data to provide valuable insights into the development of the learning analytics field since its initial days. Descriptive analysis, geocoding analysis, and topic modeling were performed on over 1.6 million tweets related to learning analytics posted between 2010-2021. The descriptive analysis reveals an increasing popularity of the field on the Twittersphere in terms of number of users, twitter posts, and hashtags emergence. The topic modeling analysis uncovers new insights of the major topics in the field of learning analytics. Emergent themes in the field were identified, and the increasing (e.g., Artificial Intelligence) and decreasing (e.g., Education) trends were shared. Finally, the geocoding analysis indicates an increasing participation in the field from more diverse countries all around the world. Further findings are discussed in the paper.

Collaboration is one of the four important 21st-century skills. With the pervasive use of sensors, interest on co-located collaboration (CC) has increased lately. Most related literature used the audio modality to detect indicators of collaboration (such as total speaking time and turn taking). CC takes place in physical spaces where group members share their social (i.e., non-verbal audio indicators like speaking time, gestures) and epistemic space (i.e., verbal audio indicators like the content of the conversation). Past literature has mostly focused on the social space to detect the quality of collaboration. In this study, we focus on both social and epistemic space with an emphasis on the epistemic space to understand different evolving collaboration patterns and collaborative convergence and quantify collaboration quality. We conduct field trials by collecting audio recordings in 14 different sessions in a university setting while the university staff and students collaborate over playing a board game to design a learning activity. This collaboration task consists of different phases with each collaborating member having been assigned a pre-fixed role. We analyze the collected group speech data to do role-based profiling and visualize it with the help of a dashboard.

In co-located situations, team members use a combination of verbal and visual signals to communicate effectively, among which positional forms play a key role. The spatial patterns adopted by team members in terms of where in the physical space they are standing, and who their body is oriented to, can be key in analysing and increasing the quality of interaction during such face-to-face situations. In this paper, we model the students’ communication based on spatial (positioning) and audio (voice detection) data captured from 92 students working in teams of four in the context of healthcare simulation. We extract non-verbal events (i.e., total speaking time, overlapped speech,and speech responses to team members and teachers) and investigate to what extent they can serve as meaningful indicators of students’ performance according to teachers’ learning intentions. The contribution of this paper to multimodal learning analytics includes: i) a generic method to semi-automatically model communication in a setting where students can freely move in the learning space; and ii) results from a mixed-methods analysis of non-verbal indicators of team communication with respect to teachers’ learning design.

Academic analytics focuses on collecting, analysing and visualising educational data to generate institutional insights and improve decision-making for academic purposes. However, challenges that arise from navigating a complex organisational structure when introducing analytics systems have called for the need to engage key stakeholders widely to cultivate a shared vision and ensure that implemented systems create desired value. This paper presents a study that takes co-design steps to identify design needs and strategic approaches for the adoption of academic analytics, which serves the purpose of enhancing the measurement of educational quality utilising institutional data. Through semi-structured interviews with 54 educational stakeholders at a large research university, we identified particular interest in measuring student engagement and the performance of courses and programmes. Based on the observed perceptions and concerns regarding data use to measure or evaluate these areas, implications for adoption strategy of academic analytics, such as leadership involvement, communication, and training, are discussed.

Self-regulated learning (SRL) skills are essential for successful learning in a technology-enhanced learning environment. Learning Analytics techniques have shown a great potential in identifying and exploring SRL strategies from trace data in various learning environments. However, these strategies have been mainly identified through analysis of sequences of learning actions, and thus interpretation of the strategies is heavily task and context dependent. Further, little research has been done on the association of SRL strategies with different influencing factors or conditions. To address these gaps, we propose an analytic method for detecting SRL strategies from theoretically supported SRL processes and applied the method to a dataset collected from a multi-source writing task. The detected SRL strategies were explored in terms of their association with the learning outcome, internal conditions (prior-knowledge, metacognitive knowledge and motivation) and external conditions (scaffolding). The study results showed our analytic method successfully identified three theoretically meaningful SRL strategies. The study results revealed small effect size in the association between the internal conditions and the identified SRL strategies, but revealed a moderate effect size in the association between external conditions and the SRL strategy use.

Brazilian universities have included essay writing assignments in the entrance examination procedure to select prospective students. The essay scorers manually look for the presence of required Rhetorical Structure Theory (RST) categories and evaluate essay coherence. However, identifying RST categories is a time-consuming task. The literature reported several attempts to automate the identification of RST categories in essays with machine learning. Still, previous studies have focused on using machine learning algorithms trained on content-dependent features that can diminish classification performance, leading to over-fitting and hindering model generalisability. Therefore, this paper proposes: (i) the analysis of state-of-the-art classifiers and content-independent features to the task of RST rhetorical moves; (ii) a new approach that considers the sequence of the text to extract features – i.e. sequential content-independent features; (iii) an empirical study about the generalisability of the machine learning models and sequential content-independent features for this context; (iv) the identification of the most predictive features for automated identification of RST categories in essays written in Portuguese. The best performing classifier, XGBoost, based on sequential content-independent features, outperformed the classifiers used in the literature and are based on traditional content-dependent features. The XGBoost classifier based on sequential content-independent features also reached promising accuracy when tested for generalisability.

Roles that learners assume during online discussions are an important aspect of educational experience. The roles can be assigned to learners and/or can spontaneously emerge through student-student interaction. While existing research proposed several approaches for analytics of emerging roles, there is limited research in analytic methods that can i) automatically detect emerging roles that can be interpreted in terms of higher-order constructs of collaboration; ii) analyse the extent to which students complied to scripted roles and how emerging roles compare to scripted ones; and iii) track progression of roles in social knowledge progression over time. To address these gaps in the literature, this paper propose a network-analytic approach that combines techniques of cluster analysis and epistemic network analysis. The method was validated in an empirical study discovered emerging roles that were found meaningful in terms of social and cognitive dimensions of the well-known model of communities of inquiry. The study also revealed similarities and differences between emerging and script roles played by learners and identified different progression trajectories in social knowledge construction between emerging and scripted roles. The proposed analytic approach and the study results have implications that can inform teaching practice and development techniques for collaboration analytics.

In language learning, getting corrective feedback for writing tasks is an essential didactical concept to improve learners' language skills. Although various tools for automatic correction do exist, open writing texts still need to be corrected manually by teachers to provide helpful feedback to learners. In this paper, we explore the usefulness of an auto-correction tool in the context of language learning. In the first step, we compare the corrections of 100 learner texts suggested by a correction tool with those done by human teachers and examine the differences. In a second step, we do a qualitative analysis, where we investigate the requirements that need to be tackled to make existing proofreading tools useful for language learning. The results reveal that the aim of enhancing texts by proofreading, in general, is quite different from the purpose of providing corrective feedback in language learning. Only one of four relevant errors (recall=.26) marked by human teachers is recorded correctly by the tool, whereas many expressions thought to be faulty by the tool are sometimes no errors at all (precision=.33). We provide and discuss the challenges that need to be addressed to adjust those tools for language learning.

Discussion forums are important for students’ knowledge inquiry in online contexts, with help-seeking being an essential learning strategy in discussion forums. This study aimed to explore innovative methods to build a peer recommender that can provide fair and accurate intelligence to support help-seeking in online learning. Specifically, we have examined existing network embedding models, Node2Vec and FairWalk, to benchmark with the proposed fair network embedding (Fair-NE). A dataset of 187,450 post-reply pairs by 10,182 Algebra I students from 2015 to 2020 was sampled from Algebra Nation, an online algebra learning platform. The dataset was used to train and evaluate the engines of peer recommenders. We evaluated models with representation fairness, predictive accuracy, and predictive fairness. Our findings suggest that constructing fairness-aware models in learning analytics (LA) is crucial to tackling the potential bias in data and to creating trustworthy LA systems.

Programming education has received extensive attention in recent years due to the increasing demand for programming ability in almost all industries. Educational institutions have widely employed online judges for programming training, which can help teachers automatically assess programming assignments by executing students’ code with test cases. However, a more important teaching process with online judges should be to evaluate how students master each of the programming skills such as strings or pointers, so that teachers may give personalized feedback and help them proceed to the success more efficiently. Previous studies have adopted deep models of knowledge tracing to evaluate a student’s mastery level of skills during the interaction with programming exercises. However, existing models generally follow the conventional assumption of knowledge tracing that each programming exercise requires only one skill, whereas in practice a programming exercise usually inspects the comprehensive use of multiple skills. Moreover, the feature of student code is often simply concatenated with other input features without the consideration of its relationship with the inspected programming skills. To bridge the gap, we propose a simple attention-based approach to learn from student code the features reflecting the multiple programming skills inspected by each programming exercise. In particular, we first use a program embedding method to obtain the representations of student code. Then we use the skill embeddings of each programming exercise to query the embeddings of student code and form an aggregated hidden state representing how the inspected skills are used in the student code. We combine the learned hidden state with DKT (Deep Knowledge Tracing), an LSTM (Long Short-Term Memory)-based knowledge tracing model, and show the improvements over baseline model. We point out some possible directions to improve the current work.

Being a promising constructionist pedagogy in recent years, maker education empowers students to take agency of their learning process through constructing both knowledge and real-world physical or digital products and fosters peer interactions for collective innovation. Learning Analytics (LA) excels at generating personalized, fine-grained feedback in near real-time and holds much potential in supporting process-oriented and peer-supported learning activities, including maker activities. In the context of virtual reality (VR) content creation for cultural heritage education, this study qualitatively solicited 27 students’ needs on progress monitoring, reflection, and feedback during their making process. Findings have inspired the prototype design of a student-facing LA dashboard (LAVR). Leveraging multimodal learning analytics (MmLA) such as text and audio analytics to fulfill students’ needs, the prototype has various features and functions including automatic task reminders, content quality detection, and real-time feedback on quality of audio-visual elements. A preliminary evaluation of the prototype with 10 students confirms its potential in supporting students’ self-regulated learning during the making process and for improving the quality of VR content. Implications on LA design for supporting maker education are discussed. Future work is planned to include implementation and evaluation of the dashboard in classrooms.

In times of pandemic-induced challenges, virtual reality (VR) allows audience to learn about cultural heritage sites without temporal and spatial constraints. The design of VR content is largely determined by professionals, while evaluations of content often rely on learners’ self-report data. Learners’ attentional focus and understanding of VR content might be affected by the presence or absence of different multimedia elements including text and audio-visuals. It remains an open question which design variations are more conducive for learning about heritage sites. Leveraging eye-tracking, a technology often adopted in recent multimodal learning analytics (MmLA) research, we conducted an experiment to collect and analyze 40 learners’ eye movement and self-reported data. Results of statistical tests and heatmap elicitation interviews indicate that 1) text in the VR environment helped learners better understand the presented heritage sites, regardless of having audio narration or not, 2) text diverted learners’ attention away from other visual elements that contextualized the heritage sites, 3) exclusively having audio narration best simulated the experience of a real-world heritage tour, 4) narration accompanying text prompted learners to read the text faster. We make recommendations for improving the design of VR learning materials and discuss the implications for MmLA research.

Explainable recommendations, which provide explanations about why an item is recommended, help to improve the transparency, persuasiveness, and trustworthiness. However, few research in educational technology utilize explainable recommendations. We developed an explanation generator using the parameters from Bayesian knowledge tracing models. We used this educational explainable recommendation system to investigate the effects of explanation on the summer vacation assignment for high school students. Comparing the click counts of recommended quizzes with and without explanations, we found that the number of clicks was significantly higher for quizzes with explanations. Furthermore, system usage pattern mining revealed that students can be divided to three clusters— none, steady and late users. In the cluster of steady users, recommended quizzes with explanations were continuously used. These results suggest the effectiveness of an explainable recommendation system in the field of education.

Analyzing classroom video data provides valuable insights about the interactions between students and teachers, albeit often through time-consuming qualitative coding or the use of bespoke sensors to record individual movement information. We explore measuring classroom posture and movement in secondary classroom video data through computer vision methods (especially OpenPose), and introduce a simple but effective approach to automatically track movement via post-processing of OpenPose output data. Analysis of 67 videos of mathematics classes from middle school and high school levels highlighted the challenges associated with analyzing movement in typical classroom videos: occlusion from low camera angles, difficulty detecting lower body movement due to sitting, and the close proximity of students to one another and their teachers. Despite these challenges, our approach tracked person IDs across classroom videos for 93.0% of detected individuals. The tracking results were manually verified through randomly sampling 240 instances, which revealed notable OpenPose tracking inconsistencies. Finally, we discuss the implications for supporting more scalability of video data classroom movement analysis, and future potential explorations.

Self-directed learning (SDL) is an important skill in the 21st century, while the understanding of its process in behavior has not been well explored. Analysis of the sequential behavior patterns in SDL and the relations with students’ academic performance could help to advance our understanding of SDL in theory and practice. In this study, we mined the behavioral sequences of self-directed extensive reading from students’ learning and self-directed behavioral logs using differential pattern mining technique. Furthermore, we built models to predict students’ academic performance using the conventional behavior frequency features and the behavior sequence features. Experimental results identified 14 sequential patterns of SDL behaviors in the high-performance student group. The prediction model revealed the importance of sequential patterns in SDL behavior, which was built with an acceptable AUC. These findings suggested that several SDL strategies in behavior contribute to students’ academic performance, such as analysis learning status before planning, planning before learning, monitoring after learning.

Providing formative feedback to foster collaboration and improve students’ practice has been an emerging topic in CSCL and LA research communities. However, this pedagogical practice could be unrealistic in authentic classrooms, as observing and annotating improvements for every student and group exceeds the teacher’s capabilities. In the research area of group work and collaborative learning, current learning analytics solutions have reported accurate computational models to understand collaboration processes, yet evaluating formative collaboration feedback, where the final user is the student, is an under-explored research area. This paper reports an exploratory evaluation to understand the effects a collaboration feedback report through an authentic study conducted in regular classes. Fifty students from a Computer Science undergraduate program participated in the study. We followed an user-centered design approach to define six collaboration aspects that are relevant to students. These aspects were part of initial prototypes for the feedback report. From the exploratory intervention, we did not find effects between students who received the feedback (experimental condition) report and those who did not (control condition). Finally, this paper discusses design implications for further feedback report designs and interventions.

It is difficult for instructors, and even students themselves, to become aware in real-time of inequitable behaviors occurring on student teams. Here, we explored a potential measure for inequitable teamwork drawing on data from a digital pedagogical tool designed to surface and disrupt such team behaviors. Students in a large, undergraduate business course completed seven surveys about team health (called team checks) at regular intervals throughout the term, providing information about team dynamics, contributions, and processes. The ways in which changes in students’ scores from team check to team check compared to the median changes for their team were used to identify the proportions of teams with outlier student scores. The results show that for every team size and team check item, the proportion of teams with outliers at the end of the term was smaller than at the beginning of the semester, indicating stabilization in how teammates evaluated their team experiences. In all but two cases, outlying students were not disproportionately likely to identify with historically marginalized groups based on gender or race/ethnicity. Thus, we did not broadly identify teamwork inequities in this specific context, but the method provides a basis for future studies about inequitable team behavior.

Writing-to-learn pedagogies are an evidence-based practice known to aid students in constructing knowledge. Barriers exist for the implementation of such assignments; namely, instructors feel they do not have time to provide each student with feedback. To ease implementation of writing-to-learn assignments at scale, we have incorporated automated peer review, which facilitates peer review without input from the instructor. Participating in peer review can positively impact students’ learning and allow students to receive feedback on their writing. Instructors may want to monitor these peer interactions and gain insight into their students’ understanding using the feedback generated by their peers. To facilitate instructors’ use of the content from students’ peer review comments, we pre-trained a transformer model called PeerBERT. PeerBERT was fine-tuned on several downstream tasks to categorize students’ peer review comments as praise, problem/solution, or verification/summary. The model exhibits high accuracy, even across different peer review prompts, assignments, and courses. Additional downstream tasks label problem/solution peer review comments as one or more types: writing/formatting, missing content/needs elaboration, and incorrect content. This approach can help instructors pinpoint common issues in student writing by parsing out which comments are problem/solution and which type of problem/solution students identify.

Effective collaborative discourse requires both cognitive and social engagement of students. To investigate complex socio-cognitive dynamics in collaborative discourse, this paper proposes to model collaborative discourse as a socio-semantic network (SSN) and then use network motifs – defined as recurring, significant subgraphs – to characterize the network and hence the discourse. To demonstrate the utility of our SSN motifs framework, we applied it to a sample dataset. While more work needs to be done, the SSN motifs framework shows promise as a novel, theoretically informed approach to discourse analysis.

This paper proposes an overarching framework for automated collaboration feedback that bridges theory and tool as well as technology and pedagogy. This pragmatic and theory-driven framework guides our thinking by outlining the components involved in converting theoretical collaboration constructs into features that can be automatically extracted and then converted into actionable feedback. Focusing on the pedagogical components of the framework, the constructs are validated by mapping them onto a selection of multi-disciplinary collaboration frameworks. The resulting behavioral indicators are then applied to measure collaboration in a sample scenario and those measurements are then used to exemplify how feedback analytics could be calculated. The paper concludes with a discussion on how those analytics could be converted into feedback for students and the next steps needed to advance the technological part of the framework.

This paper describes a process for operationally defining productive struggle in a widely used digital learning environment called ST Math. The process for developing an operational definition involved examining the existing literature for ways in which researchers have previously quantified productive struggle in digital learning environments. Using prior research, we defined productive struggle as a student persisting in a digital learning task while maintaining a likelihood of future success. To develop a machine-executable definition of productive struggle, we identified the typical number of attempts learners needed to complete a level in ST Math and applied a modified Performance Factors Analysis algorithm to estimate learners’ probability of success on a subsequent puzzle attempt within a level. Using definitions that differentially combined re-attempts and predicted probabilities, we examined the proportion of level attempts that could be newly classified as instances of productive struggle. The pragmatic approach described in this paper is intended to serve as an example for other digital learning environments seeking to develop indicators of productive struggle.

Without a sense of belonging, students may become disheartened and give up when faced with new challenges. Moreover, with the sudden growth of remote learning due to COVID-19, it may be even more difficult for students to feel connected to the course and peers in isolation. Therefore, we propose a recommendation system to build connections between students while recommending solutions to challenges. This pilot system utilizes students’ reflections from previous semesters, asking about learning challenges and potential solutions. It then generates sentence embeddings and calculates cosine similarities between the challenges of current and prior students. The possible solutions given by previous students are then recommended to present students with similar challenges. Self-reflection encourages students to think deeply about their learning experiences and benefit both learners and instructors. This system has the potential to allow reflections also to help future learners. By demonstrating that previous students encountered and overcame similar challenges, we could help improve students’ sense of belonging. We then perform user studies to evaluate this system’s potential and find that participants rated 70% of the recommended solutions as useful. Our findings suggest an increase in students’ sense of membership and acceptance, and a decrease in the desire to withdraw.

This study addresses the gap in knowledge about differences in how instructors use analytics to inform teaching by examining the ways that thirteen college instructors engaged with a set of university-provided analytics. Using multiple walk-through interviews with the instructors and qualitative inductive coding, two profiles of instructor analytics use were identified that were distinct from each other in terms of the goals of analytics use, how instructors made sense of and took actions upon the analytics, and the ways that ethical concerns were conceived. Specifically, one group of instructors used analytics to help students get aligned to and engaged in the course, whereas the other group used analytics to align the course to meet students’ needs. Instructors in both profiles saw ethical questions as central to their learning analytics use, with instructors in one profile focusing on transparency and the other on student privacy and agency. These findings suggest the need to view analytics use as an integrated component of instructor teaching practices and envision complementary sets of technical and pedagogical support that can best facilitate the distinct activities aligned with each profile.

Participatory Design (PD), and Co-design (Co-D), can be effective ways to improve technological innovation and to incorporate users’ needs in the development of learning analytics (LA). However, these methods can be difficult to implement and there has yet to be a synopsis of how its and techniques have been applied to the specific needs of LA. This study reviewed 90 papers that described 52 cases of PD of LA between 2010 and 2020 to address the research question “How is participatory design (PD) being used within LA?”. It focuses on examining which groups of participants are normally included in PD for LA, in what phases of the design process it is used, and what specific tools and techniques have LA designers adapted or developed to co-create with design partners. Findings show that there is a growing number of researchers using these methods in recent years, particularly in higher education and with instructor stakeholders. However, it was also found that often the literature would describe the PD activities only superficially, and that some aspects of PD, such as recruitment, were seldom considered overtly in the descriptions of these processes.

The benefits of incorporating scaffolds that promote strategies of self-regulated learning (SRL) to help student learning are widely studied and recognised in the literature. However, the best methods for incorporating them in educational technologies and empirical evidence about which scaffolds are most beneficial to students are still emerging. In this paper, we report our findings from conducting an in-the-field controlled experiment with 797 post-secondary students to evaluate the impact of incorporating scaffolds for promoting SRL strategies in the context of assisting students in creating novel content, also known as learnersourcing. The experiment had five conditions, including a control group that had access to none of the scaffolding strategies for creating content, three groups each having access to one of the scaffolding strategies (planning, externally-facilitated monitoring and self-assessing) and a group with access to all of the aforementioned scaffolds. The results revealed that the addition of the scaffolds for SRL strategies increased the complexity and effort required for creating content, were not positively assessed by learners and led to slight improvements in the quality of the generated content. We discuss the implications of our findings for incorporating SRL strategies in educational technologies.

Handwritten notes are one important component of students’ learning process, which is used to record what they have learned in class or tease out knowledge after class for reflection and further strengthen the learning effect. It also helps a lot during review. We hope to divide handwritten notes (Japanese) into different parts, such as text, mathematical expressions, charts, etc., and quantify them to evaluate the condition of the notes and compare them among students. At the same time, data on students’ learning behaviors in the course are collected through the online education platform, such as the use time of textbook and attendance, as well as the scores of the online quiz and course grade. In this paper, the analysis of the relationship between the segmentation results of handwritten notes and learning behavior are reported, as well as the research on automatic page segmentation based on deep learning.

Work throughout the learning analytics community has examined associations between Learning Analytics Dashboard (LAD) features and a number of important student outcomes, including academic motivation and self-regulated learning strategies. While there are many potential implications of visualized academic information within a LAD on student outcomes, there remains an unanswered question: are there causal differences between showing performance information (e.g., comparing students’ progress to the class average) vs. mastery information (e.g., their individual score) on students’ motivation?

Grounded in Achievement Goal Theory, this study answers this question experimentally by analyzing the difference between college students’ (n=445) reported achievement goal orientations as well as their motivated information seeking orientations after being presented with performance or mastery feedback. Results indicate that students in a performance condition which displayed ”above average” achievement on an academic measure reported lower performance-avoidance goals (e.g., not wanting to do worse than everyone else), and performance-avoidance information-seeking goals (e.g., not wanting to seek out information showing that one does worse than peers) when compared to students in the mastery control condition.

This study contributes to our understanding of the motivational implications of academic feedback presented to students, and suggests that comparative information has direct effects on student motivation. Results thus uncover a potential tension between what might seem intuitive feedback to give students versus what might be more motivationally appropriate. The implications of this work point to the need to understand LADs not simply as feedback mechanisms, but as embedded features of a learning environment that influence how students engage with course content.

Informal learning procedures have been changing extremely fast over the recent decades not only due to the advent of online learning, but also due to changes in what humans need to learn to meet their various life and career goals. Consequently, online, educational platforms are expected to provide personalized, up-to-date curricula to assist learners. Therefore, in this paper, we propose an Artificial Intelligence (AI) and Crowdsourcing based approach to create and update curricula for individual learners. We show the design of this curriculum development system prototype, in which contributors receive AI-based recommendations to be able to define and update high-level learning goals, skills, and learning topics together with associated learning content. This curriculum development system was also integrated into our personalized online learning platform. To evaluate our prototype we compared experts’ opinion with our system’s recommendations, and resulted in 89%, 79%, and 93% F1-scores when recommending skills, learning topics, and educational materials respectively. Also, we interviewed eight senior level experts from educational institutions and career consulting organizations. Interviewees agreed that our curriculum development method has high potential to support authoring activities in dynamic, personalized learning environments.

Concept maps are used in education to illustrate ideas and relationships among them. Instructors employ such maps to evaluate a student’s knowledge on a subject. Collective concept maps have been recently proposed as a tool to graphically summarize a group’s rather than an individual’s understanding on a topic. In this paper, we present a methodology that automatically generates collective concept maps, which relies on grouping similar ideas into node-clusters. We present a novel clustering algorithm that is shown to produce more informational maps compared to Markov clustering. We evaluate the collective map framework by applying it to sets of a total of 56 individual maps created by teachers (grades 2-12) and students (grades 6-11) during a week-long cybersecurity camp. Finally, we discuss how collective concept maps can support longitudinal research studies on program and student outcomes by providing a novel format for knowledge exchange. We have made our tool implementation publicly available.

Ethics and privacy issues have been recognized as important factors for acceptance and trustworthy implementation of learning analytics. A large number of different issues has been recognized in the literature. Guidelines related to these issues are continuously being developed and discussed in research literature. The aim of this research was to identify patterns of co-occurrence of issues and guidelines in research papers discussing ethics and privacy issues, to gain better understanding of relationships between different ethics and privacy issues arising during implementation of learning analytics in higher education. A total of 93 papers published between 2010 and 2021 were qualitatively analyzed, and nine categories of issues and respective guidelines related to ethics and privacy in learning analytics were identified. Association rules mining Apriori algorithm was applied, where 93 papers represented transactions, and 18 categories of issues or guidelines (nine each) represented items. Two clusters of issues co-occurring in papers were identified, corresponding to deontology ethics (related to rules and duties), and consequentialism ethics (related to consequences of unethical behavior).