Chapter 10

Handbook of Learning Analytics
First Edition

Emotional Learning Analytics

Sidney K. D’Mello


This chapter discusses the ubiquity and importance of emotion to learning. It argues that substantial progress can be made by coupling the discovery-oriented, data-driven, analytic methods of learning analytics (LA) and educational data mining (EDM) with theoretical advances and methodologies from the affective and learning sciences. Core, emerging, and future themes of research at the intersection of these areas are discussed.

Export Citation: Plain Text (APA)     BIBTeX     RIS

Supplementary Material
No Supplementary Material Available
References (81)

Aguiar, E., Ambrose, G. A. A., Chawla, N. V., Goodrich, V., & Brockman, J. (2014). Engagement vs. performance: Using electronic portfolios to predict first semester engineering student persistence. Journal of Learning Analytics, 1(3), 7–33.

Ai, H., Litman, D. J., Forbes-Riley, K., Rotaru, M., Tetreault, J. R., & Pur, A. (2006). Using system and user performance features to improve emotion detection in spoken tutoring dialogs. Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech 2006).

Arroyo, I., Woolf, B., Cooper, D., Burleson, W., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. In V. Dimitrova, R. Mizoguchi, B. Du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 17–24). Amsterdam: IOS Press.

Baker, R., & Ocumpaugh, J. (2015). Interaction-based affect detection in educational software. In R. Calvo, S. D’Mello, J. Gratch, & A. Kappas (Eds.), The Oxford handbook of affective computing (pp. 233–245). New York: Oxford University Press.

Barth, C. M., & Funke, J. (2010). Negative affective environments improve complex solving performance. Cognition and Emotion, 24(7), 1259–1268. doi:10.1080/02699930903223766

Blanchard, N., Donnelly, P., Olney, A. M., Samei, B., Ward, B., Sun, X., . . . D’Mello, S. K. (2016). Identifying teacher questions using automatic speech recognition in live classrooms. Proceedings of the 17th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL 2016) (pp. 191–201). Association for Computational Linguistics.

Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium. New York: ACM.

Bosch, N., Chen, H., Baker, R., Shute, V., & D’Mello, S. K. (2015). Accuracy vs. availability heuristic in multimodal affect detection in the wild. Proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI 2015). New York: ACM.

Bosch, N., & D’Mello, S. K. (in press). The affective experience of novice computer programmers. International Journal of Artificial Intelligence in Education.

Bosch, N., D’Mello, S., Baker, R., Ocumpaugh, J., & Shute, V. (2016). Using video to automatically detect learner affect in computer-enabled classrooms. ACM Transactions on Interactive Intelligent Systems, 6(2), doi:10.1145/2946837.

Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37. doi:10.1109/T-AFFC.2010.1

Clore, G. L., & Huntsinger, J. R. (2007). How emotions inform judgment and regulate thought. Trends in Cognitive Sciences, 11(9), 393–399. doi:10.1016/j.tics.2007.08.005

Conati, C., & Maclaren, H. (2009). Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19(3), 267–303.

Corbett, A., & Anderson, J. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278.

Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper & Row.

D’Mello, S. K. (2016). On the influence of an iterative affect annotation approach on inter-observer and self-observer reliability. IEEE Transactions on Affective Computing, 7(2), 136–149.

D’Mello, S. K., Blanchard, N., Baker, R., Ocumpaugh, J., & Brawner, K. (2014). I feel your pain: A selective review of affect-sensitive instructional strategies. In R. Sottilare, A. Graesser, X. Hu, & B. Goldberg (Eds.), Design recommendations for adaptive intelligent tutoring systems: Adaptive instructional strategies (Vol. 2, pp. 35–48). Orlando, FL: US Army Research Laboratory.

D’Mello, S., Craig, S., Sullins, J., & Graesser, A. (2006). Predicting affective states expressed through an emote-aloud procedure from AutoTutor’s mixed-initiative dialogue. International Journal of Artificial Intelligence in Education, 16(1), 3–28.

D’Mello, S., & Graesser, A. (2012a). AutoTutor and Affective AutoTutor: Learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Transactions on Interactive Intelligent Systems, 2(4), 23:22–23:39.

D’Mello, S., & Graesser, A. (2012b). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. doi:10.1016/j.learninstruc.2011.10.001

D’Mello, S., & Graesser, A. (2012c). Malleability of students’ perceptions of an affect-sensitive tutor and its influence on learning. In G. Youngblood & P. McCarthy (Eds.), Proceedings of the 25th Florida Artificial Intelligence Research Society Conference (pp. 432–437). Menlo Park, CA: AAAI Press.

D’Mello, S., & Graesser, A. (2014a). Inducing and tracking confusion and cognitive disequilibrium with breakdown scenarios. Acta Psychologica, 151, 106–116.

D’Mello, S. K., & Graesser, A. C. (2014b). Confusion. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 289–310). New York: Routledge.

D’Mello, S. K., & Kory, J. (2015). A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys, 47(3), 43:41–43:46.

D’Mello, S., Lehman, B., & Person, N. (2010). Monitoring affect states during effortful problem solving activities. International Journal of Artificial Intelligence in Education, 20(4), 361–389.

D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., . . . Graesser, A. (2010). A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In J. Kay & V. Aleven (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (pp. 245–254). Berlin/Heidelberg: Springer.

D’Mello, S., & Mills, C. (2014). Emotions while writing about emotional and non-emotional topics. Motivation and Emotion, 38(1), 140–156.

D’Mello, S., Taylor, R. S., & Graesser, A. (2007). Monitoring affective trajectories during complex learning. In D. McNamara & J. Trafton (Eds.), Proceedings of the 29th Annual Cognitive Science Society (pp. 203–208). Austin, TX: Cognitive Science Society.

Darwin, C. (1872). The expression of the emotions in man and animals. London: John Murray.

Dillon, J., Bosch, N., Chetlur, M., Wanigasekara, N., Ambrose, G. A., Sengupta, B., & D’Mello, S. K. (2016). Student emotion, co-occurrence, and dropout in a MOOC context. In T. Barnes, M. Chi, & M. Feng (Eds.), Proceedings of the 9th International Conference on Educational Data Mining (EDM 2016) (pp. 353–357). International Educational Data Mining Society.

Donnelly, P., Blanchard, N., Samei, B., Olney, A. M., Sun, X., Ward, B., . . . D’Mello, S. K. (2016a). Automatic teacher modeling from live classroom audio. In L. Aroyo, S. D’Mello, J. Vassileva, & J. Blustein (Eds.), Proceedings of the 2016 ACM International Conference on User Modeling, Adaptation, & Personalization (ACM UMAP 2016) (pp. 45–53). New York: ACM.

Donnelly, P., Blanchard, N., Samei, B., Olney, A. M., Sun, X., Ward, B., . . . D’Mello, S. K. (2016b). Multi-sensor modeling of teacher instructional segments in live classrooms. Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI 2016). New York: ACM.

Ekman, P., & Friesen, W. (1978). The facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.

Emotient. (2014). FACET: Facial expression recognition software.

Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance: A critical literature review. Chicago, IL: University of Chicago Consortium on Chicago School Research.

Forbes-Riley, K., & Litman, D. J. (2011). Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor. Speech Communication, 53(9–10), 1115–1136. doi:10.1016/j.specom.2011.02.006

Fredrickson, B., & Branigan, C. (2005). Positive emotions broaden the scope of attention and thought-action repertoires. Cognition & Emotion, 19(3), 313–332. doi:10.1080/02699930441000238

Galla, B. M., Plummer, B. D., White, R. E., Meketon, D., D’Mello, S. K., & Duckworth, A. L. (2014). The academic diligence task (ADT): Assessing individual differences in effort on tedious but important schoolwork. Contemporary Educational Psychology, 39(4), 314–325.

Graesser, A., Chipman, P., Haynes, B., & Olney, A. (2005). AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48(4), 612–618. doi:10.1109/TE.2005.856149

Gross, J. (2008). Emotion regulation. In M. Lewis, J. Haviland-Jones, & L. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 497–512). New York: Guilford.

Hess, U. (2003). Now you see it, now you don’t — the confusing case of confusion as an emotion: Commentary on Rozin and Cohen (2003). Emotion, 3(1), 76–80.

Isen, A. (2008). Some ways in which positive affect influences decision making and problem solving. In M. Lewis, J. Haviland-Jones, & L. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 548–573). New York: Guilford.

Izard, C. (2010). The many meanings/aspects of emotion: Definitions, functions, activation, and regulation. Emotion Review, 2(4), 363–370. doi:10.1177/1754073910374661

James, W. (1884). What is an emotion? Mind, 9, 188–205.

Jayaprakash, S. M., Moody, E. W., Lauría, E. J., Regan, J. R., & Baron, J. D. (2014). Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47.

Kai, S., Paquette, L., Baker, R., Bosch, N., D’Mello, S., Ocumpaugh, J., . . . Ventura, M. (2015). Comparison of face-based and interaction-based affect detectors in physics playground. In C. Romero, M. Pechenizkiy, J. Boticario, & O. Santos (Eds.), Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015) (pp. 77–84). International Educational Data Mining Society.

Kory, J., D’Mello, S. K., & Olney, A. (2015). Motion tracker: Camera-based monitoring of bodily movements using motion silhouettes. PloS ONE, 10(6), doi:10.1371/journal.pone.0130293.

Lench, H. C., Bench, S. W., & Flores, S. A. (2013). Searching for evidence, not a war: Reply to Lindquist, Siegel, Quigley, and Barrett (2013). Psychological Bulletin, 113(1), 264–268.

Lewis, M. D. (2005). Bridging emotion theory and neurobiology through dynamic systems modeling. Behavioral and Brain Sciences, 28(2), 169–245.

Lindquist, K. A., Siegel, E. H., Quigley, K. S., & Barrett, L. F. (2013). The hundred-year emotion war: Are emotions natural kinds or psychological constructions? Comment on Lench, Flores, and Bench (2011). Psychological Bulletin, 139(1), 264–268.

McKeown, G. J., & Sneddon, I. (2014). Modeling continuous self-report measures of perceived emotion using generalized additive mixed models. Psychological Methods, 19(1), 155–174.

Mesquita, B., & Boiger, M. (2014). Emotions in context: A sociodynamic model of emotions. Emotion Review, 6(4), 298–302.

Microsoft. (2015). Kinect for Windows SDK MSDN. Retrieved 5 August 2015 from

Moors, A. (2014). Flavors of appraisal theories of emotion. Emotion Review, 6(4), 303–307.

Nystrand, M. (1997). Opening dialogue: Understanding the dynamics of language and learning in the English classroom. Language and Literacy Series. New York: Teachers College Press.

Ocumpaugh, J., Baker, R. S., & Rodrigo, M. M. T. (2012). Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual version 1.0. New York.

OECD (Organisation for Economic Co-operation and Development). (2015). PISA 2015 collaborative problem solving framework.

Ohman, A., & Soares, J. (1994). Unconscious anxiety: Phobic responses to masked stimuli. Journal of Abnormal Psychology, 103(2), 231–240.

Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2), 1–135.

Pardos, Z., Baker, R. S. J. d., San Pedro, M. O. C. Z., & Gowda, S. M. (2013). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In D. Suthers, K. Verbert, E. Duval, & X. Ochoa (Eds.), Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 117–124). New York: ACM.

Parkinson, B., Fischer, A. H., & Manstead, A. S. (2004). Emotion in social relations: Cultural, group, and interpersonal processes. Psychology Press.

Pekrun, R., & Stephens, E. J. (2011). Academic emotions. In K. Harris, S. Graham, T. Urdan, S. Graham, J. Royer, & M. Zeidner (Eds.), APA educational psychology handbook, Vol 2: Individual differences and cultural and contextual factors (pp. 3–31). Washington, DC: American Psychological Association.

Porayska-Pomsta, K., Mavrikis, M., D’Mello, S. K., Conati, C., & Baker, R. (2013). Knowledge elicitation methods for affect modelling in education. International Journal of Artificial Intelligence in Education, 22, 107–140.

Raca, M., Kidzinski, L., & Dillenbourg, P. (2015). Translating head motion into attention: Towards processing of student’s body language. Proceedings of the 8th International Conference on Educational Data Mining. International Educational Data Mining Society.

Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N. T., Koedinger, K. R., Junker, B., . . . Choksey, S. (2005). The assistment project: Blending assessment and assisting. In C. Loi & G. McCalla (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 555–562). Amsterdam: IOS Press.

Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013). Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. Proceedings of the 2nd International Workshop on Emotion Representation, Analysis and Synthesis in Continuous Time and Space (EmoSPACE) in conjunction with the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Washington, DC: IEEE.

Rosenberg, E. (1998). Levels of analysis and the organization of affect. Review of General Psychology, 2(3), 247–270. doi:10.1037//1089-2680.2.3.247

Rozin, P., & Cohen, A. (2003). High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of maturally occurring facial expressions of Americans. Emotion, 3, 68–75.

Sabourin, J., Mott, B., & Lester, J. (2011). Modeling learner affect with theoretically grounded dynamic Bayesian networks. In S. D’Mello, A. Graesser, B. Schuller, & J. Martin (Eds.), Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction (pp. 286–295). Berlin/Heidelberg: Springer-Verlag.

San Pedro, M., Baker, R. S., Bowers, A. J., & Heffernan, N. T. (2013). Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In S. D’Mello, R. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (EDM 2013) (pp. 177–184). International Educational Data Mining Society.

Schwarz, N. (2012). Feelings-as-information theory. In P. Van Lange, A. Kruglanski, & T. Higgins (Eds.), Handbook of theories of social psychology (pp. 289–308). Thousand Oaks, CA: Sage.

Shute, V. J., Ventura, M., & Kim, Y. J. (2013). Assessment and learning of qualitative physics in Newton’s playground. The Journal of Educational Research, 106(6), 423–430.

Sinha, T., Jermann, P., Li, N., & Dillenbourg, P. (2014). Your click decides your fate: Inferring information processing and attrition behavior from MOOC video clickstream interactions. Paper presented at the Empirical Methods in Natural Language Processing: Workshop on Modeling Large Scale Social Interaction in Massively Open Online Courses.

Strain, A., & D’Mello, S. (2014). Affect regulation during learning: The enhancing effect of cognitive reappraisal. Applied Cognitive Psychology, 29(1), 1–19. doi:10.1002/acp.3049

Tracy, J. L. (2014). An evolutionary approach to understanding distinct emotions. Emotion Review, 6(4), 308–312.

VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 21(3), 209–249. doi:10.1207/S1532690XCI2103_01

Wang, Z., Miller, K., & Cortina, K. (2013). Using the LENA in teacher training: Promoting student involvement through automated feedback. Unterrichtswissenschaft, 4, 290–305.

Wen, M., Yang, D., & Rosé, C. (2014). Sentiment analysis in MOOC discussion forums: What does it tell us? In J. Stamper, S. Pardos, M. Mavrikis, & B. M. McLaren (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), 4–7 July, London, UK (pp. 130–137): International Educational Data Mining Society.

Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: Recognizing and responding to student affect. International Journal of Learning Technology, 4(3/4), 129–163.

Yang, D., Wen, M., Howley, I., Kraut, R., & Rosé, C. (2015). Exploring the effect of confusion in discussion forums of massive open online courses. Proceedings of the 2nd ACM conference on Learning@Scale (L@S 2015), 14–18 March 2015, Vancouver, BC, Canada (pp. 121–130). New York: ACM. doi:10.1145/2724660.2724677

Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39–58.

About this Chapter

Emotional Learning Analytics

Book Title
Handbook of Learning Analytics

pp. 115-127




Society for Learning Analytics Research

Sidney K. D’Mello

Author Affiliations
Departments of Psychology and Computer Science & Engineering, University of Notre Dame, USA

Charles Lang1
George Siemens2
Alyssa Wise3
Dragan Gašević4

Editor Affiliations
1. Teachers College, Columbia University, USA
2. LINK Research Lab, University of Texas at Arlington, USA
3. Learning Analytics Research Network, New York University, USA
4. Schools of Education and Informatics, University of Edinburgh, UK

Founding Members
Previous Image
Next Image

info heading

info content