Chapter 11

Handbook of Learning Analytics
First Edition

Multimodal Learning Analytics

Xavier Ochoa


Abstract

This chapter presents a different way to approach learning analytics (LA) praxis through the capture, fusion, and analysis of complementary sources of learning traces to obtain a more robust and less uncertain understanding of the learning process. The sources or modalities in multimodal learning analytics (MLA) include the traditional log-file data captured by online systems, but also learning artifacts and more natural human signals such as gestures, gaze, speech, or writing. The current state-of-the-art of MLA is discussed and classified according to its modalities and the learning settings where it is usually applied. This chapter concludes with a discussion of emerging issues for practitioners in multimodal techniques.

Export Citation: Plain Text (APA)     BIBTeX     RIS

Supplementary Material
No Supplementary Material Available
References (66)

Alibali, M. W., Nathan, M. J., Fujimori, Y., Stein, N., & Raudenbush, S. (2011). Gestures in the mathematics classroom: What’s the point? In N. Stein & S. Raudenbush (Eds.), Developmental cognitive science goes to school (pp. 219–234). New York: Routledge, Taylor & Francis.

Almeda, M. V., Scupelli, P., Baker, R. S., Weber, M., & Fisher, A. (2014). Clustering of design decisions in classroom visual displays. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 44–48). New York: ACM.

Anderson, J. R. (2002). Spanning seven orders of magnitude: A challenge for cognitive modeling. Cognitive Science, 26(1), 85–112.

Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 267–270). New York: ACM.

Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 102–106). New York: ACM.

Boraston, Z., & Blakemore, S.-J. (2007). The application of eye-tracking technology in the study of autism. The Journal of Physiology, 581(3), 893–898.

Bull, P. E. (2013). Posture & Gesture. Elsevier.

Boncoddo, R., Williams, C., Pier, E., Walkington, C., Alibali, M., Nathan, M., Dogan, M. & Waala, J. (2013). Gesture as a window to justification and proof. In M. V. Martinez & A. C. Superfine (Eds.), Proceedings of the 35th annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (PME-NA 35) 14–17 November 2013, Chicago, IL, USA (pp. 229–236). http://www.pmena.org/proceedings/

Chen, L., Leong, C. W., Feng, G., & Lee, C. M. (2014). Using multimodal cues to analyze MLA ’14 oral presentation quality corpus: Presentation delivery and slides quality. Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 45–52). New York: ACM.

Clarke, B., & Svanaes, S. (2014, April 9). An updated literature review on the use of tablets in education. Family Kids and Youth. https://smartfuse.s3.amazonaws.com/mysandstorm.org/uploads/2014/05/T4S-Use-of-Tablets-in-Education.pdf

Cobb, P., Confrey, J., Lehrer, R., Schauble, L., & others. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.

Craig, S. D., D’Mello, S., Witherspoon, A., & Graesser, A. (2008). Emote aloud during learning with AutoTutor: Applying the facial action coding system to cognitive–affective states during learning. Cognition and Emotion, 22(5), 777–788.

Crossley, S. A., Roscoe, R. D., & McNamara, D. S. (2013). Using automatic scoring models to detect changes in student writing in an intelligent tutoring system. Proceedings of the 26th Annual Florida Artificial Intelligence Research Society Conference (FLAIRS-13), 20–22 May 2013, St. Pete Beach, FL, USA (pp. 208–213). Menlo Park, CA: The AAAI Press.

Debnath, M., Pandey, M., Chaplot, N., Gottimukkula, M. R., Tiwari, P. K., & Gupta, S. N. (2012). Role of soft skills in engineering education: Students’ perceptions and feedback. In C. S. Nair, A. Patil, & P. Mertova (Eds.), Enhancing learning and teaching through student feedback in engineering (pp. 61–82). ScienceDirect. http://www.sciencedirect.com/science/book/9781843346456

D’Mello, S. K., Jackson, G. T., Craig, S. D., Morgan, B., Chipman, P., White, H., Person, N., Kort, B., el Kaliouby, R., Picard, R., & Graesser, A. C. (2008). AutoTutor detects and responds to learners’ affective and cognitive states. Workshop on Emotional and Cognitive Issues in ITS, held in conjunction with the 9th International Conference on Intelligent Tutoring Systems (ITS 2008), 23–27 June 2008, Montreal, PQ, Canada. https://www.researchgate.net/publication/228673992_AutoTutor_detects_and_responds_to_learners_affective_and_cognitive_states

D’Mello, S., Olney, A., Blanchard, N., Samei, B., Sun, X., Ward, B., & Kelly, S. (2015). Multimodal capture of teacher–student interactions for automated dialogic analysis in live classrooms. Proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI ’15), 9–13 November 2015, Seattle, WA, USA (pp. 557–566). New York: ACM.

Dominguez, F., Echeverría, V., Chiluiza, K., & Ochoa, X. (2015). Multimodal selfies: Designing a multimodal recording device for students in traditional classrooms. Proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI ’15), 9–13 November 2015, Seattle, WA, USA (pp. 567–574). New York: ACM.

Echeverría, V., Avendaño, A., Chiluiza, K., Vásquez, A., & Ochoa, X. (2014). Presentation skills estimation based on video and Kinect data analysis. Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 53–60). New York: ACM.

Freedman, D. H. (2010, December 10). Why scientific studies are so often wrong: The streetlight effect. Discover Magazine, 26. http://discovermagazine.com/2010/jul-aug/29-why-scientific-studies-often-wrong-streetlight-effect

Frischen, A., Bayliss, A. P., & Tipper, S. P. (2007). Gaze cueing of attention: Visual attention, social cognition, and individual differences. Psychological Bulletin, 133(4), 694–724.

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. Longman Publishing.

Graesser, A. C., Chipman, P., Haynes, B. C., & Olney, A. (2005). AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48(4), 612–618.

Householder, D. L., & Hailey, C. E. (2012). Incorporating engineering design challenges into STEM courses. National Center for Engineering and Technology Education. http://ncete.org/flash/pdfs/NCETECaucusReport.pdf

Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. Psychology Press.

Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 170–179). New York: ACM.

Kress, G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. Edward Arnold.

Krugman, D. M., Fox, R. J., Fletcher, J. E., Fischer, P. M., & Rojas, T. H. (1994). Do adolescents attend to warnings in cigarette advertising? An eye-tracking approach. Journal of Advertising Research, 34, 39–51.

Kruschke, J. K. (2003). Attention in learning. Current Directions in Psychological Science, 12(5), 171–175.

Leong, C. W., Chen, L., Feng, G., Lee, C. M., & Mulholland, M. (2015). Utilizing depth sensors for analyzing multimodal presentations: Hardware, software and toolkits. Proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI ’15), 9–13 November 2015, Seattle, WA, USA (pp. 547–556). New York: ACM.

Lin, Y.-T., Lin, R.-Y., Lin, Y.-C., & Lee, G. C. (2013). Real-time eye-gaze estimation using a low-resolution webcam. Multimedia Tools and Applications, 65(3), 543–568.

Lubold, N., & Pon-Barry, H. (2014). Acoustic-prosodic entrainment and rapport in collaborative learning dialogues. Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 5–12). New York: ACM.

Lund, K. (2007). The importance of gaze and gesture in interactive multimodal explanation. Language Resources and Evaluation, 41(3–4), 289–303.

Luzardo, G., Guamán, B., Chiluiza, K., Castells, J., & Ochoa, X. (2014). Estimation of presentations skills based on slides and audio features. Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 37–44). New York: ACM.

Luz, S. (2013). Automatic identification of experts and performance prediction in the multimodal math data corpus through analysis of speech interaction. Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI ’13), 9–13 December 2013, Sydney, Australia (pp. 575–582). New York: ACM.

Markaki, V., Lund, K., & Sanchez, E. (2015). Design digital epistemic games: A longitudinal multimodal analysis. Paper presented at the conference Revisiting Participation: Language and Bodies in Interaction, 24–27 June 2015, Basel, Switzerland.

Mazur-Palandre, A., Colletta, J. M., & Lund, K. (2014). Context sensitive “how” explanation in children’s multimodal behavior, Journal of Multimodal Communication Studies, 2, 1–17.

Mishra, B., Fernandes, S. L., Abhishek, K., Alva, A., Shetty, C., Ajila, C. V., … Shetty, P. (2015). Facial expression recognition using feature based techniques and model based techniques: A survey. Proceedings of the 2nd International Conference on Electronics and Communication Systems (ICECS 2015), 26–27 February 2015, Coimbatore, India (pp. 589–594). IEEE.

Mitra, S., & Acharya, T. (2007). Gesture recognition: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 37(3), 311–324.

Morency, L.-P., Oviatt, S., Scherer, S., Weibel, N., & Worsley, M. (2013). ICMI 2013 grand challenge workshop on multimodal learning analytics. Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI ’13), 9–13 December 2013, Sydney, Australia (pp. 373–378). New York: ACM.

Newell, A. (1994). Unified theories of cognition. Harvard University Press.

Ochoa, X., Chiluiza, K., Méndez, G., Luzardo, G., Guamán, B., & Castells, J. (2013). Expertise estimation based on simple multimodal features. Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI ’13), 9–13 December 2013, Sydney, Australia (pp. 583–590). New York: ACM.

Ochoa, X., Worsley, M., Chiluiza, K., & Luz, S. (2014). MLA ’14: Third multimodal learning analytics workshop and grand challenges. Proceedings of the 16th ACM International Conference on Multimodal Interaction (ICMI ’14), 12–16 November 2014, Istanbul, Turkey (pp. 531–532). New York: ACM.

Oviatt, S., Cohen, A., & Weibel, N. (2013). Multimodal learning analytics: Description of math data corpus for ICMI grand challenge workshop. Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI ’13), 9–13 December 2013, Sydney, Australia (pp. 583–590). New York: ACM.

Oviatt, S., & Cohen, P. R. (2015). The paradigm shift to multimodality in contemporary computer interfaces. San Rafael, CA: Morgan & Claypool Publishers.

Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational Technology, 45(3), 438–450.

Poole, A., & Ball, L. J. (2006). Eye tracking in HCI and usability research. Encyclopedia of Human Computer Interaction, 1, 211–219.

Raca, M., & Dillenbourg, P. (2013). System for assessing classroom attention. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 265–269). New York: ACM.

Raca, M., Tormey, R., & Dillenbourg, P. (2014). Sleepers’ lag: Study on motion and attention. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 36–43). New York: ACM.

Scherer, S., Worsley, M., & Morency, L.-P. (2012). 1st international workshop on multimodal learning analytics. Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI ’12), 22–26 October 2012, Santa Monica, CA, USA (pp. 609–610). New York: ACM.

Schlömer, T., Poppinga, B., Henze, N., & Boll, S. (2008). Gesture recognition with a Wii controller. Proceedings of the 2nd International Conference on Tangible and Embedded Interaction (TEI ’08), 18–21 February 2008, Bonn, Germany (pp. 11–14). New York: ACM.

Schneider, J., Börner, D., van Rosmalen, P., & Specht, M. (2015). Presentation trainer, your public speaking multimodal coach. Proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI ’15), 9–13 November 2015, Seattle, WA, USA (pp. 539–546). New York: ACM.

Serrano-Laguna, A., & Fernández-Manjón, B. (2014). Applying learning analytics to simplify serious games deployment in the classroom. Proceedings of the 2014 IEEE Global Engineering Education Conference (EDUCON 2014), 3–5 April 2014, Istanbul, Turkey (pp. 872–877). IEEE.

Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 9, 51–60. doi:0002764213498851.

Silver, E. A. (2013). Teaching and learning mathematical problem solving: Multiple research perspectives. Routledge.

Simsek, D., Sándor, Á., Buckingham Shum, S., Ferguson, R., De Liddo, A., & Whitelock, D. (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 355–359). New York: ACM.

Swan, M. (2013). The quantified self: Fundamental disruption in big data science and biological discovery. Big Data, 1(2), 85–99.

Thompson, K. (2013). Using micro-patterns of speech to predict the correctness of answers to mathematics problems: An exercise in multimodal learning analytics. Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI ’13), 9–13 December 2013, Sydney, Australia (pp. 591–598). New York: ACM.

Vasquez, H. A., Vargas, H. S., & Sucar, L. E. (2015). Using gestures to interact with a service robot using Kinect 2. Advances in Computer Vision and Pattern Recognition, 85. Springer.

Vatrapu, R., Reimann, P., Bull, S., & Johnson, M. (2013). An eye-tracking study of notational, informational, and emotional aspects of learning analytics representations. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 125–134). New York: ACM.

Worsley, M., & Blikstein, P. (2013). Towards the development of multimodal action based assessment. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 94–101). New York: ACM.

Worsley, M., & Blikstein, P. (2014a). Deciphering the practices and affordances of different reasoning strategies through multimodal learning analytics. Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 21–27). New York: ACM.

Worsley, M., & Blikstein, P. (2014b). Using multimodal learning analytics to study learning mechanisms. In J. Stamper, Z. Pardos, M. Mavrikis, & B. M. McLaren (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM2014), 4–7 July, London, UK (pp. 431–432). International Educational Data Mining Society.

Worsley, M., & Blikstein, P. (2015a). Leveraging multimodal learning analytics to differentiate student learning strategies. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 360–367). New York: ACM.

Worsley, M., & Blikstein, P. (2015b). Using learning analytics to study cognitive disequilibrium in a complex learning environment. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 426–427). New York: ACM.

Zhang, Z. (2012). Microsoft Kinect sensor and its effect. IEEE MultiMedia, 19(2), 4–10.

Zhou, J., Hang, K., Oviatt, S., Yu, K., & Chen, F. (2014). Combining empirical and machine learning techniques to predict math expertise using pen signal features. Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 29–36). New York: ACM.

About this Chapter

Title
Multimodal Learning Analytics

Book Title
Handbook of Learning Analytics

Pages
pp. 129-141

Copyright
2017

DOI
10.18608/hla17.011

ISBN
978-0-9952408-0-3

Publisher
Society for Learning Analytics Research

Authors
Xavier Ochoa

Author Affiliations
Escuela Superior Politécnica del Litoral, Ecuador

Editors
Charles Lang1
George Siemens2
Alyssa Wise3
Dragan Gašević4

Editor Affiliations
1. Teachers College, Columbia University, USA
2. LINK Research Lab, University of Texas at Arlington, USA
3. Learning Analytics Research Network, New York University, USA
4. Schools of Education and Informatics, University of Edinburgh, UK

Founding Members
Previous Image
Next Image

info heading

info content