March 5, 2025
Keywords: multimodal AI analytics, teamwork analytics in healthcare, clinical simulation training, team communication dynamics, healthcare simulation tools, human-centered AI design, AI in education
Target audience: healthcare and nursing educators and trainers, education technology researchers and enthusiasts, healthcare educational institutions, healthcare technology companies, healthcare professionals, and corporate training development teams
Authors: Assoc. Prof. Roberto Martinez-Maldonado, Dr Vanessa Echeverria & Riordan Alfredo
Roberto Martinez-Maldonado is an Associate Professor in Learning Analytics and Human-Computer Interaction at Monash University, with a PhD from the University of Sydney. His research focuses on Learning and Teamwork Analytics, blending human-computer interaction, collaborative learning, and AI to develop tools like analytics dashboards and warning systems for education and healthcare. He pioneers co-design methods at theintersection of education and data science, aiming to enhance team performance through multimodal data analysis.
Vanessa Echeverria is a Lecturer at RMIT University in Melbourne, Australia, with a background as a post-doctoral Research Fellow at Monash University and an Assistant Professor at ESPOL, Ecuador. Her research focuses on designing and evaluating AI-driven, human-centred digital tools, such as dashboards, to enhance learning and collaboration in educational and healthcare contexts. By combining Human-Computer Interaction (HCI) and AI, she employs co-design methods to create adaptable solutions that support STEM skills, team-based learning, and healthcare education.
Riordan Alfredo is a PhD Candidate in Learning Analytics and Artificial Intelligence in Education at Monash University’s Faculty of Information Technology, with a Bachelor of Software Engineering (Honours) from the same institution. As a software engineer and researcher, he focuses on advancing the design and development of learning analytics and educational technologies through co-design and Human-Centred AI principles. His work prioritises creating safe, reliable, and trustworthy systems that empower end-users, bridging technical expertise with a deep commitment to ethical and user-focused innovation.
MULTIMODAL AI-POWERED TEAMWORK ANALYTICS IN HEALTHCARE SIMULATION
The rapid evolution of Artificial Intelligence (AI) is revolutionising how we learn and collaborate. But are we truly ready to harness AI’s transformative potential? Our AI-powered Teamwork Analytics initiative is paving the way by leveraging cutting-edge technology to elevate teamwork skills, focusing on essential areas like effective communication and leadership. The goal? To empower learners and educators to thrive in an AI-driven world.
Bridging Innovation and Education
Imagine a state-of-the-art learning environment where healthcare students step into realistic clinical simulations equipped with advanced sensing technologies. Just as elite athletes use data from sport analytics to fine-tune their performance, every body movement, heartbeat, and spoken word of these medical and nursing students is captured by our AI tools and transformed into actionable insights. These multimodal analytics, akin to the real-time feedback used by high-performing teams to refine their strategies and improve collaboration, are presented during debrief sessions. This enables students to reflect on their clinical and teamwork practices, fostering the same kind of continuous improvement and excellence that drives top performers in the world of sports.
Human-Centred AI Approach to Learning Analytics Design
We believe that AI should enhance human potential rather than replace it. Our methods prioritise meaningful insights and intuitive user experiences, including:
- Voice Conversation Analysis: Using large language models (LLMs) to automate the transcription and analysis of spoken conversations, identifying key teamwork dynamics such as closed-loop communication and responsiveness. [paper]
- Workflow Positioning Modelling: Real-time mapping of individuals’ movements in learning spaces, offering insights into spatial interactions. [paper]
- Team Insights Provision: Delivering immediate feedback on critical teamwork skills through user-friendly applications. [paper]
- Stress and Arousal Measurement: Assessing physiological indicators like heart rate to gauge stress and emotional arousal in real-time. [paper]
- Automated Video Analysis: Identifying team behaviours and interactions using advanced privacy-preserving video analytics to provide deeper insights into team dynamics. [paper]
These innovations have been developed through close collaboration with students and teachers engaged over five years of partnerships. Regular co-design sessions with educators have ensured the solutions address practical classroom and simulation needs. Additionally, a paid advisory board of students has provided invaluable feedback, ensuring that our tools are both effective and user-friendly for those who will use them most.
Examples of Innovations in Action
Below, we highlight some exciting examples of how our AI-powered solutions are transforming teamwork analytics. These real-world applications showcase the tangible impact of integrating advanced technology into education and healthcare.
Voice Conversation Analysis
Our analytics leverage AI to automate transcription and analyse team communication dynamics in real-time. By identifying patterns such as effective query responsiveness and information sharing, educators can better guide students in improving these crucial skills.
Indoor-positioning Workflows
Utilising indoor-positioning analytics combined with human-voice detection, and even physiological signals such as heart rate, we can identify spatial positioning patterns that correlate with highly effective teamwork in clinical settings or situations where students may have to perform under pressure. This can provide actionable insights into optimal team configurations and movement strategies in ward environments.
AI-Augmented Debriefing
Through user-friendly interfaces, we visualise AI-generated insights to empower educators during debriefing sessions, inspired on the notion of data storytelling. These evidence-based tools enhance reflective learning, offering students a richer understanding of their teamwork performance.
SimVision: AI-powered Teamwork and Communication Reflection Tool
After the group debrief, students can access an analytics dashboard in-class or at home, allowing deeper individual reflection on their performance. With VizChat, an AI-powered chat assistant available on the right side of the screen, learners can ask specific questions about their teamwork and communication skills, guided by the data displayed on the left. By combining AI technology with actionable insights, students are empowered to take charge of their learning. In other words, this interactive setup transforms reflection into a dialogic process, enabling students to analyse their performance and identify areas for improvement.
Beyond Healthcare: Transforming Classrooms
Our innovations extend beyond healthcare simulation, bringing the benefits of AI-driven analytics to classroom teaching:
Reflective Co-Teaching Analytics
By combining indoor positioning analytics with voice detection, we provide data-driven evidence of spatial patterns and interactions in classrooms. These insights can guide educators and pre-service teachers in optimising teaching strategies and classroom dynamics, fostering more effective learning environments.
Partnering for Innovation
As we look to the future, the potential for AI-driven teamwork analytics to transform education and healthcare is boundless. Unlike approaches that rely solely on online learning or personal devices, our innovations also extend to physical and hybrid learning spaces enhanced with digital tools, maximising inclusivity and adaptability across diverse environments. Whether you’re an educator, researcher, or industry professional, there are countless opportunities to innovate together. Let’s discuss how we can co-create solutions that not only enhance learning experiences but also inspire a new generation of leaders. Join us in shaping the future—one breakthrough at a time!
Learn more at: Human-Centred Teamwork Analytics – Assisting the assessment and improvement of collocated teamwork
References:
Zhao, L., Gašević, D., Swiecki, Z., Li, Y., Lin, J., Sha, L., Yan, L., Alfredo, R., Li, X. & Martinez‐Maldonado, R. (2024). Towards automated transcribing and coding of embodied teamwork communication through multimodal learning analytics. British Journal of Educational Technology. [link]
Yan, L., Martinez-Maldonado, R., Swiecki, Z., Zhao, L., Li, X., & Gašević, D. (2024). Dissecting the temporal dynamics of embodied collaborative learning using multimodal learning analytics. Journal of Educational Psychology. [link]
Echeverria, V., Nieto, G. F., Zhao, L., Palominos, E., Srivastava, N., Gašević, D., Pammer‐Schindler, V. & Martinez‐Maldonado, R. (2024). A learning analytics dashboard to support students’ reflection on collaboration. Journal of Computer Assisted Learning. [link]
Alfredo, R. D., Nie, L., Kennedy, P., Power, T., Hayes, C., Chen, H., McGregor, C., Swiecki, Z., Gašević, D. & Martinez-Maldonado, R. (2023, March). ” That Student Should be a Lion Tamer!” StressViz: Designing a Stress Analytics Dashboard for Teachers. In LAK23: 13th International Learning Analytics and Knowledge Conference (pp. 57-67). [link]
Li, X., Yan, L., Zhao, L., Martinez-Maldonado, R., & Gasevic, D. (2023, March). CVPE: A computer vision approach for scalable and privacy-preserving socio-spatial, multimodal learning analytics. In LAK23: 13th International Learning Analytics and Knowledge Conference (pp. 175-185). [link]