Handbook of Learning Analytics
Chapter 17
Institutional Analytics
Leah P. Macfadyen
Abstract
Since its emergence, the field of learning analytics has proposed that educational institutions can and should make better use of learner data to optimize learning and learning environments. A range of social, political and economic forces have also encouraged educational institutions to consider system-wide implementations of learning analytics. In spite of a decade of optimism and interest, however, very few examples of effective institutional LA implementation exist, and evidence of positive impact on learning is sparse. This chapter provides an updated summary of the growing body of literature exploring the challenges of making systemic change with LA in complex educational contexts. Proposed frameworks for guiding institutional LA implementations are reviewed, and work describing use of the most promising – the SHEILA framework – is outlined in more detail. The need for attention to complexity leadership and institutional logics is noted as a focus of recent work, and emerging issues are highlighted: a critical need to expand the literature documenting evidence of real impact on learning, a need for institutions to make use of reliable LA evaluation strategies, and the need for critical consideration of how and if LA can also benefit learners beyond the traditional higher education contexts of the wealthy North.
Arnold, Kimberly E, Steven Lonn, and Matthew D Pistilli. 2014. “An Exercise in Institutional Reflection: The Learning Analytics Readiness Instrument (LARI).” In, 163–67. Indianapolis, IN, USA: New York, USA: ACM. https://doi.org/10.1145/2567574.2567621.
Arnold, Kimberly E, and Matthew D. Pistilli. 2012. “Course Signals at Purdue.” In, 267–70. Vancouver, BC, Canada: New York, USA: ACM. https://doi.org/10.1145/2330601.2330666.
Baker, Ryan Shaun, and Paul Salvador Inventado. 2014. “Educational Data Mining and Learning Analytics.” In Learning Analytics, edited by J. A. Larusson and B. White, 61–75. New York, USA: Springer. https://doi.org/10.1007/978-1-4614-3305-7_4.
Barton, Dominic, and David Court. 2012. “Making Advanced Analytics Work for You.” Harvard Business Review 90 (10): 78–83. https://hbr.org/2012/10/making-advanced-analytics-work-for-you.
Bichsel, J. 2012. “Analytics in Higher Education: Benefits, Barriers, Progress, and Recommendations.” Louisville, CO, USA. http://net.educause.edu/ir/library/pdf/ERS1207/ers1207.pdf.
Bienkowski, Marie, Mingyu Feng, and Barbara Means. 2012. “Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief.” U.S. Department of Education Office of Educational Technology. https://tech.ed.gov/wp-content/uploads/2014/03/edm-la-brief.pdf.
Bolden, Richard. 2011. “Distributed Leadership in Organizations: A Review of Theory and Research.” International Journal of Management Reviews 13 (3): 251–69. https://doi.org/10.1111/j.1468-2370.2011.00306.x.
Brooks, Christopher, and Craig Thompson. 2017. “Predictive Modelling in Teaching and Learning.” In The Handbook of Learning Analytics, edited by Charles Lang, George Siemens, Alyssa Friend Wise, and Dragan Gaševic, 1st ed., 61–68. Alberta, Canada: Society for Learning Analytics Research (SoLAR). http://solaresearch.org/hla-17/hla17-chapter1.
Broos, Tom, Isabel Hilliger, Mar Pérez-Sanagustín, Nyi‐Nyi Htun, Martijn Millecamp, Paola Pesántez-Cabrera, Lizandro Solano‐Quinde, Lorena Siguenza‐Guzman, Miguel Zuñiga‐Prieto, and Katrien Verbert. 2020. “Coordinating Learning Analytics Policymaking and Implementation at Scale.” British Journal of Educational Technology 51 (4): 938–54. https://doi.org/10.1111/bjet.12934.
Campbell, John P, Peter B DeBlois, and Diana G Oblinger. 2007. “Academic Analytics: A New Tool for a New Era.” EDUCAUSE Review 42 (4): 40.
Capra, Fritjof. 1996. The Web of Life: A Necessary Unity. New York, USA: Doubleday.
Cilliers, Paul. 1998. Complexity and Post-Modernism: Understanding Complex Systems. London: Routledge.
Clow, Doug. 2012. “The Learning Analytics Cycle: Closing the Loop Effectively.” In, Proceedings of the 2nd international conference on learning analytics and knowledge:134–38. Vancouver, BC, Canada: New York, USA: ACM. https://doi.org/10.1145/2330601.2330636.
Colvin, Cassandra, Shane Dawson, Alexandra Wade, and Dragan Gašević. 2017. “Addressing the Challenges of Institutional Adoption.” In The Handbook of Learning Analytics, edited by Charles Lang, George Siemens, Alyssa Friend Wise, and Dragan Gaševic, 1st ed., 281–89. Alberta, Canada: Society for Learning Analytics Research (SoLAR). http://solaresearch.org/hla-17/hla17-chapter1.
Colvin, Cassandra, Tim Rogers, Shane Dawson, Dragan Gasevic, Simon Buckingham Shum, K. Nelson, Shirely Alexander, et al. 2016. “Student Retention and Learning Analytics: A Snapshot of Australian Practices and a Framework for Advancement.” Canberra, ACT: Australian Government Office for Learning and Teaching. http://hdl.handle.net/10453/117173.
Davenport, Thomas, and Jeanne Harris. 2007. Competing on Analytics: The New Science of Winning. Boston, MA, USA: Harvard Business School Press.
Dawson, Shane, Leah P. Macfadyen, Evan F Risko, Tom Foulsham, and Alan Kingstone. 2012. “Using Technology to Encourage Self-Directed Learning: The Collaborative Lecture Annotation System (CLAS).” In. Wellington, New Zealand. http://www.ascilite.org/conferences/Wellington12/2012/images/custom/dawson,_shane_-_using_technology.pdf.
Dawson, Shane, Oleksandra Poquet, Cassandra Colvin, Tim Rogers, Abelardo Pardo, and Dragan Gasevic. 2018. “Rethinking Learning Analytics Adoption Through Complexity Leadership Theory.” In, 236–44. Sydney, Australia. https://doi.org/10.1145/3170358.3170375.
Denley, Tristan. 2012. “Austin Peay State University: Degree Compass.” EDUCAUSE Review. https://er.educause.edu/articles/2012/9/austin-peay-state-university-degree-compass.
Drachsler, Hendrik, and Wolfgang Greller. 2016. “Privacy and Analytics: It’s a DELICATE Issue. A Checklist for Trusted Learning Analytics.” In, 89–98. Edinburgh, Scotland: New York, USA: ACM. https://doi.org/10.1145/2883851.2883893.
Ferguson, Rebecca. 2012. “Learning Analytics: Drivers, Developments and Challenges.” International Journal of Technology Enhanced Learning 4 (5/6): 304–17. http://oro.open.ac.uk/36374/.
Ferguson, Rebecca, and Doug Clow. 2017. “Where Is the Evidence? A Call to Action for Learning Analytics.” In, 56–65. Vancouver, British Columbia, Canada: New York, USA: ACM. https://doi.org/10.1145/3027385.3027396.
Ferguson, Rebecca, Doug Clow, Leah Macfadyen, Alfred Essa, Shane Dawson, and Shirley Alexander. 2014. “Setting Learning Analytics in Context: Overcoming the Barriers to Large-Scale Adoption.” Journal of Learning Analytics 1 (3): 120–44. https://doi.org/10.18608/jla.2014.13.7.
Fritz, John. 2011. “Classroom Walls That Talk: Using Online Course Activity Data of Successful Students to Raise Self-Awareness of Underperforming Peers.” The Internet and Higher Education 14(2): 89–97. https://doi.org/10.1016/j.iheduc.2010.07.007.
———. 2013. “Using Analytics at UMBC: Encouraging Student Responsibility and Identifying Effective Course Designs.” Louisville, CO, USA. https://library.educause.edu/resources/2013/4/using-analytics-at-umbc-encouraging-student-responsibility-and-identifying-effective-course-designs.
Garrison, D Randy, Terry Anderson, and Walter Archer. 2000. “Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education.” The Internet and Higher Education 2 (2-3): 87–105. https://doi.org/10.1016/S1096-7516(00)00016-6.
Garrison, D Randy, and J Ben Arbaugh. 2BC. “Researching the Community of Inquiry Framework: Review, Issues, and Future Directions.” The Internet and Higher Education 10 (3): 157–72. https://doi.org/10.1016/j.iheduc.2007.04.00.
Gašević, Dragan, Shane Dawson, Tim Rogers, and Danijela Gasevic. 2016. “Learning Analytics Should Not Promote One Size Fits All: The Effects of Instructional Conditions in Predicting Academic Success.” The Internet and Higher Education 28: 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002.
Gašević, Dragan, Vitomir Kovanović, and Srećko Joksimović. 2017. “Piecing the Learning Analytics Puzzle: A Consolidated Model of a Field of Research and Practice.” Learning: Research and Practice 3 (1): 63–78. https://doi.org/10.1080/23735082.2017.1286142.
Gasevic, Dragan, Yi-Shan Tsai, Shane Dawson, and Abelardo Pardo. 2019. “How Do We Start? An Approach to Learning Analytics Adoption in Higher Education.” The International Journal of Information and Learning Technology 36 (4): 342–53. https://doi.org/10.1108/IJILT-02-2019-0024.
Goldstein, Philip J, and Richard N Katz. 2005. “Academic Analytics: The Uses of Management Information and Technology in Higher Education.” Vol. 8. Louisville, CO, USA: EDUCAUSE Center for Applied Research. https://library.educause.edu/resources/2005/12/academic-analytics-the-uses-of-management-information-and-technology-in-higher-education.
Gottipatti, Swapna, and Venky Shankararaman. 2014. “Learning Analytics Applied to Curriculum Analysis.” In, 1–11. http://aisel.aisnet.org/siged2014/2.
Greer, Jim, C Thompson, R Banow, and Stephanie Frost. 2016. “Data-Driven Programmatic Change at Universities: What Works and How.” In Companion Proceedings, Sixth International Conference on Learning Analytics & Knowledge, 25:30–33. Edinburgh, Scotland, April 2016: Society for Learning Analytics Research (SoLAR). http://ceur-ws.org/Vol-1590/.
Greller, Wolfgang, and Hendrik Drachsler. 2012. “Translating Learning into Numbers: A Generic Framework for Learning Analytics.” Educational Technology & Society 15 (3): 42–57. http://www.jstor.org/stable/pdf/jeductechsoci.15.3.42.pdf.
Gupta, Amit, and S Anish. 2009. “Insights from Complexity Theory: Understanding Organizations Better.” IIMB Management Review 21 (3). http://tejas.iimb.ac.in/articles/12.php.
Hainey, Ainsley, Brian Green, Helyn Gould, Howard Ramsay, and C Milligan. 2018. “Developing an Institutional Policy Using the RAPID Outcome Monitoring Approach.” In Companion Proceedings, Eighth International Conference on Learning Analytics & Knowledge, 70–75. Sydney, Australia, March 2018: Society for Learning Analytics Research (SoLAR). http://bit.ly/lak18-companion-proceedings.
Hazy, James K, and Mary Uhl-Bien. 2014. “Changing the Rules: The Implications of Complexity Science for Leadership Research and Practice.” In The Oxford Handbook of Leadership and Organizations, edited by D Day, 709–32. Oxford: Oxford University Press.
Herodotou, Christothea, Bart Rienties, Martin Hlosta, Avinash Boroowa, Chrysoula Mangafa, and Zdenek Zdrahal. 2020. “The Scalable Implementation of Predictive Learning Analytics at a Distance Learning University: Insights from a Longitudinal Case Study.” The Internet and Higher Education 45: 100725. https://doi.org/10.1016/j.iheduc.2020.100725.
Herodotou, Christothea, Bart Rienties, Barry Verdin, and Avinash Boroowa. 2019. “Predictive Learning Analytics ‘at Scale’: Guidelines to Successful Implementation in Higher Education Based on the Case of the Open University UK.” Journal of Learning Analytics 6 (1): 85–95. https://doi.org/10.18608/jla.2019.61.5.
Huberth, M., N. Michelotti, and T. McKay. 2013. “E2Coach: Tailoring Support for Students in Introductory STEM Courses.” EDUCAUSE Review. https://er.educause.edu/articles/2013/12/e2coach-tailoring-support-for-students-in-introductory-stem-courses.
Jivet, Ioana, Maren Scheffel, Hendrik Drachsler, and Marcus Specht. 2017. “Awareness Is Not Enough: Pitfalls of Learning Analytics Dashboards in the Educational Practice.” In Data Driven Approaches in Digital Education. EC-TEL 2017. Lecture Notes in Computer Science, 10474, 82–95. Cham: Springer. https://doi.org/10.1007/978-3-319-66610-5_7.
Jivet, Ioana, Maren Scheffel, Marcus Specht, and Hendrik Drachsler. 2018. “License to Evaluate: Preparing Learning Analytics Dashboards for Educational Practice.” In Proceedings, Eighth International Conference on Learning Analytics & Knowledge, 31–40. New York, USA: ACM. https://doi.org/10.1145/3170358.3170421.
Klein, Carrie. 2020. “The Institutional Logics of Learning Analytics in Higher Education: Implications for Organizational and Individual Outcomes.” In Companion Proceedings, Tenth International Conference on Learning Analytics & Knowledge, 242–47. Virtual, March 2020: Society for Learning Analytics Research (SoLAR). https://www.solaresearch.org/wp-content/uploads/2020/06/LAK20_Companion_Proceedings.pdf.
Lauria, Eitel JM, Erik W Moody, Sandeep M Jayaprakash, Nagamani Jonnalagadda, and Joshua D Baron. 2013. “Open Academic Analytics Initiative: Initial Research Findings.” In Proceedings of the Third International Conference on Learning Analytics and Knowledge, 150–54.
Lim, Cher, and Victoria Tinio. 2018. Learning Analytics for the Global South. Quezon City, Philippines: Foundation for Information Technology Education and Development. http://dl4d.org/portfolio-items/learning-analytics-for-the-global-south/.
Lockyer, L., E. Heathcote, and Shane Dawson. 2013. “Informing Pedagogical Action: Aligning Learning Analytics with Learning Design.” American Behavioral Scientist 57 (10): 1439–59. https://doi.org/10.1177/0002764213479367.
Long, Phillip, and George Siemens. 2011. “Penetrating the Fog: Analytics in Learning and Education.” EDUCAUSE Review 45 (6). https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education.
Lonn, Steven, Stephen Aguilar, and Stephanie D. Teasley. 2013. “Issues, Challenges, and Lessons Learned When Scaling up a Learning Analytics Intervention.” In, 235–39. Leuven, Belgium: New York, USA: ACM. https://doi.org/10.1145/2460296.2460343.
Macfadyen, Leah P., and Shane Dawson. 2012. “Numbers Are Not Enough. Why E-Learning Analytics Failed to Inform an Institutional Strategic Plan.” Educational Technology & Society 15 (3): 149–63. http://www.jstor.org/stable/jeductechsoci.15.3.149.
Macfadyen, Leah P., Shane Dawson, Abelardo Pardo, and Dragan Gasevic. 2014. “Embracing Big Data in Complex Educational Systems: The Learning Analytics Imperative and the Policy Challenge.” Research & Practice in Assessment 9: 17–28. http://www.rpajournal.com/dev/wp-content/uploads/2014/10/A2.pdf.
MacLennan, Bruce. 2007. “Evolutionary Psychology, Complex Systems, and Social Theory.” Soundings: An Interdisciplinary Journal 90 (3/4): 169–89. http://www.jstor.org/stable/41179154.
Maldonado-Mahauad, Jorge, Isabel Hilliger, Tinne De Laet, Martijn Millecamp, Katrien Verbert, Xavier Ochoa, and Mar Pérez-Sanagustín. 2018. “The LALA Project: Building Capacity to Use Learning Analytics to Improve Higher Education in Latin America.” In Companion Proceedings, Eighth International Conference on Learning Analytics & Knowledge, 630–37. Sydney, Australia, March 2018: Society for Learning Analytics Research (SoLAR). http://bit.ly/lak18-companion-proceedings.
Manyika, James, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh, and Angela Hung Byers. 2011. “Big Data: The Next Frontier for Innovation, Competition and Productivity.” http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation.
Mazziotti, Claudia, Vitomir. Kovanović, Shane Dawson, and George Siemens. 2020. “Implementing Learning Analytics in Schools: Towards a Theory-Based and Data-Driven Framework for Analytics Implementation.” In Companion Proceedings, Tenth International Conference on Learning Analytics & Knowledge, 116–18. Virtual, March 2020: Society for Learning Analytics Research (SoLAR). https://www.solaresearch.org/wp-content/uploads/2020/06/LAK20_Companion_Proceedings.pdf.
McIntosh, Naomi E. 1979. “Barriers to Implementing Research in Higher Education.” Studies in Higher Education 4 (1): 77–86. https://doi.org/10.1080/03075077912331377121.
McKay, Tim, Kate Miller, and Jared Tritz. 2012. “What to Do with Actionable Intelligence: E2Coach as an Intervention Engine.” In, 88–91. Vancouver, BC, Canada: New York, USA: ACM. https://doi.org/10.1145/2330601.2330627.
Mendez, Gonzalo, Xavier Ochoa, Katherine Chiluiza, and Bram de Wever. 2014. “Curricular Design Analysis: A Data-Driven Perspective.” Journal of Learning Analytics 1 (3). https://doi.org/10.18608/jla.2014.13.6.
Mitleton-Kelly, Eve. 2003. “Ten Principles of Complexity & Enabling Infrastructures.” In Complex Systems & Evolutionary Perspectives of Organisations: The Application of Complexity Theory to Organisations, edited by Eve Mitleton-Kelly, 23–50. Bingley, UK: Emerald Group.
Norris, Donald M, and Linda Baer. 2013. “Building Organizational Capacity for Analytics.” EDUCAUSE. https://library.educause.edu/resources/2013/2/building-organizational-capacity-for-analytics.
Oliver, Beverley. 2013. “Graduate Attributes as a Focus for Institution-Wide Curriculum Renewal: Innovations and Challenges” 32 (3): 450–63. https://doi.org/10.1080/07294360.2012.682052.
Pelánek, Radek. 2020. “Learning Analytics Challenges: Trade-Offs, Methodology, Scalability.” In, 554–58. New York, USA: ACM. https://doi.org/10.1145/3375462.3375463.
Prinsloo, Paul. 2020. “Data Frontiers and Frontiers of Power in (Higher) Education: A View of/from the Global South.” Teaching in Higher Education 25 (4): 366–83. https://doi.org/10.1080/13562517.2020.1723537.
Sanagustín1, Mar Pérez, Isabel Hilliger1, Jorge Maldonado1, Ronald Pérez, Luís Ramírez1, Pedro J. Muñoz-Merino, Yi-Shan Tsai, et al. 2019. “LALA Framework.” https://www.lalaproject.org/wp-content/uploads/2019/04/LALA_framework_English.pdf.
Scheffel, Maren. 2017. “The Evaluation Framework for Learning Analytics.” Open Universiteit in the Netherlands at Welten Institute – Research Centre for Learning, Teaching and Technology. The Netherlands: Open Universiteit in the Netherlands.
Scheffel, Maren, Hendrik Drachsler, Christian Toisoul, Stefaan Ternier, and Marcus Specht. 2017. “The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics.” In, 194–208. Cham: Springer. https://doi.org/10.1007/978-3-319-66610-5_15.
Sclater, Niall, Alice Peasgood, and Joel Mullan. 2016. “Learning Analytics in Higher Education. A Review of UK and International Practice.” Jisc Reports. https://www.jisc.ac.uk/reports/learning-analytics-in-higher-education.
Siemens, George, Shane Dawson, and Grace Lynch. 2013. “Improving the Quality and Productivity of the Higher Education Sector. Policy and Strategy for Systems-Level Deployment of Learning Analytics.” https://solaresearch.org/wp-content/uploads/2017/06/SoLAR_Report_2014.pdf.
Slade, Sharon, and Paul Prinsloo. 2013. “Learning Analytics: Ethical Issues and Dilemmas.” American Behavioral Scientist 57 (10): 1510–29. https://doi.org/10.1177/0002764213479366.
Thornton, Patricia H, William Ocasio, and Michael Lounsbury. 2012. The Institutional Logics Perspective: A New Approach to Culture, Structure, and Process. Ocford, England: Oxford University Press.
Tsai, Yi-Shan, Pedro Manuel Moreno-Marcos, Ioana Jivet, Maren Scheffel, Kairit Tammets, Kaire Kollom, and Dragan Gašević. 2018. “The SHEILA Framework: Informing Institutional Strategies and Policy Processes of Learning Analytics.” Journal of Learning Analytics 5 (3): 5–20. https://doi.org/10.18608/jla.2018.53.2.
Tsai, Yi-Shan, Diego Rates, Pedro Manuel Moreno-Marcos, Pedro J Muñoz-Merino, Ioana Jivet, Maren Scheffel, Hendrik Drachsler, Carlos Delgado Kloos, and Dragan Gašević. 2020. “Learning Analytics in European Higher Education–Trends and Barriers.” Computers & Education, 103933. https://doi.org/10.1016/j.compedu.2020.103933.
Viberg, Olga, Mathias Hatakka, Olof Bälter, and Anna Mavroudi. 2018. “The Current Landscape of Learning Analytics in Higher Education.” Computers in Human Behavior 89: 98–110. https://doi.org/10.1016/j.chb.2018.07.027.
Wexler, Ellen. 2017. “This Mathematician Brought Big Data to Advising. Then Deeper Questions Emerged.” EdSurge, April. https://www.edsurge.com/news/2017-04-13-this-mathematician-brought-big-data-to-advising-then-deeper-questions-emerged.
Winne, Philip. 2017. “Learning Analytics for Self-Regulated Learning.” In The Handbook of Learning Analytics, edited by Charles Lang, George Siemens, Alyssa Friend Wise, and Dragan Gaševic, 1st ed., 241–49. Alberta, Canada: Society for Learning Analytics Research (SoLAR). http://solaresearch.org/hla-17/hla17-chapter1.
Yanosky, Ronald, and Pam Arroway. 2015. “The Analytics Landscape in Higher Education.” Louisville, CO, USA: EDUCAUSE. https://library.educause.edu/resources/2016/1/the-analytics-landscape-in-higher-education-2015.
Young, John, and Enrique Mendizabel. 2009. “Helping Researchers Become Policy Entrepreneurs. How to Develop Engagement Strategies for Evidence-Based Policy-Making.” London, UK. https://www.odi.org/resources/docs/1730.pdfhttps://www.odi.org/publications/1127-helping-researchers-become-policy-entrepreneurs.
No Supplementary Material Available
Title
Institutional Analytics
Book Title
Handbook of Learning Analytics
Pages
pp. 173-186
Copyright
2022
DOI
10.18608/hla22.017
ISBN
978-0-9952408-3-4
Publisher
Society for Learning Analytics Research
Authors
Leah P. Macfadyen
Editors
Charles Lang
Alyssa Friend Wise
Agathe Merceron
Dragan Gašević
George Siemens