Handbook of Learning Analytics
Chapter 10
Natural Language Processing: Writing Analytics
Andrew Gibson & Antonette Shibani
Abstract
Writing analytics uses computational techniques to analyse written texts for the purposes of improving learning. This chapter provides an introduction to writing analytics, through the discussion of linguistic and domain orientations to the analysis of writing, and descriptive and evaluative intentions for the analytics. The chapter highlights the importance of the relationship between writing analytics and good pedagogy, contending that for writing analytics to positively impact learning, actionability must be considered in the design process. Limitations of writing analytics are also discussed, highlighting areas of concern for future research.
Allen, Laura, Sarah C. Creer, and Püren Öncel. 2022. “Natural Language Processing as a Tool for Learning Analytics - Towards a Multi-Dimensional View of the Learning Process.” In The Handbook of Learning Analytics, edited by Charles Lang, Alyssa Friend Wise, Agathe Merceron, Dragan Gašević, and George Siemens, 2nd ed. Vancouver, Canada: SOLAR. https://www.solaresearch.org/publications/hla-22/.
Allen, Laura K., Matthew E. Jacovina, Mihai Dascalu, Rod D. Roscoe, Kevin M. Kent, Aaron D. Likens, and Danielle S. McNamara. 2016. “{ENTER} Ing the Time Series {Space}: Uncovering the Writing Process Through Keystroke Analyses.” International Educational Data Mining Society.
Allen, Laura K., Cecile Perret, Aaron Likens, and Danielle S. McNamara. 2017. “What’d You Say Again? Recurrence Quantification Analysis as a Method for Analyzing the Dynamics of Discourse in a Reading Strategy Tutor.” In Proceedings of the Seventh International Learning Analytics & Knowledge Conference, 373–82.
Anthony, Laurence, and George V. Lashkia. 2003. “Mover: A Machine Learning Tool to Assist in the Reading and Writing of Technical Papers.” IEEE Transactions on Professional Communication 46 (3): 185–93.
Attali, Yigal. 2004. “Exploring the Feedback and Revision Features of Criterion.” Journal of Second Language Writing 14: 191–205.
Bahr, Nan, and Donna Pendergast. 2018. “Let’s Abandon Naplan – We Can Do Better.” The Conversation. https://theconversation.com/lets-abandon-naplan-we-can-do-better-95363.
Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. “Latent Dirichlet Allocation.” The Journal of Machine Learning Research 3: 993–1022.
Buckingham Shum, Simon, and T. McKay. 2018. “Architecting Universities for Learning Analytics: Innovating for Sustainable Impact.” EDUCAUSE Review. https://er.educause.edu/-/media/files/articles/2018/3/er182101.pdf.
Chen, Chi-Fen Emily, and Wei-Yuan Eugene Cheng Cheng. 2008. “Beyond the Design of Automated Writing Evaluation: Pedagogical Practices and Perceived Learning Effectiveness in Efl Writing Classes.” Language Learning & Technology 12 (2): 94–112.
Cheville, Julie. 2004. “Automated Scoring Technologies and the Rising Influence of Error.” The English Journal 93 (4): 47–52.
Chomsky, Noam. 1965. Aspects of the Theory of Syntax. Cambridge, MA: MIT Press.
Cotos, Elena, and Nick Pendar. 2016. “Discourse Classification into Rhetorical Functions for Awe Feedback.” Calico Journal 33 (1): 92.
Crossley, Scott A, Kristopher Kyle, Laura K Allen, Liang Guo, and Danielle S McNamara. 2014. “Linguistic Microfeatures to Predict L2 Writing Proficiency: A Case Study in Automated Writing Evaluation.” Grantee Submission 7 (1).
Dawson, Shane, Srecko Joksimovic, Oleksandra Poquet, and George Siemens. 2019. “Increasing the Impact of Learning Analytics.” In Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 446–55.
Dik, Simon C. 1997. The Theory of Functional Grammar: The Structure of the Clause. Walter de Gruyter.
Dowell, Nia, and Vitomir Kovanović. 2022. “Modeling Educational Discourse with Natural Language Processing.” In The Handbook of Learning Analytics, edited by Charles Lang, Alyssa Friend Wise, Agathe Merceron, Dragan Gašević, and George Siemens, 2nd ed. Vancouver, Canada: SOLAR. https://www.solaresearch.org/publications/hla-22/.
Dowell, Nia M, Oleksandra Skrypnyk, Srecko Joksimovic, Arthur C Graesser, Shane Dawson, Dragan Gaševic, Thieme A Hennis, Pieter de Vries, and Vitomir Kovanovic. 2015. “Modeling Learners’ Social Centrality and Performance Through Language and Discourse.” International Educational Data Mining Society.
Driscoll, Dana Lynn, and Roger Powell. 2016. “States, Traits, and Dispositions: The Impact of Emotion on Writing Development and Writing Transfer Across College Courses and Beyond.” In Composition Forum. Vol. 34. ERIC.
Duval, Erik. 2011. “Attention Please! Learning Analytics for Visualization and Recommendation.” In, 9–17.
Flower, Linda, and John R. Hayes. 1981. “A Cognitive Process Theory of Writing.” College Composition and Communication 32 (4): 365–87.
Gašević, Dragan, Shane Dawson, and George Siemens. 2015. “Let’s Not Forget: Learning Analytics Are About Learning.” TechTrends 59 (1): 64–71.
Gibson, Andrew, Kirsty Kitto, and Peter Bruza. 2016. “Towards the Discovery of Learner Metacognition from Reflective Writing.” Journal of Learning Analytics 3 (2): 22–36.
Gibson, Andrew, and Charles Lang. 2018. “The Pragmatic Maxim as Learning Analytics Research Method.” In, 461–65.
Graham, Steve, Amy Gillespie, and Debra McKeown. 2013. “Writing: Importance, Development, and Instruction.” Reading and Writing 26 (1): 1–15.
Harlen, Wynne. 2005. “Teachers’ Summative Practices and Assessment for Learning–Tensions and Synergies.” Curriculum Journal 16 (2): 207–23.
Hyland, Ken. 2007. “Genre Pedagogy: Language, Literacy and L2 Writing Instruction.” Journal of Second Language Writing 16 (3): 148–64.
icol, David J., and Debra Macfarlane-Dick. 2006. “Formative Assessment and Self‐regulated Learning: A Model and Seven Principles of Good Feedback Practice.” Studies in Higher Education 31 (2): 199–218.
Jivet, Ioana, Maren Scheffel, Marcus Specht, and Hendrik Drachsler. 2018. “License to Evaluate: Preparing Learning Analytics Dashboards for Educational Practice.” In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 31–40.
Kalantzis, Mary, Bill Cope, and Andrew Harvey. 2003. “Assessing Multiliteracies and the New Basics.” Assessment in Education: Principles, Policy & Practice 10 (1): 15–26.
Kitto, Kirsty, Simon Buckingham Shum, and Andrew Gibson. 2018. “Embracing Imperfection in Learning Analytics.” In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, 451–60.
Knight, Simon, Antonette Shibani, Sophie Abel, Andrew Gibson, Philippa Ryan, Nicole Sutton, Raechel Wight, et al. 2020. “AcaWriter: A Learning Analytics Tool for Formative Feedback on Academic Writing.” Journal of Writing Research 12 (1): 299–344.
Learning, Vantage. 2003. “How Does Intellimetric Score Essay Responses.” http://www.vantage.com/pdfs/research/RB929.pdf.
Li, Jinrong, Stephanie Link, and Volker Hegelheimer. 2015. “Rethinking the Role of Automated Writing Evaluation (Awe) Feedback in Esl Writing Instruction.” Journal of Second Language Writing 27: 1–18.
Manning, Christopher D., Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. “The Stanford Corenlp Natural Language Processing Toolkit.” In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 55–60.
Mayfield, Elijah, and Stephanie Butler. 2018. “Districtwide Implementations Outperform Isolated Use of Automated Feedback in High School Writing.”
McDonald, Jenny, and Adon Christian Michael Moskal. 2017. “Quantext: Analysing Student Responses to Short-Answer Questions.” In, 133–37.
McNamara, Danielle S., Scott A. Crossley, and Philip M. McCarthy. 2010. “Linguistic Features of Writing Quality.” Written Communication 27 (1): 57–86.
McNamara, Danielle S., Arthur C. Graesser, Philip M. McCarthy, and Zhiqiang Cai. 2014. Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge University Press.
Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. “Distributed Representations of Words and Phrases and Their Compositionality.” In, 3111–9. Advances in Neural Information Processing Systems.
Mørch, Anders I., Irina Engeness, Victor C Cheng, William K Cheung, and Kelvin C Wong. 2017. “EssayCritic: Writing to Learn with a Knowledge-Based Design Critiquing System.” Journal of Educational Technology & Society 20 (2): 213–23.
Norvig, Peter. 2012. “Colorless Green Ideas Learn Furiously: Chomsky and the Two Cultures of Statistical Learning.” Significance 9 (4): 30–33.
Pennebaker, James W., Martha E. Francis, and Roger J. Booth. 2001. “Linguistic Inquiry and Word Count: LIWC 2001.” Mahway: Lawrence Erlbaum Associates 71 (2001).
Poquet, Oleksandra, and Srećko Joksimović. 2022. “Cacophony of Networks in Learning Analytics.” In The Handbook of Learning Analytics, edited by Charles Lang, Alyssa Friend Wise, Agathe Merceron, Dragan Gašević, and George Siemens, 2nd ed. Vancouver, Canada: SOLAR. https://www.solaresearch.org/publications/hla-22/.
Powers, Donald E., Jill C. Burstein, Martin S. Chodorow, Mary E. Fowles, and Karen Kukich. 2002. “Comparing the Validity of Automated and Human Scoring of Essays.” Journal of Educational Computing Research 26 (4): 407–25. https://doi.org/10.2190/CX92-7WKV-N7WC-JL0A.
Prieto-Alvarez, Carlos G., Roberto Martinez-Maldonado, and Theresa Dirndorfer Anderson. 2018. Co-Designing Learning Analytics Tools with Learners. Learning Analytics in the Classroom: Translating Learning Analytics Research for Teachers. Abingdon-on-Thames, UK: Routledge.
Rahimi, Zahra, Diane Litman, Richard Correnti, Elaine Wang, and Lindsay Clare Matsumura. 2017. “Assessing Students’ Use of Evidence and Organization in Response-to-Text Writing: Using Natural Language Processing for Rubric-Based Automated Scoring.” International Journal of Artificial Intelligence in Education 27 (4): 694–728.
Roscoe, Rod D., and Danielle S. McNamara. 2013. “Writing Pal: Feasibility of an Intelligent Writing Strategy Tutor in the High School Classroom.” Journal of Educational Psychology 105 (4): 1010.
Shibani, Antonette. 2019. “Augmenting Pedagogic Writing Practice with Contextualizable Learning Analytics.” PhD thesis, Australia: University of Technology Sydney. http://hdl.handle.net/10453/136846.
Shibani, Antonette, Simon Knight, and Simon Buckingham Shum. 2019. “Contextualizable Learning Analytics Design: A Generic Model and Writing Analytics Evaluations.” In, 210–19.
Southavilay, Vilaythong, Kalina Yacef, Peter Reimann, and Rafael A. Calvo. 2013. “Analysis of Collaborative Writing Processes Using Revision Maps and Probabilistic Topic Models.” In Proceedings of the Third International Conference on Learning Analytics and Knowledge, 38–47.
Taras, Maddalena. 2008. “Summative and Formative Assessment: Perceptions and Realities.” Active Learning in Higher Education 9 (2): 172–92.
Villalón, Jorge, Paul Kearney, Rafael A. Calvo, and Peter Reimann. 2008. “Glosser: Enhanced Feedback for Student Writing Tasks.” In, 454–58. IEEE.
Weigand, Edda. 2019. “Dialogue and Artificial Intelligence.” Language and Dialogue 9 (2): 294–315.
Young, Tom, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. “Recent Trends in Deep Learning Based Natural Language Processing.” IEEE Computational intelligenCe Magazine 13 (3): 55–75.
Zhang, Fan. 2017. “Towards Building an Intelligent Revision Assistant for Argumentative Writings.” PhD thesis, University of Pittsburgh.
No Supplementary Material Available
Title
Natural Language Processing: Writing Analytics
Book Title
Handbook of Learning Analytics
Pages
pp. 96-104
Copyright
2022
DOI
10.18608/hla22.010
ISBN
978-0-9952408-3-4
Publisher
Society for Learning Analytics Research
Authors
Andrew Gibson
Antonette Shibani
Editors
Charles Lang
Alyssa Friend Wise
Agathe Merceron
Dragan Gašević
George Siemens