Using AI to understand students’ self-assessments of their writing
DOI:
https://doi.org/10.17239/jowr-2026.17.03.07Keywords:
artificial intelligence, qualitative analysis, writing assessment, writing placementAbstract
This study focuses on a generative AI approach to facilitate qualitative analysis in Writing Studies research. We gathered 13,336 one-sentence to one-paragraph responses written by 3,334 incoming students in a directed self-placement program administered at a large R1 U.S. university. In these responses, students describe their high school writing experience and college writing expectations. In stage one of the project, we pilot the use of Retrieval-Augmented Generation to expedite the selection of relevant responses for a topic—in this case, students’ positive self-assessments as writers. The selected responses were then compared to a random sample and rated by three faculty with writing expertise. In stage two, these faculty generated codes and themes from a subset of the responses, incorporating ChatGPT-4 through the stages of thematic analysis. Results show that the use of AI expedites and enhances qualitative analysis, but human participation in the process is still essential. We suggest a machine-in-the-loop framework with which Writing Studies researchers can more readily integrate generative AI to study large corpora of student writing.
References
Au, W. (2022). Unequal by design: High-stakes testing and the standardization of inequality. Routledge. https://doi.org/10.4324/9781003005179
Baillargeon, K. (2025). Reflective orientations and genre navigation in dissertation writing: Insights from seven years of doctoral writing retreats. Journal of Writing Research. https://www.jowr.org/jowr/article/view/1670
Balaguer, A., Benara, V., de Freitas Cunha, R. L., Estevão Filho, R. D. M., Hendry, T., Holstein, D., ... & Chandra, R. (2024). RAG vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv e-prints, arXiv-2401. https://doi.org/10.48550/arXiv.2401.08406
Barany, A., Nasiar, N., Porter, C., Zambrano, A. F., Andres, A. L., Bright, D., ... & Baker, R. S. (2024, July). ChatGPT for education research: Exploring the potential of large language models for qualitative codebook development. In International Conference on Artificial Intelligence in Education (pp. 134-149). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-64299-9_10
Bartholomae, D. (2005). Inventing the university. Writing on the margins: Essays on composition and teaching, 60-85. https://doi.org/10.37514/JBW-J.1986.5.1.02
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Britton, J. (1975). The Development of Writing Abilities (11-18).
Connors, R. & Lunsford, A.A. (1988). Frequency of Formal Error in Current College Writing, or Ma and Pa Kettle Do Research.” College Composition and Communication, 39.4: 395–409. https://doi.org/10.2307/357695
Dai, S. C., Xiong, A., & Ku, L. W. (2023). LLM-in-the-loop: Leveraging large language model for thematic analysis. arXiv preprint arXiv:2310.15100. https://doi.org/10.18653/v1/2023.findings-emnlp.669
De Paoli, S. (2024). Performing an inductive thematic analysis of semi-structured interviews with a large language model: An exploration and provocation on the limits of the approach. Social Science Computer Review, 42(4), 997-1019. https://doi.org/10.1177/08944393231220483
Feuston, J. L., & Brubaker, J. R. (2021). Putting tools in their place: The role of time and perspective in human-AI collaboration for qualitative analysis. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1-25. https://doi.org/10.1145/3479856
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., ... & Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997. https://doi.org/10.48550/arXiv.2312.10997
Gebreegziabher, S. A., Zhang, Z., Tang, X., Meng, Y., Glassman, E. L., & Li, T. J. J. (2023, April). Patat: Human-ai collaborative qualitative coding with explainable interactive rule synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1-19). https://doi.org/10.1145/3544548.3581352
Gere, A. (2019). Developing writers in higher education: A longitudinal study (p. 385). University of Michigan Press. https://doi.org/10.3998/mpub.10079994
Gere, A. R., Aull, L., Escudero, M. D. P., Lancaster, Z., & Vander Lei, E. (2013). Local assessment: Using genre analysis to validate directed self-placement. College Composition & Communication, 64(4), 605-633. https://doi.org/10.58680/ccc201323661
Hamilton, L., Elliott, D., Quick, A., Smith, S., & Choplin, V. (2023). Exploring the use of AI in qualitative analysis: A comparative study of guaranteed income data. International Journal of Qualitative Methods, 22, https://doi.org/10.1177/16094069231201504
Haswell, R. H. (2005). NCTE/CCCC’s recent war on scholarship. Written Communication, 22(2), 198-223. https://doi.org/10.1177/0741088305275367.
Hitch, D. (2024). Artificial Intelligence Augmented Qualitative Analysis: The Way of the Future?. Qualitative Health Research, 34(7), 595-606. https://doi.org/10.1177/10497323231217392
Hong, M. H., Marsh, L. A., Feuston, J. L., Ruppert, J., Brubaker, J. R., & Szafir, D. A. (2022, October). Scholastic: Graphical human-AI collaboration for inductive and interpretive text analysis. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (pp. 1-12). https://doi.org/10.1145/3526113.3545681
Ibrahim, E. I., & Voyer, A. (2024). The Augmented Qualitative Researcher: Using Generative AI in Qualitative Text Analysis. SocArXiv Preprints. https://doi.org/10.31235/osf.io/gkc8w.
Jalali, M. S., & Akhavan, A. (2024). Integrating AI Language Models in Qualitative Research: Replicating Interview Data Analysis with ChatGPT. System Dynamics Review, 40(3), e1772. https://doi.org/10.1002/sdr.1772
Jiang, J. A., Wade, K., Fiesler, C., & Brubaker, J. R. (2021). Supporting serendipity: Opportunities and challenges for Human-AI Collaboration in qualitative analysis. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1-23. https://doi.org/10.1145/3449168
Kantor, J. (2024). Best practices for implementing ChatGPT, large language models, and artificial intelligence in qualitative and survey-based research. JAAD international, 14, 22-23. https://doi.org/10.1016/j.jdin.2023.10.001.
Lee, S., Shakir, A., Koenig, D., & Lipp, J. (2024). Open Source Strikes Bread—New Fluffy Embeddings Model. https://www.mixedbread.ai/blog/mxbai-embed-large-v1
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459-9474. https://doi.org/10.5555/3495724.3496517.
Licastro, A., & Miller, B. M. (Eds.). (2021). Composition and big data. University of Pittsburgh Press. https://doi.org/10.2307/j.ctv22tnmg2
Lubars, B., & Tan, C. (2019). Ask not what AI can do, but what AI should do: Towards a framework of task delegability. Advances in neural information processing systems, 32. https://doi.org/10.5555/3454287.3454293
Lunsford, A. A., Fishman, J., & Liew, W. M. (2013). College writing, identification, and the production of intellectual property: Voices from the Stanford study of writing. College English, 75(5), 470-492.
Lunsford, A. A., & Lunsford, K. J. (2008). “Mistakes are a fact of life”: A national comparative study. College Composition & Communication, 59(4), 781-806. https://doi.org/10.58680/ccc20086677.
Melzer, D. (2009). Writing assignments across the curriculum: A national study of college writing. College Composition & Communication, 61(2), W240-W261. https://doi.org/10.58680/ccc20099487.
Meng, H., Yang, Y., Li, Y., Lee, J., & Lee, Y. C. (2024). Exploring the Potential of Human-LLM Synergy in Advancing Qualitative Analysis: A Case Study on Mental-Illness Stigma. arXiv preprint arXiv:2405.05758. https://doi.org/10.48550/arXiv.2405.05758
Mesec, B. (2023). The language model of artificial inteligence chatGPT-a tool of qualitative analysis of texts. Authorea Preprints. https://doi.org/10.22541/au.168182047.70243364/v1.
Moos, A., & Van Zanen, K. (2019). Directed self-placement as a tool to foreground student agency. Assessing Writing, 41, 68-71. https://doi.org/10.1016/j.asw.2019.06.001.
Morgan, D. L. (2023). Exploring the use of artificial intelligence for qualitative data analysis: The case of ChatGPT. International journal of qualitative methods, 22, 16094069231211248. https://doi.org/10.1177/16094069231211248.
Nguyen-Trung, K. (2024). ChatGPT in Thematic Analysis: Can AI become a research assistant in qualitative research?. OSF Preprint.
OpenAI. (2023). text-embedding-3-small. Retrieved from https://platform.openai.com/docs/guides/embeddings
Ovadia, O., Brief, M., Mishaeli, M., & Elisha, O. (2024). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. arXiv preprint arXiv:2312.05934.
Paulus, T. M., & Marone, V. (2024). In Minutes Instead of Weeks: Discursive Constructions of Generative AI and Qualitative Data Analysis. Qualitative Inquiry, 30(1), 10778004241250065. https://doi.org/10.1177/10778004241250065
Perkins, M., & Roe, J. (2024). The use of Generative AI in qualitative analysis: Inductive thematic analysis with ChatGPT. Journal of Applied Learning and Teaching, 7(1), Article 1. https://doi.org/10.37074/jalt.2024.7.1.22
Rietz, T., & Maedche, A. (2021, May). Cody: An AI-based system to semi-automate coding for qualitative research. In Proceedings of the 2021 CHI conference on human factors in computing systems (pp. 1-14). https://doi.org/10.1145/3411764.3445591
Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(6), Article 887. https://doi.org/10.3390/healthcare11060887
Sinha, R., Solola, I., Nguyen, H., Swanson, H., & Lawrence, L. (2024, June). The Role of Generative AI in Qualitative Research: GPT-4's Contributions to a Grounded Theory Analysis. In Proceedings of the Symposium on Learning, Design and Technology (pp. 17–25). https://doi.org/10.1145/3663433.3663456
Sommers, N. (2006). Across the drafts. College Composition & Communication, 58(2), 246-266. https://doi.org/10.58680/ccc20065899
Sommers, N. (2008). The call of research: A longitudinal view of writing development. College Composition and Communication, 60(1), 152-164.
Tinkle, T., Godfrey, J., Hammond, J. W., & Moos, A. (2024). Self-Characterization in the Self-Placement Assessment Ecology: Complicating the Stories We Tell about DSP’s Effects and Effectiveness. Journal of Writing Assessment, 17(1). https://doi.org/10.5070/W4jwa.1625.
Turobov, A., Coyle, D., & Harding, V. (2024). Using ChatGPT for thematic analysis. arXiv preprint arXiv:2405.08828.
Wachinger, J., Bärnighausen, K., Schäfer, L. N., Scott, K., & McMahon, S. A. (2024). Prompts, Pearls, Imperfections: Comparing ChatGPT and a Human Researcher in Qualitative Data Analysis. Qualitative Health Research, 34(7), 10497323241244669. https://doi.org/10.1177/10497323241244669
Yan, L., Echeverria, V., Fernandez-Nieto, G. M., Jin, Y., Swiecki, Z., Zhao, L., Gašević, D., & Martinez-Maldonado, R. (2024). Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations. Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems, 1–7. https://doi.org/10.1145/3613905.3650732
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2023). Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis. arXiv preprint arXiv:2309.10771. https://doi.org/10.48550/arXiv.2309.10771
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., ... & Cui, B. (2024). Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473.
Published
Issue
Section
License
Copyright (c) 2026 Madeleine Sorapure, Seth Erickson, Sarah Hirsch, Kenny Smith
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported License.
