Can ChatGPT do the same? ChatGPT and professional editors compared
DOI:
https://doi.org/10.17239/jowr-2026.17.03.02Keywords:
AI, ChatGPT, textual characteristic, professional writing, professional editing, collaborative writingAbstract
Since the launch of ChatGPT, the use of and debate around generative AI has grown rapidly. Professionals whose work depends on writing have expressed concern about the potential impact of such tools on their roles. But are these concerns justified? Can ChatGPT truly take on the responsibilities of a professional writer? This study investigates that question by comparing the performance of ChatGPT with that of professional editors tasked with optimizing business communication. We conducted two studies, using both qualitative and quantitative methods. In the first, three experienced editors were asked to rewrite four business letters. Their editing processes were recorded using the Microsoft Snipping Tool, and immediately afterward, we conducted retrospective interviews using stimulated recall. These interviews were transcribed and analyzed. Insights from the observations and interviews informed the design of the prompt instructions used in the second study. In the second study, we asked ChatGPT to revise the same four letters using three different prompt types. The Simple prompt instructed the model to “make this text reader-focused.” The B1 prompt referred explicitly to the CEFR B1 language level, requiring ChatGPT to tailor the text for intermediate readers. Finally, the Process prompt simulated the editing steps observed in the professional editors’ workflows. To evaluate outcomes, we conducted both a qualitative comparison of the revised texts and a quantitative readability analysis using LiNT, a validated tool developed for Dutch texts. Our results show that the human editors substantially improved the readability of the original letters, reducing the use of unfamiliar words, shortening complex sentences, and increasing personal engagement through pronoun use. Among the AI outputs, ChatGPT B1 achieved results most comparable to the editors, both in readability and accuracy. In contrast, ChatGPT Simple fell short in terms of clarity and introduced errors through faulty inferences. Surprisingly, ChatGPT Process also underperformed compared to ChatGPT B1 and the human editors. Only the editors' and ChatGPT B1versions were free from errors. In the discussion, we reflect on how generative AI is reshaping the concept of writing within organizations, the skills required to produce effective written communication and the impact on writing pedagogy. Rather than replacing human editors, we argue that generative AI can play a valuable role as a collaborative tool in the organizational writing process.
References
Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus, 15(2). 0.7759/cureus.35179
Anthropic. (2023, July 11). Claude 2. https://www.anthropic.com/news/claude-2 Anthropic
Anzidei, Christopher M. (2002) The Revision Process in Legal Writing: Seeing Better to Write Better. Legal Writing 23, 8 (2002), 23-57.
Arabas, C. (1990). Technical Writing in a Corporate Culture; A Study of the Nature of Information. New Jersey: Sablex.
Benites, F., Delorme Benites, A. & Anson, C. M. (2023). Automatic Text Generation and Summarization for Academic Writing. Kruse, O. et al. (eds), Digital Writing Technologies in Higher Education, Springer. https://doi.org/10.1007/978-3-031-36033-6.
Bisaillon, J. (2007). Professional Editing Strategies Used by Six Editors. Written Communication, 24(4), 295-322. https://doi.org/10.1177/0741088307305977
Brynjolfsson, E., Li, D., & Raymond, L. R. (2023). Generative AI at work (NBER Working Paper No. 31161). National Bureau of Economic Research. https://doi.org/10.3386/w31161
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI‑assisted decision‑making. Proceedings of the ACM on Human‑Computer Interaction, 5(CSCW1), Article 188. https://doi.org/10.1145/3449287
Burrough-Boenisch, J. (2006). Negotiable Acceptability: Reflections on the Interactions between Language Professionals in Europe and NNS 1 Scientists Wishing to Publish in English. Current Issues in Language Planning, 7(1), 31–43. https://doi.org/10.2167/cilp086.0
Calderhead, J. (1981. Stimulated Recall: a Method for Research on Teaching. British Journal of Educational Psychology 51(2):211-217. https://doi.org/10.1111/j.2044-8279.1981.tb02474.x
Council of Europe. (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment—Companion volume. Council of Europe Publishing. https://rm.coe.int/common-european-framework-of-reference-for-languages-learning-teaching/16809ea0d4
Council of Writing Program Administrators. (2014). WPA Outcomes Statement for First‑Year Composition (Version 3.0). https://wpacouncil.org/aws/CWPA/pt/sp/statements
Cross, G. A. (1990). A Bakhtinian exploration of factors affecting the collaborative writing of an executive letter of an annual report. Research in the Teaching of English, 24(2), 173-203.
Dell’Acqua, F., et al. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School Technology & Operations Management Working Paper (24‑013). https://doi.org/10.2139/ssrn.4573321
Dratsch, T., Chen, X., Rezazade Mehrizi, M. H., Mähringer‑Kunz, A., Püsken, M., Baeßler, B., Sauer, S., Maintz, D., Pinto dos Santos, D., & Kloeckner, R. (2023). Automation bias in mammography: The impact of AI BI‑RADS suggestions on reader performance. Radiology, 307(4), e222176. https://doi.org/10.1148/radiol.222176
Duin, A. H. (1991). Computer-Supported Collaborative Writing: The Workplace and the Writing Classroom. Journal of Business and Technical Communication, 5(2), 123-150. https://doi.org/10.1177/1050651991005002001
Ede, L and Lunsford, A. (1990). Singular Texts /Plural Authors: Perspectives on Collaborative Writing. Carbondale. Southern Illinois Press.
Faigley, L., & Witte, S. (1981). Analyzing revision. College Composition and Communication, 32(4), 400–414. https://doi.org/10.2307/357848
Flower, L., & Hayes, J. R. (1981). A Cognitive Process Theory of Writing. College Composition and Communication, 32(4), 365–387. https://doi.org/10.2307/356600
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021). Do as AI say: Susceptibility in deployment of clinical decision‑aids. NPJ Digital Medicine, 4, 31. https://doi.org/10.1038/s41746-021-00385-9
Google. (2023, August 29). Write emails, documents and more with ease and speed using Duet AI in Google Docs and Gmail. Google Workspace Updates. Workspace Updates Blog
Google. (2023, July 13). Bard’s latest update: More features, languages and countries. https://blog.google/products/gemini/google-bard-new-features-update-july-2023/
Graham, Steve. (2018). A Revised Writer(s)-Within-Community Model of Writing. Educational Psychologist. 53. 1-22. 10.1080/00461520.2018.1481406.
Hayes, J. R. (2012). Modeling and Remodeling Writing. Written Communication, 29(3), 369-388. https://doi.org/10.1177/0741088312451260
Hayes, J. R., Flower, L., Schriver, K. A., Stratman, J. F., & Carey, L. (1987). Cognitive processes in revision. In S. Rosenberg (Ed.), Advances in applied psycholinguistics, Vol. 2. Reading, writing, and language learning (pp. 176–240). Cambridge NJ: Cambridge University Press.
Heilmann, Till. A. ( (2023). The Beginning of Word Processing; A Historical Account. Kruse, O. et al. (eds), Digital Writing Technologies in Higher Education, Springer. https://doi.org/10.1007/978-3-031-36033-6.
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023). A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports, 13 (18617). https://doi.org/10.1038/s41598-023-45644-9
Horning, A. (2006). Professional writers and revision. Revision: History, theory, and practice, 117-141.
Janssen, D. M. L., & Schilperoord, J. (1992). Policy writing research and functional text quality. In H. Pander Maat, & M. Steehouder (Eds.), Studies in functional text quality. Utrecht Studies in Language and Communication. Amsterdam: Rodophi.
Janssen, D. M. L., & van der Mast, N. P. (2001). Collaborative writing for the government. In D. M. I. Janssen, & R. Neutelings (Eds.), Reading and writing public documents (pp. 169-208). Amsterdam: John Benjamins.
Janssen, D.M.L. (1991). Schrijven aan beleidsnota’s (Writing policy papers). PhD thesis. Groningen: Wolters Noordhoff.
Janssen, D.M.L., & Neutelings, R. (2001). Reading and writing public documents. Amsterdam: John Benjamins.
Kellogg, R. T. (2008). Training writing skills: A cognitive developmental perspective. Journal of Writing Research, 1(1), 1–26. https://doi.org/10.17239/jowr-2008.01.01.1
Kleijn S. (2018). Clozing in on readability: How linguistic features affect and predict text comprehension and on-line processing. PhD-disseration. Utrecht: LOT.
Lay, M. and Karin, W.M. (1991). Collaborative Writing in Industry: Investigations in Theory and Practice. Amityville NJ: Baywood.
Lee, M., Gero, K. I., Chung, J. J. Y., et al. (2024). A design space for intelligent and interactive writing assistants. CHI ’24. https://dl.acm.org/doi/10.1145/3613904.3642697
Lee, M., Liang, P., & Yang, Q. (2022). CoAuthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities. CHI ’22. https://arxiv.org/abs/2201.06796
Leijten, M. (2007). Writing and Speech Recognition: Observing Error Correction Strategies of Profes-sional Writers. Utrecht: LOT.
Leijten, M., & Van Waes, L. (2013). Keystroke logging in writing research: Using Inputlog to analyze and visualize writing processes. Written Communication, 30(3), 358-392. https://doi.org/10.1177/0741088313491692.
Leijten, M., Waes, L. V., Schriver, K., & Hayes, J. R. (2014). Writing in the workplace: Constructing documents using multiple digital sources. Journal of Writing Research, 5(3), 285–337. DOI: 10.17239/jowr-2014.05.03.3
Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS. https://arxiv.org/abs/2005.11401
Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10.1145/3313831.3376727
Longpre, S., Hou, L., Vu, T., et al. (2023). The Flan collection: Designing data and methods for effective instruction tuning. ICML. https://proceedings.mlr.press/v202/longpre23a.html
Lundstrom, K. & Baker, W. (2009). To give is better than to receive: The benefits of peer review to the reviewer's own writing, Journal of Second Language Writing 18 (1), 30-43. https://doi. org/10.1016/j.jslw.2008.06.002.
Mistral AI. (2023a, September 27). Announcing Mistral 7B. https://mistral.ai/news/announcing-mistral-7b
Mistral AI. (2023b, December 11). Mixtral of experts (Mixtral 8×7B). https://mistral.ai/news/mixtral-of-experts
Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381, 187 - 192. https://doi.org/10.1126/science.adh2586.
OpenAI. (2023a, March 1). Introducing ChatGPT and Whisper APIs (forum announcement). https://community.openai.com/t/introducing-chatgpt-and-whisper-apis/80485
OpenAI. (2023b, May 18). Introducing the ChatGPT app for iOS. https://openai.com/index/introducing-the-chatgpt-app-for-ios/
Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS. https://arxiv.org/abs/2203.02155
Pander Maat, H. and M. Steehouder (Eds.), Studies in functional text quality. Utrecht Studies in Language and Communication. Amsterdam: Rodophi.
Pander Maat, H., & Dekker, N. (2016). Tekstgenres analyseren op lexicale complexiteit met T Scan. Tijdschrift voor Taalbeheersing, 38(3), 263-304. https://doi.org/10.5117/TVT2016.3.PAND
Ranade, N., Saravia, M., & Johri, A. (2025). Using rhetorical strategies to design prompts: A human‑in‑the‑loop approach to make AI useful. AI & Society, 40, 711–732. https://doi.org/10.1007/s00146-024-01905-3
Reuters. (2023, February 1). ChatGPT sets record for fastest-growing user base. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/
Reynolds, F.J. et al. (1995). Professional writing in context : lessons from teaching and consulting in worlds of work. Hillsdale NJ: Erlbaum.
Romeo, G., & Conti, D. (2025). Exploring automation bias in human–AI collaboration: A review and implications for explainable AI. AI & Society https://doi.org/10.1007/s00146-025-02422-7
Schriver, K. (2012). What do technical communicators need to know about information design? J. Johnson-Eilola & S. Selber, Eds. Solving problems in technical communication, pp. 386–427. University of Chicago Press.
Schriver, K. A. (1997). Dynamics in document design: Creating text for readers. Wiley.
Selzer, J. (1983). The Composing Processes of an Engineer. College Composition and Communication, 34(2), 178–187. https://doi.org/10.2307/357405
Selzer, J. (1984). Exploring Options in Composing. College Composition and Communication, 35(3), 276–284. https://doi.org/10.2307/357456
Shibani, Antonette (2023), Analytic Techniques for Automated Analysis of Writing. Kruse, O. et al. (eds), Digital Writing Technologies in Higher Education, Springer.
Sommers, N. (1980). Revision strategies of student writers and experienced adult writers. College Composition and Communication, 31(4), 378–388. https://doi.org/10.2307/356588.
Vaccaro, M., Almaatouq, A. & Malone, T. (2024) When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human. Published online.
Van der Mast, N.P. and Janssen, D.M.L. (2001). Strategic revision of public documents: how policy writers revise documents in order to reach consensus. In D.M.L. Janssen, & R. Neutelings (Eds.), Reading and writing public documents (pp. 209-230). Amsterdam: John Benjamins.
Van Riel, C. B. and Fombrun, C. J. (2007). Essentials of Corporate Communication: Implementing Practices for Effective Reputation Management. (n.p.): Taylor & Francis.
Van Steendam, E. (2016). Editorial: Forms of collaboration in writing. Journal of Writing Research, 8(2), 183–204. https://doi.org/10.17239/jowr-2016.08.02.01
Van Waes, L. (1991). Rapporteren met computer en pen & papier: Verschillen in het pauze- en revisiegedrag bij ervaren schrijvers. Toegepaste taalwetenschap in artikelen, 40(1), 176-190. https://doi.org/10.1075/ttwia.40.17wae
Wiggers, K. (2023, July 11). Anthropic releases Claude 2, its second-gen AI chatbot. TechCrunch. https://techcrunch.com/2023/07/11/anthropic-releases-claude-2-the-second-generation-of-its-ai-chatbot/ Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus, 15(2). 0.7759/cureus.35179
Willey, I., & Tanimoto, K. (2012). “Convenience Editing” in action: Comparing English teachers’ and medical professionals’ revisions of a medical abstract. English for Specific Purposes, 31(4), 249-260. https://doi.org/10.1016/j.esp.2012.04.001
Yao, S., Zhao, J., Yu, D., et al. (2022). ReAct: Synergizing reasoning and acting in language models. ICLR. https://arxiv.org/abs/2210.03629
Yuan, A., Coenen, A., Reif, E., Ippolito, D., & Davis, L. (2022). Wordcraft: Story writing with large language models. IUI ’22. https://dl.acm.org/doi/10.1145/3490099.3511105
Zhai, X., Chu, X., Wang, M., Tsai, C. C., Liang, J. C., & Spector, J. M. (2024). A systematic review of Stimulated Recall (SR) in educational research from 2012 to 2022. Humanities and Social Sciences Communications, 11(1), 1-14. https://doi.org/10.1057/s41599-024-02987-6
Zhang, Z., Gao, J., Dhaliwal, R. S., & Li, T. J.-J. (2023). VISAR: A human-AI argumentative writing assistant with visual programming and rapid draft prototyping. UIST ’23. https://toby.li/files/uist23-zhang-visar.pdf
Published
Issue
Section
License
Copyright (c) 2026 Daniël Janssen, Henri Raven, Lisanne Van Weelden, Yohannes Den Hertog
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported License.
