Using Generative AI in the teaching of academic writing

Project introduction and background information

In its Strategy 2030 document (Executive Board TU/e, 2018: 31), TU/e stresses the importance of digitization, to allow learning at any place, at any time and to support adaptive and personalized instruction and feedback. Intelligent systems could be used to fulfil this need, by providing personalized, automated and timely feedback.

The power, sophistication, and societal prominence of generative AI systems such as ChatGPT has only grown. As is evidenced by discussions in the academic, professional, and popular media, these systems pose an unprecedented challenge to established structures and institutions in many different domains, including higher education. The particular problem facing higher education is that generative AI undermines the principle of constructive alignment between learning objectives and assessment methods, impacting students and teachers alike.

For students, the advent of generative AI is poised to change the skills they require to succeed after leaving university and entering the workforce. Given their general-purpose nature, systems such as ChatGPT are likely to be used in many different domains, from software engineering to journalism and marketing. Graduating students entering these domains must possess relevant skills, which are likely to differ from the ones higher education is traditionally designed to promote. In particular, when performing writing tasks such as conducting literature review or compiling reports, students will no longer need to do so on their own, but will instead be expected to collaborate with AI technology to enhance the speed and quality of their writing. Among others, this will require mastery of skills such as prompt-engineering and machine summarizing, as well as a critical engagement with AI-generated content. Higher education generally, and TU/e specifically, should equip students with these skills, and will therefore need to identify and articulate “future-oriented” learning objectives to be achieved in writing-based university courses.

For teachers, the increased power and availability of generative AI challenges their ability to assess student learning. Because tools such as ChatGPT can be used to produce deliverables such as essays, reports, diagrams, and code, the provenance of these deliverables can no longer be traced to individual students as opposed to sophisticated machines. As a consequence, it is unclear that students are actually satisfying the stated learning objectives, and teachers will require “AI-proof” assessment methods that allow them to measure the extent to which students have mastered the ability to write clearly and effectively, either in collaboration with relevant AI systems, or on their own.

If TU/e is to stay at the forefront of societally relevant engineering education, it will have to promote the implementation of new education methods to promote effective writing. Crucially, this action will have to occur sooner rather than later: as generative AI systems become even more powerful, available, and easy to use, it is in the university’s interest to stay ahead of developments and be proactive rather than reactive.

Objective and expected outcomes

The main objectives and expected outcomes of the project are manifold, consisting of the following working packages:

WP1: Mapping the literature on future-oriented and GenAI compatible higher education

This work package will study the impact of Generative AI (GenAI) on aligning learning objectives, activities, and assessments in higher education. The main focus will be on identifying future-oriented learning goals and GenAI-compatible pedagogical and assessment methods based on literature searches. 

Expected outcomes:

• Report 1a: Report summarizing key insights about future-oriented learning objectives, and lists of learning activities and assessment methods useful to accommodate GenAI in education (Dec 2024). Closely related, research insights related to this WP will be presented at the 4TUCEE End of Year event (November 2024).
• Report 1b: An updated version of report 1a incorporating any relevant updates in light of AI technology advancements (June 2025).

W2: Designing a framework for learning assessment through AI interaction analysis.

This work package will study interactions between students and GenAI chatbots like ChatGPT and their relation with  learning outcomes. Collaborating with course teachers, education experts, and learning analytics specialists, we will develop a framework to analyze these interactions (e.g., logs) for multiple courses at TU/e that involve writing assignments. The focus will be on identifying patterns that indicate learning progress, GenAI literacy, critical thinking, and knowledge construction.

This analysis framework includes:

• Collecting student-GenAI interaction logs (user prompts and chatbot responses).
• Applying qualitative coding schemes for dialogue content to assess patterns of interaction, including critical thinking evidence, question types, and iteration patterns.
• Developing quantitative metrics for interaction analysis (e.g., rubric scores, input frequency). Conducting rubric-based assessments comparing traditional evaluations with GenAI-interaction assessments.
• Automating chat log analysis using natural language processing to identify learning patterns.
• Surveying student characteristics and perceptions through open-ended questions about their use of GenAI tools in learning.
• Collaborating with students to critically assess and optimize evaluation criteria for student-AI interactions (in alignment with TU/e BOOST objectives)

This working package will take place in AY 2024-2025 across various Bachelor’s and Master’s courses from different departments.  

Expected outcomes:

• Report 2a: A description of the methodology and framework for assessing learning outcomes through AI interaction log analysis as well as the outcomes from a set of pilot courses (April 2025). Closely related, research insights related to this WP will be presented at the 4TUCEE End of Year event (November 2024).
• Report 2b: Tutorial for teachers on practical implementations of the assessment framework.(April 2025)

WP3: Develop pedagogical activities powered by Generative AI

Based on the insights obtained from the literature review in WP1, as well as the insights gained from the development of the AI compatible assessment method from WP2, we aim to develop pedagogical activities to improve constructive alignment by teaching students how to effectively use AI to improve their performance on student-GenAI interaction assessments. The design of these GenAI tutors will be informed by insights gained in the previous work packages, and tailored to the learning objectives of the course and the specificities of the course assignments. These tools will be tested in pilot studies, in at least two courses with different types of assignments (e.g., argumentative essay vs. statistical programming and data analysis). For example:

• In a TU/e statistics course like “Advanced research methods and research ethics” where the objective for students is to learn how to think about their analytical approach to a given problem, the GenAI tutor can be designed to steer the student towards the assignment solution / desirable outcomes through questions posed to the student (scaffolding), as opposed to directly providing the solution. Simultaneously, a second GenAI tutor available to the same student could assist with another task required by the assignment, such as statistical programming, by providing explanations of the code or helping with code debugging. The behavior of the AI chatbot can be configured through prompt engineering techniques such as “Chain-of-Thought” combined with “Flipped interaction” (White et al., 2023).
• For writing-oriented courses such as “Data Science Ethics”, a GenAI tutor tool can be designed to assist with academic writing through scaffolding, such as guiding essay outlines, prompting critical thinking in the student, and offering instant feedback on their writing in a way that is aligned with the course learning objectives (embedded in the tutor’s underlying model or knowledge context). Likewise, the behavior of the AI chatbot is configured through prompt engineering techniques such as “Chain-of-Thought” combined with “Flipped interaction” (White et al., 2023).

By designing these GenAI tutor prototypes and incorporating them as tools in a course we will investigate:

• Whether using GenAI tutor tools lead to higher grades in student-GenAI interaction assessments, thereby providing insight into quality of constructive alignment in the course
• The impact of using these tools on self-perceptions of skill mastery (e.g., self-efficacy), thereby providing insights on alignment with students’ psychological
•  Students experiences using these tools (using open-ended questions in a post-assignment survey)

Expected outcomes:

• Report 3a: Tutorial on how to design a GenAI tutor and tailor it to a specific use case (March 2025)
• Report 3b: Report describing the results of the GenAI tutor studies (Nov 2025).

WP4: Workshops on GenAI tools for educational activities

AI literacy is a crucial step towards the responsible use of GenAI technology in educational practices (Kasneci et al., 2023; Redecker, 2017). In this work package, we will develop a workshop aimed at either or both students (ranging from Bachelor to PhD candidates) and staff where we teach how to properly design and implement a GenAI based assistants using the most relevant and/or accessible AI chatbot tool(s) at the time they take place. The workshop will focus on teaching the essential steps to configure the behavior of AI chatbot assistants through existing techniques (e.g. prompt engineering, fine-tuning options), and provide examples on how to tailor it to more specific use cases (e.g., drafting teaching materials, academic writing assistance, literature summarization, data analysis assistance). A second type of workshop will place a higher weight on practical tips to employ GenAI tools in academic writing activities. A third type of workshop, aimed for the ALT community (Academy for Learning and Teaching, TU/e) will focus on topics of AI literacy. Content (tentative) may encapsulate and chain the following topics: evidence-based utility of GenAI tools, building GenAI chatbots to augment teaching practice, ethical use of GenAI, plus any practical recommendations derived from insights from pilot studies. Dates and times of these workshops will be arranged with project members and project coordinators from either 4TUCEE (SEFI) and BOOST (ALT).

Expected outcomes:

• Ongoing workshops at TU/e event with focus on GenAI chatbot design for educational activities and where possible workshop at a scientific event (e.g., SEFI) (Dec 2024-Dec 2025).

References

• Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
• Redecker, C. (2017). European framework for the digital competence of educators: DigCompEdu. In Y. Punie (Ed.), Technical report. Joint Research Centre (Seville site). https://data.europa.eu/doi/10.2760/159770
• White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (arXiv:2302.11382). arXiv. https://doi.org/10.48550/arXiv.2302.11382

Results and learnings

WP1: Mapping the literature on future-oriented and GenAI compatible higher education

Report 1a

Context

Generative AI (GenAI) technologies like ChatGPT and similar chatbots are transforming higher education. As these tools become more sophisticated, they pose significant challenges and opportunities for teaching, learning, and assessment. In this summary, we provide an overview of the key insights from our more extensive report (available below), highlighting how our ongoing review of the state-of-the-art knowledge on the challenges and opportunities of GenAI in Higher Education can be translated into actionable information relevant to teachers and students, highlighting strategic directions, future research, and practical recommendations. The emphasis of this summary is primarily on practical recommendations, given the urgency to respond to the ongoing transformative impact of GenAI.

Key challenges identified

• Disruption of traditional learning and assessment: GenAI’s humanlike content generation challenges conventional teaching methods, assessment integrity, and the unique value of human instruction (Kolade et al., 2024; Rathi et al., 2024).
• Need for skill reorientation: Both educators and students must shift focus toward skills that AI cannot easily replicate—critical thinking, creativity, ethical reasoning, adaptability, and AI literacy (Bower et al., 2024; Chauncey & McKenna, 2024; Kolade et al., 2024).
• Assessment uncertainty: With AI’s capacity to generate content, verifying authorship and evaluating genuine understanding become more complex, necessitating a redesign of assessment methodologies (Fleckenstein et al., 2024; Jakesch et al., 2019).

Strategic responses and recommendations

Redefining learning objectives

• Emphasize higher-order skills: critical thinking, problem-solving, creative ideation, ethical reasoning, and adaptability (Kasneci et al., 2023; Zhai, 2022).
• Integrate AI literacy: Ensure students and teachers understand AI’s capabilities, limitations, biases, and ethical considerations (Bower et al., 2024; Chiu, 2024; Kolade et al., 2024).

Transforming assessment practices

• Move from product-focused to process-oriented assessment: Evaluate reasoning processes, metacognitive skills, and real-world application (Cheng et al., 2024; Kolade et al., 2024).
• Adopt diverse, authentic evaluation methods: Use live presentations, peer assessments, project-based assignments, and frequent low-stakes assessments to mitigate AI’s advantages in generating generic responses (Kolade et al., 2024; Xia et al., 2024).
• Leverage prompt analytics: Analyze student interactions with AI to gain insights into learning processes and provide personalized feedback (Cheng et al., 2024; Kim et al., 2024).

Faculty development and policy updates

• Invest in professional development to equip educators with skills for integrating AI responsibly into pedagogy and assessment (Chan & Tsi, 2024; Lim et al., 2023).
• Update assessment policies to establish clear guidelines on AI use, maintaining academic integrity while embracing AI-enabled learning enhancements (Mollick & Mollick, 2022; Xia et al., 2024).

Ongoing research and pilot studies

• Supporting and monitoring pilot projects (e.g., at TU/e) that assess the impact of GenAI on learning outcomes, teacher effectiveness, and student motivation.
• Continuous multidisciplinary research to refine teaching and assessment strategies, strioving for an alignment with evolving (Gen)AI capabilities and future-oriented educational goals (e.g., Deng & Joshi, 2024; Mollick & Mollick, 2024; Rowland, 2023).

Value for teachers and students

For teachers:

• Empowerment through professional development and clear guidelines on AI integration, enabling them to design engaging, authentic assessments that emphasize unique human skills.
• Improved assessment tools and strategies that provide more accurate measures of student understanding and skill acquisition.

For students:

• Development of relevant, future-oriented skills that enhance employability and adaptability in an AI-driven landscape (Chiu, 2024; Zhai, 2022).
• Learning experiences that promote creativity, critical thinking, and ethical reasoning—areas where human judgment remains indispensable (Bower et al., 2024).

Conclusion and next steps

The integration of GenAI in higher education calls for a strategic, research-backed approach to curriculum design, assessment methods, and faculty development. By focusing on uniquely human skills and transforming assessment practices, institutions can harness AI’s potential while preserving academic integrity and enhancing learning outcomes. Stakeholders are encouraged to support ongoing research, pilot projects, and policy updates that inform best practices. This proactive approach ensures that both teachers and students are prepared for and can thrive in an AI-integrated educational environment.

Next Steps

The project will continue through three parallel threads:

1. Finalizing the systematic scoping review
2. Designing and implementing pilot studies on GenAI's impact
3. Compiling and analyzing data from ongoing pilot studies in TU/e courses

Recommendations for educators:

• Provide faculty development on AI capabilities (Kasneci et al., 2023)
• Update assessment policies (Xia et al., 2024)
• Focus on higher-order thinking skills (Chiu, 2024)
• Conduct continuous research on AI in education (Lim et al., 2023)

References

• Bower, M., Torrington, J., Lai, J. W. M., Petocz, P., & Alfano, M. (2024). How should we change teaching and assessment in response to increasingly powerful generative Artificial Intelligence? Outcomes of the ChatGPT teacher survey. Education and Information Technologies, 29(12), 15403–15439. https://doi.org/10.1007/s10639-023-12405-0
• Chan, C. K. Y., & Tsi, L. H. Y. (2024). Will generative AI replace teachers in higher education? A study of teacher and student perceptions. Studies in Educational Evaluation, 83, 101395. https://doi.org/10.1016/j.stueduc.2024.101395
• Chauncey, S. A., & McKenna, H. P. (2024). Exploring the Potential of Cognitive Flexibility and Elaboration in Support of Curiosity, Interest, and Engagement in Designing AI-Rich Learning Spaces, Extensible to Urban Environments. In N. A. Streitz & S. Konomi (Eds.), Distributed, Ambient and Pervasive Interactions (pp. 209–230). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-60012-8_13
• Cheng, Y., Lyons, K., Chen, G., Gašević, D., & Swiecki, Z. (2024). Evidence-centered assessment for writing with generative AI. Proceedings of the 14th Learning Analytics and Knowledge Conference, 178–188. https://doi.org/10.1145/3636555.3636866
• Chiu, T. K. F. (2024). Future research recommendations for transforming higher education with generative AI. Computers and Education: Artificial Intelligence, 6, 100197. https://doi.org/10.1016/j.caeai.2023.100197
• Deng, X., & Joshi, K. D. (2024). Promoting ethical use of generative ai in education. SIGMIS Database, 55(3), 6–11. https://doi.org/10.1145/3685235.3685237
• Fleckenstein, J., Meyer, J., Jansen, T., Keller, S. D., Köller, O., & Möller, J. (2024). Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays. Computers and Education: Artificial Intelligence, 6, 100209. https://doi.org/10.1016/j.caeai.2024.100209
• Jakesch, M., French, M., Ma, X., Hancock, J. T., & Naaman, M. (2019). AI-Mediated Communication: How the Perception that Profile Text was Written by AI Affects Trustworthiness. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3290605.3300469
• Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
• Kim, M., Kim, S., Lee, S., Yoon, Y., Myung, J., Yoo, H., Lim, H., Han, J., Kim, Y., Ahn, S.-Y., Kim, J., Oh, A., Hong, H., & Lee, T. Y. (2024). Designing Prompt Analytics Dashboards to Analyze Student-ChatGPT Interactions in EFL Writing (arXiv:2405.19691). arXiv. https://doi.org/10.48550/arXiv.2405.19691
• Kolade, O., Owoseni, A., & Egbetokun, A. (2024). Is AI changing learning and assessment as we know it? Evidence from a ChatGPT experiment and a conceptual framework. Heliyon, 10(4), e25953. https://doi.org/10.1016/j.heliyon.2024.e25953
• Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2), 100790. https://doi.org/10.1016/j.ijme.2023.100790
• Mollick, E. R., & Mollick, L. (2022). New Modes of Learning Enabled by AI Chatbots: Three Methods and Assignments (SSRN Scholarly Paper 4300783). https://doi.org/10.2139/ssrn.4300783
• Mollick, E. R., & Mollick, L. (2024). Instructors as innovators: A future-focused approach to new AI learning opportunities, with prompts (SSRN Scholarly Paper 4802463). https://doi.org/10.2139/ssrn.4802463
• Rathi, I., Taylor, S., Bergen, B. K., & Jones, C. R. (2024). GPT-4 is judged more human than humans in displaced and inverted Turing tests (arXiv:2407.08853). arXiv. https://doi.org/10.48550/arXiv.2407.08853
• Rowland, D. R. (2023). Two frameworks to guide discussions around levels of acceptable use of generative AI in student academic research and writing. Journal of Academic Language and Learning, 17(1), Article 1.
• Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21(1), 40. https://doi.org/10.1186/s41239-024-00468-z
• Zhai, X. (2022). ChatGPT user experience: Implications for education. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4312418

Recommendations

WP1

Recommendations based on the literature review from WP1.

Higher education institutions should focus on redesigning their curricula and assessment methods to emphasize skills that AI cannot easily replicate. This means shifting from content-focused instruction to developing higher-order thinking skills through project-based learning and collaborative activities. Educators should integrate AI tools like ChatGPT into their teaching while maintaining clear guidelines for ethical use, helping students understand both the capabilities and limitations of these technologies.

Assessment strategies should move away from traditional essays and exams toward performance-based evaluation methods that demonstrate authentic learning and application of knowledge. This includes implementing more real-time assessments like presentations, group projects, and case studies that require students to demonstrate critical thinking, problem-solving, and creativity while applying their knowledge in practical contexts.

WORK IN PROGRESS...

• WP2: Assessing student interactions with GenAI in teh context of academic writing 
• WP3: Pilot studies using GenAI based tutoring applications in the classroom
• WP4: Workshops on GenAI tutor building, AI literacy, Co-working with GenAI

Practical outcomes

WP1

The insights gained from the systematic scoping review of the literature aiming to gather information on the state-of-the-art perspectives and interventions regarding  future-oriented learning objectives and AI-compatible assessment methods, informed the design of pilot studies at TU/e assessing the impact of using Generative AI chatbots on a range of outcomes ranging from student learning to teaching activities.

WORK IN PROGRESS...

• WP2: Assessing student interactions with GenAI in teh context of academic writing 
• WP3: Pilot studies using GenAI based tutoring applications in the classroom
• WP4: Workshops on GenAI tutor building, AI literacy, Co-working with GenAI