Using Generative AI in the teaching of academic writing

Project introduction and background information

In its Strategy 2030 document (Executive Board TU/e, 2018: 31), TU/e stresses the importance of digitization, to allow learning at any place, at any time and to support adaptive and personalized instruction and feedback. Intelligent systems could be used to fulfil this need, by providing personalized, automated and timely feedback.

The power, sophistication, and societal prominence of generative AI systems such as ChatGPT has only grown. As is evidenced by discussions in the academic, professional, and popular media, these systems pose an unprecedented challenge to established structures and institutions in many different domains, including higher education. The particular problem facing higher education is that generative AI undermines the principle of constructive alignment between learning objectives and assessment methods, impacting students and teachers alike.

For students, the advent of generative AI is poised to change the skills they require to succeed after leaving university and entering the workforce. Given their general-purpose nature, systems such as ChatGPT are likely to be used in many different domains, from software engineering to journalism and marketing. Graduating students entering these domains must possess relevant skills, which are likely to differ from the ones higher education is traditionally designed to promote. In particular, when performing writing tasks such as conducting literature review or compiling reports, students will no longer need to do so on their own, but will instead be expected to collaborate with AI technology to enhance the speed and quality of their writing. Among others, this will require mastery of skills such as prompt-engineering and machine summarizing, as well as a critical engagement with AI-generated content. Higher education generally, and TU/e specifically, should equip students with these skills, and will therefore need to identify and articulate “future-oriented” learning objectives to be achieved in writing-based university courses.

For teachers, the increased power and availability of generative AI challenges their ability to assess student learning. Because tools such as ChatGPT can be used to produce deliverables such as essays, reports, diagrams, and code, the provenance of these deliverables can no longer be traced to individual students as opposed to sophisticated machines. As a consequence, it is unclear that students are actually satisfying the stated learning objectives, and teachers will require “AI-proof” assessment methods that allow them to measure the extent to which students have mastered the ability to write clearly and effectively, either in collaboration with relevant AI systems, or on their own.

If TU/e is to stay at the forefront of societally relevant engineering education, it will have to promote the implementation of new education methods to promote effective writing. Crucially, this action will have to occur sooner rather than later: as generative AI systems become even more powerful, available, and easy to use, it is in the university’s interest to stay ahead of developments and be proactive rather than reactive.

Objective and expected outcomes

The main objectives and expected outcomes of the project are manifold.

WP1: Mapping the literature on future-oriented and GenAI compatible higher education

This work package will study the impact of Generative AI (GenAI) on aligning learning objectives, activities, and assessments in higher education. The main focus will be on identifying future-oriented learning goals and GenAI-compatible pedagogical and assessment methods based on literature searches. 

Expected outcomes:

• Report 1a: Report summarizing key insights about future-oriented learning objectives, and lists of learning activities and assessment methods useful to accommodate GenAI in education (Dec 2024). Closely related, research insights related to this WP will be presented at the 4TUCEE End of Year event (November 2024).
• Report 1b: An updated version of report 1a incorporating any relevant updates in light of AI technology advancements (June 2025).

W2: Designing a framework for learning assessment through AI interaction analysis.

This work package will study interactions between students and GenAI chatbots like ChatGPT and their relation with  learning outcomes. Collaborating with course teachers, education experts, and learning analytics specialists, we will develop a framework to analyze these interactions (e.g., logs) for multiple courses at TU/e that involve writing assignments. The focus will be on identifying patterns that indicate learning progress, GenAI literacy, critical thinking, and knowledge construction.

This analysis framework includes:

• Collecting student-GenAI interaction logs (user prompts and chatbot responses).
• Applying qualitative coding schemes for dialogue content to assess patterns of interaction, including critical thinking evidence, question types, and iteration patterns.
• Developing quantitative metrics for interaction analysis (e.g., rubric scores, input frequency). Conducting rubric-based assessments comparing traditional evaluations with GenAI-interaction assessments.
• Automating chat log analysis using natural language processing to identify learning patterns.
• Surveying student characteristics and perceptions through open-ended questions about their use of GenAI tools in learning.
• Collaborating with students to critically assess and optimize evaluation criteria for student-AI interactions (in alignment with TU/e BOOST objectives)

This working package will take place in AY 2024-2025 across various Bachelor’s and Master’s courses from different departments.  

Expected outcomes:

• Report 2a: A description of the methodology and framework for assessing learning outcomes through AI interaction log analysis as well as the outcomes from a set of pilot courses (April 2025). Closely related, research insights related to this WP will be presented at the 4TUCEE End of Year event (November 2024).
• Report 2b: Tutorial for teachers on practical implementations of the assessment framework.(April 2025)

WP3: Develop pedagogical activities powered by Generative AI

Based on the insights obtained from the literature review in WP1, as well as the insights gained from the development of the AI compatible assessment method from WP2, we aim to develop pedagogical activities to improve constructive alignment by teaching students how to effectively use AI to improve their performance on student-GenAI interaction assessments. The design of these GenAI tutors will be informed by insights gained in the previous work packages, and tailored to the learning objectives of the course and the specificities of the course assignments. These tools will be tested in pilot studies, in at least two courses with different types of assignments (e.g., argumentative essay vs. statistical programming and data analysis). For example:

• In a TU/e statistics course like “Advanced research methods and research ethics” where the objective for students is to learn how to think about their analytical approach to a given problem, the GenAI tutor can be designed to steer the student towards the assignment solution / desirable outcomes through questions posed to the student (scaffolding), as opposed to directly providing the solution. Simultaneously, a second GenAI tutor available to the same student could assist with another task required by the assignment, such as statistical programming, by providing explanations of the code or helping with code debugging. The behavior of the AI chatbot can be configured through prompt engineering techniques such as “Chain-of-Thought” combined with “Flipped interaction” (White et al., 2023).
• For writing-oriented courses such as “Data Science Ethics”, a GenAI tutor tool can be designed to assist with academic writing through scaffolding, such as guiding essay outlines, prompting critical thinking in the student, and offering instant feedback on their writing in a way that is aligned with the course learning objectives (embedded in the tutor’s underlying model or knowledge context). Likewise, the behavior of the AI chatbot is configured through prompt engineering techniques such as “Chain-of-Thought” combined with “Flipped interaction” (White et al., 2023).

By designing these GenAI tutor prototypes and incorporating them as tools in a course we will investigate:

• Whether using GenAI tutor tools lead to higher grades in student-GenAI interaction assessments, thereby providing insight into quality of constructive alignment in the course
• The impact of using these tools on self-perceptions of skill mastery (e.g., self-efficacy), thereby providing insights on alignment with students’ psychological
•  Students experiences using these tools (using open-ended questions in a post-assignment survey)

Expected outcomes:

• Report 3a: Tutorial on how to design a GenAI tutor and tailor it to a specific use case (March 2025)
• Report 3b: Report describing the results of the GenAI tutor studies (Nov 2025).

WP4: Workshops on GenAI tools for educational activities

AI literacy is a crucial step towards the responsible use of GenAI technology in educational practices (Kasneci et al., 2023; Redecker, 2017). In this work package, we will develop a workshop aimed at either or both students (ranging from Bachelor to PhD candidates) and staff where we teach how to properly design and implement a GenAI based assistants using the most relevant and/or accessible AI chatbot tool(s) at the time they take place. The workshop will focus on teaching the essential steps to configure the behavior of AI chatbot assistants through existing techniques (e.g. prompt engineering, fine-tuning options), and provide examples on how to tailor it to more specific use cases (e.g., drafting teaching materials, academic writing assistance, literature summarization, data analysis assistance). A second type of workshop will place a higher weight on practical tips to employ GenAI tools in academic writing activities. A third type of workshop, aimed for the ALT community (Academy for Learning and Teaching, TU/e) will focus on topics of AI literacy. Content (tentative) may encapsulate and chain the following topics: evidence-based utility of GenAI tools, building GenAI chatbots to augment teaching practice, ethical use of GenAI, plus any practical recommendations derived from insights from pilot studies. Dates and times of these workshops will be arranged with project members and project coordinators from either 4TUCEE (SEFI) and BOOST (ALT).

Expected outcomes:

• Ongoing workshops at TU/e event with focus on GenAI chatbot design for educational activities and where possible workshop at a scientific event (e.g., SEFI) (Dec 2024-Dec 2025).

References

• Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
• Redecker, C. (2017). European framework for the digital competence of educators: DigCompEdu. In Y. Punie (Ed.), Technical report. Joint Research Centre (Seville site). https://data.europa.eu/doi/10.2760/159770
• White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (arXiv:2302.11382). arXiv. https://doi.org/10.48550/arXiv.2302.11382

Results and learnings

WP1: Mapping the literature on future-oriented and GenAI compatible higher education

Report 1a

Introduction

The rapid advancement and increasing accessibility of generative AI technologies, such as ChatGPT, are transforming the landscape of higher education. As these tools become more sophisticated and widely adopted, they present both opportunities and challenges for educators and students alike. The ability of generative AI to produce humanlike content, solve complex problems, and engage in natural language interactions is disrupting traditional models of teaching, learning, and assessment. This reality calls for a critical re-examination and redefinition of educational objectives, focusing on the skills and competencies students will need to thrive in an AI-driven world. Higher education institutions must adapt their curricula, pedagogical approaches, and assessment strategies to align with this new reality while maintaining the integrity and value of the educational experience.

Future-Oriented Learning Objectives and Skills

In light of the pervasive influence of generative AI, higher education must prioritize the development of a range of cognitive, social, emotional, and technical competencies that will enable students to navigate and succeed in an AI-infused society. Recent literature emphasizes the importance of fostering skills such as critical thinking, problem-solving, creativity, and collaboration (Kasneci et al., 2023; Zhai, 2022). These higher-order thinking skills are essential for students to effectively leverage AI tools while maintaining the ability to think independently and generate original ideas. Additionally, digital literacy and AI literacy have emerged as crucial competencies, as students must learn to use AI tools ethically and responsibly, understanding their capabilities, limitations, and potential biases (Chiu, 2024; Zhai, 2022). Emotional intelligence and adaptability are also key skills, as students must be able to navigate the complexities and uncertainties of an AI-driven world with resilience and empathy. By focusing on these future-oriented learning objectives and skills, higher education can empower students to thrive in a rapidly evolving technological landscape.

AI-Compatible Assessment Methods

Traditional assessment methods, such as essays and exams, are increasingly vulnerable to the disruptive effects of generative AI, which can easily produce content that mimics human-generated responses. To maintain the integrity and effectiveness of assessments in an AI-infused educational environment, educators must adopt innovative strategies that minimize the impact of AI-generated content and promote authentic learning. Formative assessments that focus on the process of learning, such as reflective journals, peer feedback, and self-assessments, can provide valuable insights into students' growth and skill acquisition (Zhai, 2022). Performance-based assessments, such as project work, presentations, and simulations, allow students to demonstrate their understanding and application of knowledge in realistic contexts (Kasneci et al., 2023). Additionally, the use of AI-assisted assessment tools, such as automated essay scoring and adaptive testing, can enhance the efficiency and personalization of assessments while maintaining academic rigor (Gašević et al., 2023). By embracing these AI-compatible assessment methods, higher education can effectively measure student learning and ensure the validity of educational outcomes.

Learning Activities with AI

To prepare students for an AI-driven future, higher education must design learning activities and experiences that foster the development of key skills and competencies while leveraging the potential of generative AI tools. Project-based learning and problem-based learning are effective approaches that engage students in authentic, collaborative, and interdisciplinary tasks, promoting critical thinking, creativity, and teamwork (Essel et al., 2024; Rudolph et al., 2023). Integrating generative AI tools into these activities can enhance student engagement and provide opportunities for students to develop AI literacy and ethical practices. For example, students can use ChatGPT to brainstorm ideas, generate content, or receive feedback on their work, while critically evaluating the output and making informed decisions about its use (Rudolph et al., 2023). Additionally, AI-supported personalized learning experiences, such as adaptive learning systems and intelligent tutoring systems, can tailor instruction to individual students' needs and preferences, promoting mastery and engagement (Kasneci et al., 2023). By designing learning activities that seamlessly integrate AI tools and foster the development of future-oriented skills, higher education can prepare students to thrive in an AI-infused world.

Discussion

The rise of generative AI in higher education necessitates a holistic approach to redefining learning objectives, assessment methods, and learning activities. For instance, a university course on data science could incorporate learning objectives related to AI literacy and critical thinking, use formative assessments such as reflective journals to gauge students' understanding of ethical AI practices, and implement project-based learning activities where students collaborate to develop and evaluate AI models. By ensuring a strong connection between these elements, higher education can create a cohesive and transformative educational experience that empowers students to navigate the challenges and opportunities of an AI-driven future.

The future-oriented learning objectives and skills identified serve as the foundation for designing effective assessment methods and learning activities. For example, to foster creativity and problem-solving skills, educators could design performance-based assessments such as simulations or case studies that require students to apply their knowledge in realistic scenarios. These assessments align with the learning objectives by measuring students' ability to think critically and generate innovative solutions.

Learning activities that integrate generative AI tools provide opportunities for students to develop and apply the key skills and competencies needed to thrive in an AI-infused world. In a language learning course, students could use ChatGPT to practice conversation skills and receive immediate feedback on their language use. This activity aligns with the learning objective of developing digital literacy and AI literacy while providing a personalized learning experience.

By making sure there are clear links between what students need to learn, how they are tested, and the activities they do, universities can build an educational journey that is well-integrated and transformative. This will give students the tools they need to face the challenges and make the most of the opportunities that come with a future shaped by AI.

The discussion above highlights the importance of a holistic approach to integrating generative AI in higher education. To provide a more concrete understanding of how this can be achieved, we will now delve into the specific elements that comprise this approach: future-oriented learning objectives, AI-compatible assessment methods, and learning activities that leverage AI. By examining each of these components in detail, we can gain a clearer picture of how educators can effectively adapt to the rapidly evolving landscape of AI in higher education and equip students with the skills and knowledge they need to succeed in an AI-driven world. Below we explore these elements in depth, providing examples and insights drawn from the current literature:

Future-Oriented Learning Objectives

1. Critical thinking
   1. Analyzing and evaluating information from multiple sources
   2. Questioning assumptions and biases
   3. Making reasoned judgments and decisions
2. Problem-solving
   1. Identifying and defining problems
   2. Generating and testing potential solutions
   3. Implementing and evaluating solutions
3. Creativity
   1.  Generating novel and original ideas
   2.  Combining existing ideas in new ways
   3. Thinking outside the box and taking risks
4.  Collaboration
   1.  Working effectively in teams
   2. Communicating and sharing ideas
   3. Negotiating and resolving conflicts
5. Digital literacy
   1. Using digital tools and platforms effectively
   2. Evaluating the credibility and reliability of online information
   3. Understanding digital privacy and security
6. AI literacy
   1. Understanding AI concepts and terminology
   2. Recognizing AI capabilities and limitations
   3. Using AI tools ethically and responsibly
7. Emotional intelligence
   1. Recognizing and managing one's own emotions
   2. Empathizing with others and understanding their perspectives
   3. Building and maintaining positive relationships
8. Adaptability
   1. Embracing change and uncertainty
   2. Learning new skills and knowledge quickly
   3. Adjusting to new situations and challenges

AI-Compatible Assessment Methods

1. Formative assessments
   1. Reflective journals: Students write regular entries reflecting on their learning experiences and progress
   2. Peer feedback: Students provide constructive feedback on each other's work and ideas
   3. Self-assessments: Students evaluate their own strengths, weaknesses, and areas for improvement
2. Performance-based assessments
   1. Project work: Students complete authentic, real-world projects that demonstrate their knowledge and skills
   2. Presentations: Students present their work and ideas to an audience and respond to questions and feedback
   3. Simulations: Students engage in realistic simulations or case studies that test their ability to apply knowledge in context
3. AI-assisted assessment tools
   1. Automated essay scoring: AI algorithms evaluate the quality and content of student essays and provide feedback
   2. Adaptive testing: AI algorithms adjust the difficulty and content of test questions based on student performance
   3. Intelligent grading: AI algorithms assist educators in grading assignments and providing personalized feedback

Learning Activities with AI

1. Project-based learning
   1. Collaborative AI projects: Students work in teams to design, develop, and evaluate AI models or applications
   2. Interdisciplinary AI projects: Students apply AI techniques to solve problems in different domains, such as healthcare, finance, or environmental science
   3. AI ethics projects: Students explore the ethical implications of AI and develop guidelines for responsible AI use
2. Problem-based learning
   1. AI case studies: Students analyze real-world AI applications and identify potential problems or improvements
   2. AI design challenges: Students design AI solutions to specific problems or scenarios, considering technical, ethical, and social factors
   3. AI debate clubs: Students engage in structured debates on AI topics, such as the impact of AI on jobs or privacy
3. Integration of generative AI tools
   1. AI-assisted brainstorming: Students use generative AI tools like ChatGPT to generate ideas and explore new perspectives
   2. AI-assisted content creation: Students use generative AI tools to create drafts, outlines, or examples that they can build upon and refine
   3. AI-assisted feedback: Students use generative AI tools to receive immediate feedback on their writing, coding, or problem-solving
4. AI-supported personalized learning experiences
   1. Adaptive learning systems: AI algorithms adjust the content and pace of instruction based on student performance and preferences
   2. Intelligent tutoring systems: AI-powered tutors provide personalized guidance, explanations, and feedback to students
   3. Personalized learning paths: AI algorithms recommend learning resources and activities based on student interests and goals

Table 1. Overview of learning objectives, assessment methods and learning activities addressed in the literature.

References

• Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kuhn, J., & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
• Zhai, X. (2023). ChatGPT and AI: The game changer for education. SSRN. ChatGPT and AI: The Game Changer for Education by Xiaoming Zhai :: SSRN
• Chiu, T. K. F. (2024). The impact of students’ use of ChatGPT on their research skills: The mediating role of feedback literacy. Education and Information Technologies, 29(1), 123–140. https://doi.org/10.1007/s10639-024-12981-9
• Gašević, D., Dawson, S., & Siemens, G. (2023). Let’s not forget: Learning analytics are about learning. TechTrends, 67(2), 123–130. https://doi.org/10.1007/s11528-014-0822-x
• Essel, H. B., Nunoo, F. K. N., & Agyemang, B. (2024). Adapting to the future: ChatGPT as a means for supporting constructivist learning environments. TechTrends, 68(1), 37–46. https://doi.org/10.1007/s11528-023-00899-x
• Rudolph, J., Tan, S. M., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning & Teaching, 6(1), 1–22. https://doi.org/10.37074/jalt.2023.6.1.9

Recommendations

WP1

Recommendations based on the literature review from WP1.

Higher education institutions should focus on redesigning their curricula and assessment methods to emphasize skills that AI cannot easily replicate. This means shifting from content-focused instruction to developing higher-order thinking skills through project-based learning and collaborative activities. Educators should integrate AI tools like ChatGPT into their teaching while maintaining clear guidelines for ethical use, helping students understand both the capabilities and limitations of these technologies.

Assessment strategies should move away from traditional essays and exams toward performance-based evaluation methods that demonstrate authentic learning and application of knowledge. This includes implementing more real-time assessments like presentations, group projects, and case studies that require students to demonstrate critical thinking, problem-solving, and creativity while applying their knowledge in practical contexts.

WORK IN PROGRESS...

• WP2: Assessing student interactions with GenAI in teh context of academic writing 
• WP3: Pilot studies using GenAI based tutoring applications in the classroom
• WP4: Workshops on GenAI tutor building, AI literacy, Co-working with GenAI

Practical outcomes

WP1

Practical recommendations based on literature review of future-oriented learning objectives and AI-compatible assessment method:

1. Redesign assignments to focus on process rather than just final output
2. Implement regular reflective journaling and self-assessment components
3. Use project-based learning with real-world applications
4. Incorporate AI literacy training into course content
5. Develop clear guidelines for appropriate AI tool use
6. Design assessments that require in-person demonstration of skills
7. Create collaborative projects that emphasize human interaction and teamwork
8. Use formative assessments throughout courses rather than relying on final exams
9. Include ethical considerations of AI use in course discussions
10. Implement peer review and feedback processes to enhance learning

WORK IN PROGRESS...

• WP2: Assessing student interactions with GenAI in teh context of academic writing 
• WP3: Pilot studies using GenAI based tutoring applications in the classroom
• WP4: Workshops on GenAI tutor building, AI literacy, Co-working with GenAI