Project introduction and background information
A course often ends with a final exam. This exam tests how well a student understands the various topics discussed in the course. An analysis of the work of an individual student, as well as, the work of the whole class, provides valuable information for the student and for the teacher.
Informed students will be aware of their strong and weak points. For students that passed the course, this can be of importance when starting new courses in which pre-knowledge on topics of the exam is used, and, for those that failed the exam, when they are preparing for the resit.
Informed teachers will not only get a better idea of the quality of the exam, but will also be able to offer better support to students preparing for a resit of the exam.
We have developed a tool based on a spreadsheet that on input of the partial scores on an exam provides the teacher with an analysis of the exam. This includes a standard item analysis, including p-values as well as rir and rit-values of each individual item. Moreover, the teacher can divide the various items of the exam into topics. In this way he or she also gets an analysis of the exam with respect to these topics.
The tool also provides information on the performance of an individual student with respect to its peers. By connecting the spreadsheet to Mathematica, a graphical representation of this information is created which can be communicated to the student by a personal email. For mailing the personal information and feedback to individual students we have been using the Mail Merge facilities of an email client.
Objective and expected outcomes
The goal of the project FEEST is twofold:
- Provide teachers with a tool to analyze exams and report this analysis to their students;
- Use the analysis of the exam to guide students preparing for a resit of the exam.
For the first goal we have developed a tool based on a spreadsheet that on input of the partial scores on an exam provides the teacher with an analysis of the exam. This includes a standard item analysis, including p-values as well as rir and rit-values of each individual item.
Moreover, the teacher can divide the various items of the exam into topics. In this way he or she also gets an analysis of the exam with respect to these topics. The tool also provides information on the performance of an individual student with respect to its peers. By connecting the spreadsheet to Mathematica, a graphical representation of this information is created which can be communicated to the student by a personal email. For mailing the personal information and feedback to individual students we have been using the Merge Mail facilities of an email client. How to use the Spreadsheet, the Mathematica-program and Mail Merge has been described in manuals.
The second goal was to use the analysis to help students that failed the exam in their preparation for the resit. For the Calculus B resit in January 2016 the participants did not only get information on how they performed on the individual items of the first test, but also how they performed on the various topics covered by this test.
Results and learnings
Based on the analysis they were given advice to pay extra attention to those topics on which their performance was poor. For this purpose they also have been offered extra homework mainly focusing on the topics in which they did not perform well.
The homework was offered to the students as an assignment within our learning management system (LMS, based on Moodle). The students that handed in this homework by uploading it to the LMS, were offered extra support:
- The student obtained a worked out solution of the homework and a grading scheme. In this way the student was able to not only correct his or her own work, but also grade it. This worked out solution was provided to the student via the feedback tools of the LMS assignment.
- The student could attend an extra one hour tutor meeting in groups of 8 to 10 students, to discuss the homework and ask questions preparing for the resit.
The last two days before the resit, a help desk was installed where all students, not only those that did hand in homework, could ask questions.
Exactly 50 students (11%) handed in the homework and made use of the extra support. It turned out that their scores for the resit have been considerably better than for the remaining students.
There where the passing rate for the resit was only 31%, the passing rate among the students participating in the experiment was 44%. (Over the last couple of years, the passing rate for the resit of this Calculus exam has always been around the 30%.) Of the 50 students participating, 78% scored better for the resit than for the original exam, while this was only the case for 64% of the remaining students. The increase in score was much higher for the participants in the test. Indeed, on average they improved their grade with 1.1 point out of 10, while the average improvement for all students was 0.6 points.
We created a tool that not only provides a teacher with an analysis of an exam on item or topic level, but also gives him or her the opportunity to give students personalized feedback on the exam. We have used the tool to give students that failed a Calculus exam feedback on their performance and advice on how to prepare for the resit. There have been 50 out of 441 students that took advantage of our efforts and participated in the experiment. These student performed much better (although still not very good) than the remaining students. This indicates that our intervention has a positive effect on the performance of the participants.
Of course there are many other factors that can have an impact on the performance of these students, than just the feedback and advice and support that we have offered them. For example, the fact that these students voluntarily participated in the experiment, shows that they are well motivated to work for the resit. However, besides the positive impact being a reason for providing students with this feedback and help, we also feel it to be right to offer students that prove to be motivated, extra possibilities to prepare well for an exam.
Practical outcomes
The spreadsheet developed in the project is an easy to use tool that can be used to analyze exams. The analysis can not only help teachers to improve their examination but also inform the educational management on the quality of the examination. The tools also provide a way to inform students individually on their performance at the exam. The experiment indicates that this has a positive effect on the results obtained by these students. However, as there are many other factors involved, a better analysis of the effects is in order. See the downloads.