Intermittent quizzes in video lectures

Project introduction and background information

Online lectures provide many benefits to students. Students are more flexible in when and where they learn, and they can go through the material at their own pace. However, online lectures also provide challenges. Research examining how students learn has revealed that online education critically depends on self-control skills (Schacter & Szpunar, in press). Selfcontrol fosters the ability to stay on-task when our minds would rather wander. It allows people to restrain momentary desires or distractions to reach long-term goals. One of the biggest challenges students face during lectures is to prevent mind-wandering, and to keep paying attention to the lecture. It is well known that students find it difficult to keep paying attention the longer a lecture takes, and that mind-wandering negatively affects how much students learn from a lecture. Psychological research has found that one way to prevent mind-wandering during online lectures is to provide short quiz questions throughout an online lecture. Recent studies show that including such short quiz questions improve performance on tests about the lecture content (Soderstrom & Bjork, 2014; Szpunar, Khan, & Schacter, 2013).

In addition to trying to prevent mind-wandering, self-regulated learning requires other metacognitive processes. One of these is monitoring how well students understand the contents of the lecture. In classrooms, students can ask clarifying questions (either to fellow students or the lecturer), but this is not possible in online lectures. Intermittent quizzes in video lectures can provide students feedback on how well they understand the lecture content, and have been found to reduce students’ anxiety about the final test for a course (Schacter & Szpunar, in press). Making online lectures more attractive and easier to set one’s mind to is of special importance in the TU/e context, where the number of courses with substantial numbers of students is large, online lectures are offered more and more, but the attendance to standard online video lectures is low (Gorissen, Van Bruggen, & Jochems, 2012). These studies provide a clear indication that introducing short quiz questions during video lectures can help to improve students’ performance. However, there are a number of important questions that need to be addressed before these insights can be applied in education. These questions are both theoretical and practical in nature. 

Objective and expected outcomes

In this project, the goal is to examine both the practical aspects of implementing quizzes in online environments (so that responses can easily be stored and analyzed by the lecturer), as well as the theoretical aspects of when quizzes work optimally, and what their long-term effects are. When intermittent quizzes are used in online lectures, one important question is how performance on these quizzes can be tracked. While it is relatively easy to introduce quizzes and questions in online lectures where performance is not tracked (e.g., ‘please pause this video and answer this question for yourself, then continue the video’) actively taking part in quizzes where performance is not tracked is in itself a self-regulation challenge. A further reason to keep track of students’ performance on quizzes during online lectures is that it allow lecturers to easily keep track of topics that require additional clarification during contact hours. Thus, it is important to examine how we can create quizzes where the performance is tracked, preferably by integrating quizzes in web lectures that are integrated in the Learning Management System(s) used at our university. One of the objectives of this project is to find the best (existing) software to achieve this, and work out the practical details to implement this software in our Learning Management Systems (LMS). We will focus on SCORM compliant solutions (the standard for web-based electronic educational technology) to make sure our solution will work with other (or future) learning management systems, and such that even if proprietary software that is not SCORM compatible would be used at the TU/e in the future, teachers can easily use free open software LMS solutions (e.g., MOODLE) to implement intermittent quizzes in video lectures. The practical aim is that any lecturer with a basic understanding of computer software should be able to add quizzes to online videos and analyze the results, as effortlessly as possible. A more theoretical question is where the trade-off lies between the use of quiz questions to increase students’ attention, and their disruptive effect on the flow of a lecture. Little is known on the number of quizzes to be used to achieve optimal efficiency, and whether this optimum purely depends on the duration of a lecture, or on its content (e.g., density of information, topic, etc.). A second theoretical question is to what extent the beneficial effects of quizzes that have been observed in recent experiments depend on their novelty (e.g., students trying out something novel) and how well these effects hold up in online lecture series that continue for several weeks. A third theoretical question is whether only knowledge and insight (as has been demonstrated in the literature), or also practical skills can be tested. We aim to examine these theoretical questions in our student population.

Results and learnings

In this project, we have shown intermittent quizzes can be created in online video lectures using easy to use software (Camtasia), and package in a format that is easily uploaded in a general format (SCORM). The grades students get for the online quizzes can be stored in the LMS and calculated at the end of the course. 

The project has two outcomes:

1. On the one hand, we worked out the practical details of how to create online quizzes in video lectures, providing teachers with the knowhow of which software to use, and a practical ‘how-to’ explaining step-by-step how to create such quizzes, and make sure students’ scores are accessible in Canvas (or another Learning Management System). We chose to work with TechSmith's Camtasia. Camtasia is a commercial screen-recording product for producing demonstrations and tutorials. It has SCORM compliant output as an option, and permits quizzing and branching. The SCORM output option creates a single .zip file that is appropriate for uploading into a SCORM activity. This package had the following benefits: (1) Camtasia is very easy to use and it integrates powerpoint and allows you to easily record your lectures (2). The last reason is that the SCORM format has a very specific output, which includes the total Score (e.g., for a video lecture with 3 questions, whether learners had 100% correct, 66.67% correct, or 33.33% correct), but does not provide information about each individual question. As such, teachers know how well students did overall, and can incorporate these grades in the formal assessment, but teachers can not see which specific questions were difficult for students. TechSmith has acknowledged this limitation in SCORM output, and provides an additional service when creating video lectures, where you can opt-in for free e-mail summaries that contain more detailed information (see the example below). In our view, this option, together with the lower price and ease of use, make Camtasia the best choice of software.
2. On the other hand, we examined whether online quizzes are actually beneficial in our student population. Finally, evaluations indicate that even though students initially expected they would prefer quizzes at the end of the lectures, they preferred intermittent video quizzes after experiencing them. Furthermore, students indicated the quizzes helped them to keep their attention with the online video lecture. This project provides support for the idea that intermittent video quizzes are beneficial in online video lectures. Future research (planned in the Coursera course) is needed to examine whether intermittent video quizzes actually improve quiz performance and grades compares to not using intermittent quizzes in online video lectures, but it seems advisable to incorporate intermittent video quizzes in online video lectures. 

A full evaluation of this project is planned for the summer of 2017.

Recommendations

A benefit of using Camtasia is that the process of adding quizzes to videos, and packaging the videos for SCORM, is straightforward and very well supported through instructional videos. A video explaining the process to create video quizzes is available and also a video explaining the process to create SCORM compliant video quizzes. We recommend users to watch these videos.

Practical outcomes

Please have a look at the downloads.