MEMI: Measurable effects of mini-lectures on improving student engagement and outcomes

Project introduction and background information

Update 21/12/2022: A BSc thesis analysing the data of the project has been added.

Learning mathematics is a complex process, requiring many conceptual lenses and rich data sources to document and understand students' construction of knowledge. Specifically within the context of learning mathematics in university, using online videos in university mathematics modules has been shown to have advantages. Perceived benefits of videos include flexibility of scheduling and pace, and avoidance of large, long lectures. Research has also shown that  though videos provide a useful resource, they should be used in this context only in conjunction with lectures. Findings show that students use videos as either a complement to, or substitute for, the lecture, and time spent using either or both resources has a significant impact on learning.

In this project, we test these findings in the context of a newly redesigned course on the Masters level. The course is an established Masters course in the core of the programme of the department. It comprises three distinct topics in stochastics. The current design involves 4 lecture hours per week, 4 guided self-study instruction hours, three midterm examinations (one on each topic), and a final exam. The material currently offered to students is lecture notes, one single book covering all topics, instruction sets, sketches of solutions to the instruction exercises, low-level comprehension quizzes per lecture, videos of the complete series of lectures from previous years, and practice exam sets for the final examination.

Objective and expected outcomes

For the project, new videos of mini-lectures are created to complement this material. A light board is used, which mimics the experience of a teacher writing a proof in class in terms of pacing, visualisation, and level of information. A third of the videos works through material already offered to the students but not explained in class and two thirds present new notions (that are not part of the assessment of the course). The objectives are to increasing the students’ engagement with the course and improving their outcomes. The project aims to estimate whether students are triggered more by innovation and are willing to explore new resources intrinsically due to their novelty or whether these resources have an inherent added value to students (in the specific context of a specialised and rather challenging first year Master course of the IAM curriculum.)

In the design of the project, an appropriate control group has been created that allows comparison of grades between different student populations. The engagement of the students will be evaluated through observation of their interactions and their self-evaluation on how engaging they found the material as reported in the course evaluation.

Results and learnings

In the design of the project, appropriate control groups have been created that allow comparison of grades between different student populations. The engagement of the students is evaluated through their self-evaluation on how engaging they found the material as reported in the course evaluation.

The new material was created in Spring 2021. The experiment ran in Q1 2021-2022 and the results are reported in the BSc thesis attached here by V. Engbers. The thesis was supervised by the PI, Prof. Dr. Maria Vlasiou, together with Dr. Marta Regis. 

The results are summarised in the attached report, which was a paper accepted at SEFI 2023.

Recommendations

There no statistically significant improvement in grades between Years 1 and 2. On topic (A) of the final exam, grades seem to be improved, but statistical significance is not reached after Bonferroni correction (𝑝-value=0.033). On the other hand, the effect size is positive (0.265, 95%CI=[0.003, 0.493]). Also on midterm (A), the effect size is positive (0.217, 95%CI=[-0.087, 0.483]), although the result is not statistically significant (𝑝-value=0.149).

As evidenced by the lack of significant unilateral changes in grades for (B,C) between years, we find that one cohort of students was not significantly overall performing better or worse than the other. Only on the midterm (C), we observed an increase in grades (Y1 7.50 [7.00, 8.50], Y2 8.70 [8.06, 9.71], 𝑝-value<0.001), but the results on the final (B,C,D) have negative effect sizes and reach statistical significance for (C,D).

We also do not observe a significant difference in grades between students that watched at least one video and students that did not watch any videos within Year 2. This indicates that selection bias did not play a significant effect in the analysis. Also, by the removal of non-discriminatory questions, the most important confounding factors in this analysis have been accounted for. Therefore, we find only a mild marginal effect of videos improving students’ grades.

We find some evidence of improvement in motivation in the group of students who watched the videos. Concretely, there is no significant improvement in motivation for (A), but there is a significant improvement in motivation for the course as a whole.

Moreover, students who watched the videos did not engage more in the course in terms of hours spent per topic. As the group that watched at least one video did not significantly work harder for the course than those that did not, we find that the former group was not engaged with the course more than the latter in terms of hours spent on the course. This outcome shows that the group that chose to watch the videos are not inherently working harder, if measured purely by the amount of time they spend on the course. Therefore, the effect of videos on motivation is marginal, which concludes our second research question. On the other hand, it could be that students already inclined to the subject (and that need to spend less time to grasp the compulsory material), also engage with the videos. However, this is not visible in the grades.

Practical outcomes

Combining these results implies that the effect of including video material in combination with in-person lectures is modest. Our sample size was limited, especially when considering the complex framework that is analysed and the small effect sizes at hand. This may have prevented us from finding statistically significant results. For similar courses, it is therefore advisable to critically assess if the time and effort for the creation and integration of video is justifiable when the benefits are expected to be limited. On the other hand, the analysis shows no significant negative effect on students’ grades or motivation. If videos are readily available and easily to implement, we find no evidence against including them as optional material; however, expectations should be managed accordingly.

Overall, our research shows no statistical significant results, but modest effects of this intervention. In closing, it is important to highlight and emphasize the critical significance of not merely assuming the effectiveness of technological interventions in education. Instead, it is crucial to encourage rigorous educational research that enables more thoughtful and informed assessments of similar integrations.