Evaluation of Educational Technology Innovations at WUR

Project introduction and background information

WUR has created a new procedure for innovating education via Educational Technology (EdTech), the so called Community for Education Innovation with EdTech. It aims to support teachers that want to experiment with new IT tools (EdTech) in their education in the best possible way. This community tries to prepare for a possible scale-up of the new IT tool within WUR at a later stage. The education innovation with EdTech procedure totally entails 11 steps beginning with an idea about educational innovation with an IT tool that is not in use at WUR (step 1) and ending with a decision on either further implementation of the IT tool or giving up using that tool (step 11) (see Figure 1). All WUR teachers are welcome to apply for the education innovation with a new IT tool (EdTech) experiment.

^{Figure 1. The education innovation with EdTech procedure at WUR}

As part of education innovation with the EdTech procedure at WUR, one of the important and required steps is the “evaluation” (step 9) which basically means that at the end of the IT tool experimentation, evaluation will be conducted in terms of different factors such as general satisfaction, ease of use, added value, strength and weakness, and usefulness of the experimented IT tool to decide whether the it should be supported for further implementation on a large scale or not. The Education and Learning Sciences chair group (ELS) is in principle¹ responsible to conduct these evaluations, in cooperation with the teachers/course coordinators involved, the IT support department and ESA. For EdTech evaluation, Dr. Omid Noroozi from ELS is the project coordinator.

Types of the EdTech evaluation varies from qualitative evaluation including analysis of the documents, semi-structured interviews and focus group discussions with teachers and also students who are experiencing with the IT tool to quantitative evaluation including surveys which are being collected through PACE evaluation. Although based on the nature and goals of the IT tool, evaluation factors might differ, generally, it includes collecting and analysis of data regarding the IT tool’s weaknesses and strengths, ease of use, added values, effectiveness and usefulness, time-saving for both teachers and students, contribution to educational goals and processes, and teachers and students general satisfaction with the IT tool.

^{[1]. If the courses involved are ELS courses, it is ensured that the researching staff are not also involved as teachers. Potentially, in such a case a researcher from another chair group or someone from ESA might be involved. In case teachers want to investigate innovations with EdTech as part of a Senior Qualification in Education, this is also possible; in such cases they will be supervised by ELS staff.}

Objective and expected outcomes

The main goal of EdTech experimentation at WUR is to increase the quality of education and learning. Suggested IT tools by teachers are experimented on a small scale to see whether they support teachers’ training and students’ learning or not. In this regard, evaluation (step 9) plays a key role to provide an overall and detailed view of the effectiveness and usefulness of the experimented IT tool in achieving respected goals. Thus, the goal of the EdTech evaluation is to provide a report on the experimented IT tool’s effectiveness for WUR courses. If the results for the IT tool are positive then WUR will consider further implementation of the IT tool.

Results and learnings

Currently, the EdTech evaluation is responsible for evaluating four IT tools at WUR including Comproved, TrainTool, CodeGrade, and Grasple. For each tool, the evaluation reports will be shared in the downloads section on this page.

Comproved

Comproved is an assessment tool that works based on comparative judgments. This tool structures comparisons of students’ work while aiming to be a valid, reliable and user-friendly instrument.  The goal of this tool is to reduce the workload of teachers and improve the quality of assessment. The main users of Comproved are teachers. At WUR, Comproved was used in one of the courses for assessing and grading students. After experimenting with the tool, data was collected in both quantitative and qualitative formats from teachers and students through focus group discussions and surveys and analysed. The overall conclusion of the tool shows that the assessment is more valid (different teachers) and less biased and that the tool is technically easy to use by teachers, but not pleasant emotionally. The teachers’ satisfaction rate was 5.8 out of 10. Students’ satisfaction rate was 6.2 out of 10. Overall, teachers and students were dissatisfied with the tool. They found that the feedback is more general and superficial, it increased the workload of teachers and they had to invest a lot of time in it. The tool is not successful in attaining its goals.

TrainTool

TrainTool is an online video training tool that focuses on practicing and testing students' communication skills. The main goal of this tool is to improve students' communication skills.  With this tool, students can record their speech, watch it in private for self-practice or they can ask for feedback from their peers or teachers. To evaluate this tool, varied types of data from teachers and students were collected in two rounds within two academic years through focus group discussions, in-depth interviews, online surveys, and reflection reports. The findings showed that TrainTool is well-structured, user-friendly, easy to use, and it helps teachers and students with achieving the learning goal of the tool which is for practicing communication skills. Our general impression is that TrainTool can contribute to the quality of teaching and learning by having features such as (a) self-practice and self-reflection, (b) safe self-pace, active and engaging learning environment, (c) structure, user-friendliness, and clearance (d) peer and teacher feedback functionalities. However, It would be great to make improvements in terms of interactivity, in-depth learning and assessment, decreasing workload, and group activities.

CodeGrade

CodeGrade is a grading and feedback platform for computer science. The tool helps both teachers and students in computer science courses that deal with scripting and coding. CodeGrade was used by different courses at WUR in the 2020-2021 academic year. To evaluate CodeGrade, different kinds of data from teachers and students were collected in two rounds within two academic years through focus group discussions, online surveys, and demo recordings on the tool. Our general impression is that the benefits of CodeGrade slightly outweigh its drawbacks. The tool contributes to the quality of teaching and learning by having features such as auto-grading, plagiarism check, self-assessment, structure, and feedback. However, further improvements are suggested in terms of reliability, flexibility, interactivity, interface, and intelligence.

Grasple

Grasple is an online practice and assessment platform for math and statistics education. The main goal of this platform is to improve match and statistics teaching and learning by providing an interactive and user-friendly environment for practice. Grasple was used in two courses at WUR in the 2021-2022 academic year. To evaluate Grasple, different kinds of data from teachers and students were collected through focus group discussions, individual interviews, and online surveys. The results showed that both teachers and students were satisfied with using Grasple and they recommended using Grasple for similar match and statistics courses. However, Grasple was not found as a user-friendly tool by both teachers or students and they reported no decrease in their workload. In addition, teachers reported that although Grasple added educational value to the courses, its cost-effectiveness was under question.

More

News article: EdTech Project to evaluate educational technology innovations at WUR

^{Header image: Steffie Sauren}