Efficient and Reliable Online Homologation Recommendation

Project introduction and background information

The Embedded Systems (ES) master program is 3TU master program, organized in close collaboration with the Technical University of Delft and the University of Twente. At the TU/e, it is a co-production of the departments of Mathematics and Computer Science and Electrical Engineering. The ES master program admits students with BSc degrees ranging from Electrical Engineering and Electronics to Computer Engineering and Computer Science. It has, moreover, a predominantly international intake. Due to the multidisciplinary aspect of the program and the diversity of the intake, virtually all students admitted to it need to repair at least some deficiencies at BSc level. For the purpose of repairing deficiencies at BSc level, the Embedded Systems program offers the possibility of homologation: taking BSc level courses as part of the master program. In the past, assigning homologation packages to ES students worked as follows:

• Students with a BSc degree in Computer Science (CS) or Electrical Engineering (EE) from a Dutch university, which are automatically admitted to the program, had to take a fixed homologation package consisting of three courses from the BSc majors on Software Science or Electrical Engineering, respectively, as a mandatory part of their Embedded Systems program.
• International applicants were evaluated by the program manager Embedded Systems at the time of their admission, and on the basis of that evaluation, a homologation package was assigned.

The approach for students with a BSc degree in CS or EE from a Dutch university is out of date. These students nowadays have had a much larger elective space in their BSc program, which they may have used to already prepare for the ES master program. The approach for international applicants is inefficient and unreliable. It is inefficient because the program manager needs to assess the background of a student in much more detail than would be necessary for admission alone. It is unreliable because the admission data often does not provide sufficient detail about the contents of courses part of the BSc program of the applicant to assess his or her deficiencies. The ES master program has undergone a considerable redesign to satisfy the requirements of the TU/e Graduate School. The mandatory part of the program has been reduced, and students now have a choice of four thematic streams. This has further complicated the task of assigning appropriate homologation. Whether a student has deficiencies now only depends on his or her background, but also on the choice of stream. Since the homologation package is considered part of the elective space of the program, it is to be preferred that students do not spend too much time on homologation. Starting in the academic year 2015-2016, homologation is offered in the form of socalled homologation modules, special 2.5 EC variants of courses of the BSc majors on Software Science or Electrical Engineering. Students are allowed to spend at most 10 EC on such homologation modules, which then count as free electives in their study program. Students are no longer assigned fixed and mandatory homologation packages. Instead, at the start of the program, they are explained in rather general terms what are the prerequisite knowledge and skills for the mandatory courses of the ES program and for the different streams. At present it is, however, hard for students to assess whether they satisfy the prerequisite knowledge and skills.

Objective and expected outcomes

The goal of the project is to improve the method to assess to what extent an admitted ES student satisfies the prerequisite knowledge and skills needed for the ES program, and, on the basis of that assessment, provide a qualitative and reliable recommendation to the ES student on how to repair deficiencies. The improved method should be reliable to the extent that, on the one hand, students do satisfy the prerequisites of the program after completing their homologation package, but, on the other hand, do not take homologation modules they do not need. If possible, the method should refer the student to a more efficient way of repairing deficiencies than taking an entire homologation module (e.g., refer the student to some online course content). It is a concern that the method requires little or no interference of ES lecturers or ES program management. We intend to implement the online test using the new LMS to be acquired by the TU/e. Thus, the project can, at the same time, serve as a test case for the LMS.  

Scope

A previous project (Dutch title: Handel met Voorkennis), sponsored by the TU/e Innovation Fund in 2014, has resulted in an online test to measure prerequisite knowledge. The present project will build on the experience obtained in that project, and will focus more on how to measure prerequisite skills. Furthermore, the present project will focus on giving an adequate recommendation to the student. We are simultaneously working on turning homologation modules into online courses that students may follow in their own time and at their own pace. This is a project on its own and thus not included in this proposal. 

Expected outcomes

At the end of the project the following results will be delivered:

An automated online assessment testing prior (knowledge and) skills. Including:

• assessment items for all agreed upon prior knowledge and skills the students should have to successfully get through the master program.
• qualitative standardized feedback that students receive depending on their answers/solutions.
• an automatically generated advice for each individual student for a homologation package and/or use of online available study materials based on the assessment result.
• extra study materials that will be provided within the LMS, directly linked to the feedback the student receives. (In some cases, it may not be necessary to follow an entire homologation module, but only part of it, and in some cases studying some piece of online course material may suffice.)

If necessary: a tool that extends the OnCourse/LMS system making automated analyses of the results possible. 

Results and learnings

In the summer of 2016 we have invited 38 newly admitted Embedded Systems students to use our homologation recommendation tool. Of these 38 students, 21 students have used the tool, doing the preknowledge checks on the two topics most relevant for the Embedded Systems program (Computer Architecture and C programming and Logic and Set Theory). On the topic of Computer Architecture and C programming, 13 of the 21 participants passed the preknowledge check; of these 13, only 4 completed the associated quiz, and only 1 of these 4 participants passed the quiz. For the quizzes on the topic of Logic and Set Theory, a similar the phenomenon was observed: 3 out of 16 participants passed the preknowledge check and 1 out of 3 participants passed the quiz.On the one hand, one may conclude from the low passing percentage that it is difficult for students to give a good estimation whether they possess the required knowledge and skills on a topic. The quizzes are apparently necessary to provide a more reliably confirmation. On the other hand, students also indicated that the quizzes currently perhaps rely too much on specific terminology that is introduced in the corresponding bachelor courses. For a homologation recommendation tool, it is particularly important to either use generally accepted terminology, or include an explanation of terminology used in the formulation of the question. We are under the impression that it still takes students too long to use the tool. Not many students complete all the offered quizzes. In future versions of the tool, we would therefore like to experiment more extensively with adaptive testing, and with more refined recommendations. To this end, we should establish, within the tool, a more refined correspondence between intended learning outcomes, quiz questions and recommendation. For some of the quizzes, we should still confirm their validity with the lecturers of the mandatory courses relying on the knowledge and skills assessed.

Recommendations

Constructing digital tests is a lot of work. Unfortunately, a considerable part of this work is hard to delegate to student assistants, but student assistants can alleviate the task of entering digital quiz questions into the system. With the current infrastructure, it is rather cumbersome to implement the necessary control flow (Phase 4 above). In 2017, we will run a Software Engineering project in which a group of Software Science bachelor students are supposed to improve the underlying infrastructure of the tool. This should make it easier for other master programmes to use the technology.

Practical outcomes

See end report at the downloads on the right.