UNCAGE

Project introduction and background information

Due to the proliferation of intelligent personal assistants like Siri, Amazon Echo, Microsoft Cortana, and Google Home, and the use of chatbots and voice/speech enabled systems, human-machine interaction is becoming increasingly common. Conversational interfaces and user interactions with conversational agents have been argued to have advantages over traditional GUIs due to having a more human-like interaction [2]. Microsoft alone reports that three thousand new bots are built every week on their bot framework [3]. A study by Markets and Markets predicted that the conversational AI market size will grow from $4.2 billion in 2019 to $15.7 billion by 2024 reaching a compound annual growth rate of 30.2% [4]. The usage of conversational bots has been found to significantly improve both the academic skills [5] and well-being of university students [6,7]. Given the newness of the field and its importance to transforming education and wellbeing, there are significant research gaps that must be addressed. Before completely embracing the role, that conversational agents may play in promoting student education and wellbeing, we must first fully comprehend the associated opportunities and limitations.

In this context, the purpose of this initiative is to assist students in meeting their emotional and academic goals. As a result, this project has two goals:

1.  To assist students in developing effective study skills
2. To meet their emotional needs and well-being.

Regarding the first goal, we will develop novel conversational agents to aid learners in the development of effective study skills. More precisely, we are interested in how can conversational agents be adapted to student needs and personalized based on their preferences? What design factors can influence the reliance on and engagement with the conversational agents? And how can conversational agents help students in the processes of study planning, goal setting, and goal monitoring? The second goal of the project is that we aim to build conversational agents that can help tackle specific mental health problems (for example, stress and anxiety) and improve wellbeing of students. We are focusing on students because they are particularly more vulnerable due to the sudden and radical changes in on-campus education. We are specifically interested in the following question: How can we seamlessly assess student wellbeing through interactions with conversational agents in educational contexts?  We are interested in two elements of school life in this project: using conversational agents as mentors to educate student mentors about stress management skills and using conversational agents as emotional support tools or companions to help students with mental health concerns.

Objective and expected outcomes

With regards to the project's first objective, we have already developed and deployed a conversational agent called Goalkeeper (https://goalkeeper2021.herokuapp.com/). The purpose of this Goalkeeper is to assist Erasmus students with setting academic goals for a single academic 'block' or 'semester,' which would last approximately five weeks. Students will be randomized randomly to either the Goalkeeper or the online version of the intervention. Students will have access to goal setting training and intervention modules prior to the commencement of the course. They will be asked to establish academic goals for the upcoming academic course. Participants will receive a link to a brief update module once a week for the duration of the course, in which they will be reminded of their goals and asked about their progress and motivation. At the conclusion of the course, a final assessment will be used to determine objective attainment. More precisely, we are looking for information on the following study questions:

1. Does the effectiveness of goal setting training vary depending on whether it is provided via conversational agent or static webpage?
2. To what extent does the medium in which the goal setting intervention is delivered influence the content and quality of the objectives set?
3. What effect does the medium through which the goal setting intervention is delivered have on motivation and goal attainment?

Regarding the second objective, we have built a Trainbot [1], which is a conversational interface that trains non-expert individuals on the technique of Motivational Interviewing (MI), which is a powerful counseling approach for treating anxiety, depression, and other mental problems (Miller and Rollnick 2012). Results from previous controlled experiments using crowd workers show that workers using Trainbot: felt lesser pressure than those in the control group; provided psychological interventions that were rated consistently higher by psychologists than those in the control group; felt a higher self-efficacy in helping deal with stress management after the training process. In this project, we aim to support student mentors at TU Delft in learning the technique of Motivational Interviewing (MI) by going through a training process with Trainbot. We expect that student mentors can benefit from learning to provide support to stressed students and escalate cases where professional help is required. We also believe that students who interact with mentors who have completed training with Trainbot, will demonstrate greater satisfaction from their interactions with their mentors.

References

_{[1] Abbas, T., Khan, V. J., Gadiraju, U., & Markopoulos, P. (2020, October). Trainbot: A Conversational Interface to Train Crowd Workers for Delivering On-Demand Therapy. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 8, No. 1, pp. 3-12).}

_{[2]. Moore, R. J., Arar, R., Ren, G. J., & Szymanski, M. H. (2017, May). Conversational UX design. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (pp. 492-497).}

_{[3]. Holland, B. (2018, April).  The Rise of Intelligent Conversational UI. Retrieved from:   https://www.smashingmagazine.com/2018/04/rise-intelligent-conversational-ui/.}

_{[4]. Jassova, B. (2020, February). Conversational AI Statistics: NLP Chatbots in 2020. Retrieved from: https://landbot.io/blog/conversational-ai-statistics/.}

_{[5]. Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549-1565.}

_{[6] Gabrielli, S., Rizzi, S., Bassi, G., Carbone, S., Maimone, R., Marchesoni, M., & Forti, S. (2021). Engagement and Effectiveness of a Healthy-Coping Intervention via Chatbot for University Students During the COVID-19 Pandemic: Mixed Methods Proof-of-Concept Study. JMIR mHealth and uHealth, 9(5), e27965.}

_{[7] Jeong, S., Alghowinem, S., Aymerich-Franch, L., Arias, K., Lapedriza, A., Picard, R., ... & Breazeal, C. (2020, September). A robotic positive psychology coach to improve college students’ wellbeing. In 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) (pp. 187-194). IEEE.}

Results and learnings

Overall Summary

With the growing importance of teamwork in higher education, effective communication, and goal congruence have become vital in improving the effectiveness of student teamwork and collaboration. To this end, within the UNCAGE project, we designed and implemented a novel system that combines a goal-setting chatbot and an effort visualizer to facilitate effective collaboration in student teams. The chatbot guides students to set specific collaborative goals through the SMART framework. At the same time, the effort visualizer displays each member’s contribution, thus increasing accountability among team members and facilitating greater participation. We carried out a controlled study across four experimental conditions in a collaborative creative writing task (N=84) to evaluate the benefits of the collaborative goal-setting chatbot and the effort visualizer. Our results showed that neither the chatbot nor the effort visualizer alone positively impacted student engagement and collaboration. However, when using the chatbot and effort visualizer in combination, we found evidence that suggests improved student engagement and collaboration. This has useful implications for the design of AI tools to support learning activities in education. More information can be found at: https://edu.nl/qpxru. 

Furthermore, to support collaborative goal-setting using conversational agents, we created an open-source agent called ‘GoalKeeper’ which can be integrated into Slack workspaces. Inspired by initiatives in workplace learning, we explored goal-setting practices in crowdsourcing marketplaces and the role of conversational agent representations on user engagement and performance. We explored the potential of leveraging the ‘wisdom of crowds’ to power conversational agents in different contexts such as information retrieval or supporting stress management, while analyzing the tolerance that users have for delays in such real-time contexts. These efforts have led to peer-reviewed scientific articles in premier HCI and AI conferences and journals. 

Relevant Publications

1. Abbas, T., & Gadiraju, U. (2022, October). Goal-Setting Behavior of Workers on Crowdsourcing Platforms: An Exploratory Study on MTurk and Prolific. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing(Vol. 10, No. 1, pp. 2-13).
2. Allen, G., Yang, J., Pera, M. S., & Gadiraju, U. (2021). Using Conversational Artificial Intelligence to Support Children's Search in the Classroom. arXiv preprint arXiv:2112.00076.
3. Abbas, T., Gadiraju, U., Khan, V. J., & Markopoulos, P. (2022). Understanding User Perceptions of Response Delays in Crowd-Powered Conversational Systems. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2), 1-42.
4. Jung, J. Y., Qiu, S., Bozzon, A., & Gadiraju, U. (2022, April). Great chain of agents: The role of metaphorical representation of agents in conversational crowdsourcing. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-22).
5. Jung, J., Murray-Rust, D. S., Gadiraju, U., & Bozzon, A. (2022). Gender Choices of Conversational Agent: How Today’s Practice Can Shape Tomorrow’s Values. Proceedings CHI 2022.
6. Gupta, A., Basu, D., Ghantasala, R., Qiu, S., & Gadiraju, U. (2022, April). To trust or not to trust: How a conversational interface affects trust in a decision support system. InProceedings of the ACM Web Conference 2022 (pp. 3531-3540).