The Risk of AI chatbots in mHealth apps

Guest author: Teun Voost

AI-Bots in Healthcare

AI is, as it has been for many years, a hot topic of discussion. Both positive and negative consequences of use have been part of the discourse of AI (Coeckelbergh, 2020). The potential of AI is seen by many people in various fields, eagerly trying to find problems where the AI-solution could provide useful (AI for climate change solutions: Cowls et al., 2023; AI for security: Bertino et al., 2021). One such implementations that has developed in recent years is the implementation of AI in mental mobile healthcare technology (mHealth). It is since recent years that AI-bots are increasingly used to guide users through mental difficulties in mHealth apps such as Wysa and Replika (Ozolcer et al., 2024). Studies on user experience shows that users can derive significant positive experiences from interacting with AI-bots.

“The app [Youper] has helped me with calming my nerves and destroying the irrational thoughts I often have. It makes you feel better about yourself and brings support and happiness at time, I would suggest it.” (Ozolcer et al., 2024, p. 11)

Source

“First things off, everyone needs someone to talk to once in a while […]. Having a talk with Wysa [-the chatbot-] helps me to get calm, relax, organize my thoughts, uplifted, grateful – All the great feelings are getting back. (Ozolcer et al., 2024, p. 11)

Source

These statements give an impression of how users experience the improvement of their general mental health. But let’s say that the AI-bot gives an undesirably output. It gives the user an undesirable piece of advice. How can we look at those moments of risk of harm, and how do we reconcile those risks with possible benefits that users can experience?

Possibility of Risk

It might sound weird to think of chatbots that are designed to help people with mental distress as potential harms to user well-being. Aren’t they designed to do the exact opposite of harming the user? Well, the fact that this technology is designed for one purpose does not rule out possible (negative) consequences. Where the use of apps for mental health was already not without controversy (e.g., researchers found an app, aimed at helping people cope with bipolar disorder, that suggested people to drink alcohol during manic episodes; Nicholas et al., 2015) recently, some horror stories about human-chatbot interactions have surfaced. In one case, a young boy fell in love with the chatbot that he was interacting with, expressing that he “could not go a day without talking to [the chatbot]” (Independent, 2024). The boy’s mother sued the company that made the AI after her son took his life, minutes after an interaction with the chatbot.

This is an example of how the output of the chatbot can have devastating consequences. The human-chatbot interactions are not the same as human-human interactions. It cannot understand someone’s emotions as it does not understand or emote. Anthropomorphizing the chatbot, where the user (a human) sees the chatbot (a non-human) as human, then becomes a risk (Deshpande, 2023). Consequently, the user can start to feel that a relationship is being formed between them and the chatbot.

However, there are more risks in the human-chatbot interaction. For example, only last year a chatbot (Eliza) suggested to a user suffering from severe eco-anxiety, that he should take his own life (Euronews, 2023). Regardless of the logic that Eliza utilized in this instance, should we accept that there is the possibility of harmful suggestions given by chatbots? In the following section I suggest an approach towards preventing this harm through the Precautionary Principle.

The Precautionary Principle applied

The Precautionary Principle (PP) is a principle that can be implemented when caution is warranted. In the literature there has been some debate on the definition and the criteria for the implementation of this principle (Gardiner, 2006). The PP I present here can be understood as a Maximin principle, “The phrase “maximin” means “maximize the minimum.” Maximin principles assess the possible outcomes of various courses of action and then decide what to do by focusing on the worst possible outcome of each course of action and choosing that action which has the least bad worst outcome.” (Gardiner, 2006, p. 45). This, however, can quickly become irrational, as sometimes an action with possible worse outcomes needs to be preferred over safer options (e.g., “you could not ever cross the street (after all, you might be hit by a car) (Harsanyi, 1975, p. 39)).

Rawls (1999) consequently counters such objections by providing three general circumstances under which the maximin principle should be applied:

Decision makers either lack, or have reason to sharply discount information about the probabilities of the possible outcomes of their actions.
Decision makers care relatively little for potential gains that might be made above the minimum that can be guaranteed by the maximin approach.
Decision makers face unacceptable alternatives (Gardiner, 2006, p. 47).

Now, let us review the AI-chatbot technology through this framework:

Firstly, AI algorithms are notorious for their impenetrability. As Rawls would say, “A knowledge of likelihoods is impossible” (Rawls, 1999, p. 134). As a knowledge of likelihood of the AI-chatbot saying something potentially distressing or disastrous to the user is obscured by the black box of the algorithm, caution seems to get the upper hand here.
The second element is a bit trickier, as there are advantages to be gained by using this chatbot, as evidenced by the user statements in the first section of this text. With the possibility of good in mind, would people leave this option behind for the potential risks that might ensue? I think people care for the potential gains that might be made above the minimum that can be guaranteed by the maximin approach. A maximin approach at this stage would probably suggest avoiding using the chatbot, as there is a risk of severe harm. Furthermore, this risk is not that obvious. It is not as if the chatbot is able to rise up, and give you a lethal shock! Could part of the acceptance of these risks come from the idea that people think that they would not be as easily influenced to take desperate measures?
The third element talks of “unacceptable alternatives” (Gardiner, 2006, p. 47), which is relevant in these cases. Even though there are benefits that can be obtained in the human-chatbot interaction for well-being, it has become clear that unacceptable alternatives are part of the discussion. The reality is that people have died most probably from the interaction that they had with the chatbot. And it is exactly this horrifying fact that requires us to take the PP seriously.

Main Takeaway

Taking stock of the situation, the full application of the Maximin Principle as suggested might go a bit far. The possibility for good can be significant in this case, albeit that the possibility of bad things happening is looming, as uncertainty about possible outputs from the chatbot is relatively high. While a Maximin approach, such as stopping the use of human-chatbot interaction might go a bit far, a restriction in the influence of the chatbot might not be such a far-fetched idea.

References

Bertino, E., Kantarcioglu, M., Akcora, C. G., Samtani, S., Mittal, S., & Gupta, M. (2021). AI

for Security and Security for AI. In Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy (pp. 333-334).

Coeckelbergh, M. (2020). AI ethics. The MIT Press.

Cowls, J., Tsamados, A., Taddeo, M., & Floridi, L. (2023). The AI gambit: leveraging artificial intelligence to combat climate change—opportunities, challenges, and recommendations. Ai & Society, 1-25.

Deshpande, A., Rajpurohit, T., Narasimhan, K., & Kalyan, A. (2023). Anthropomorphization of AI: opportunities and risks. arXiv preprint arXiv:2305.14784.

Independent. (2024). Teenager took his own life after falling in love with AI chatbot. Now his devastated mom is suing the creators. https://www.independent.co.uk/news/world/americas/crime/character-ai-suicide-lawsuit-sewell-setzer-iii-death-b2634706.html

Euronews. (2023). Man ends his life after an AI chatbot 'encouraged' him to sacrifice himselfto stop climate change. https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate-

Ozolcer, M., Yang, Y., Kate, S., Samant, P., & Bae, S. W. (2024). Toward Enhancing User Experience and Adoption of AI Chatbot Apps for Mental Health: Insights from Natural Language Processing of User Reviews. In Conference on Systems Engineering Research (pp. 569-586). Cham: Springer Nature Switzerland.

Nicholas, J., Larsen, M. E., Proudfoot, J., & Christensen, H. (2015). Mobile apps for bipolar disorder: a systematic review of features and content quality. Journal of medical Internet research, 17(8), e198.

About the author

Teun Voost is a Master student at the University of Twente, in the Master programme Philosophy of Science, Technology and Society. In this Master programme he is following the 4TU Ethics and Technology track. His main areas of interest are Philosophy of Technology, Philosophical Anthropology, Philosophy of Psychology, Philosophy of AI, Psychoanalysis and Clinical & Narrative Psychology. He is currently writing his master's thesis on the relation between Mental Self-Help technologies and Psychoanalysis.