This article features results from a research project aimed at assessing the use, satisfaction and perceived effectiveness of a digital application, featuring a conversational robot and a toolbox, designed to provide mental health and academic support to college students. 

For further details, please consult: Gallais, B. & Desjardins, F. (2022). Analyse logique du déploiement d’un agent conversationnel destiné au soutien scolaire : tome 1. Satisfaction et efficacité perçue d’un agent conversationnel destiné au soutien de la santé mentale et au soutien scolaire chez les étudiantes et étudiants collégiaux. ÉCOBES – Recherche et transfert, Cégep de Jonquière.

Background

Why use Digital Applications to Provide Emotional and Academic Support for College Students?

Lacking sufficient resources, access to mental health services in colleges is often limited in relation to demand. This situation arises at a time when students are faced with a number of challenges, including the context of study and a developmental period of great need: the transition from adolescence to adulthood (Gallais, Bikie Bi Nguema & Turcotte, 2021). Requests for help from psychosocial services have increased dramatically in recent decades (Center for Collegiate Mental Health, 2020). The COVID-19 pandemic, which significantly deteriorated the mental health of college student populations, exacerbated this problem (Gallais, Blackburn, Paré, Maltais & Brassard, 2022).

As a result, a number of experts believe that the integration of technology in mental health could allow meeting the needs of the population. For example, conversational applications and agents have several attractive features: they are accessible at all times (Kretzschmar, Tyroll, Pavarini, Manzini, Singh & NeurOx Young People’s Advisory Group, 2019), easy to access and preserve the anonymity of the user (Abd-Alrazaq, Alajlani, Ali, Denecke, Bewick & Househ, 2021), appeal to young people and would overcome the stigma barrier (Fitzpatrick, Darcy & Vierhile, 2017). While there is a multitude of similar applications, it is essential that each application dedicated to mental health be evaluated to determine its effectiveness with the target clientele (Gamble, 2020; Palmer & Burrows, 2021).

Key Concept

Chatbot or Conversational Agent

Chatbots, also known as conversational agents/robots, can provide a conversation programmed through questions and answer choices, or generate a dialogue using artificial intelligence (Vaidyam, Wisniewski, Halamka, Kashavan & Torous, 2019). The conversational agent studied in our research was programmed to promote well-being and mental health, and to prevent psychosocial disorders through rapid awareness of the student’s difficulties (Optania, n.d.). This also includes facilitating referrals to professional staff, thereby allowing for early intervention.

Methodology

An Online Survey of CEGEP Students   

  • Type of research: Quantitative
  • Study population: 648 students
  • Place and period of research: 2 CEGEPs, between January and March 2022

We conducted a descriptive survey study of students in two CEGEPs that had made the Ali application available1. This application features a conversational agent in the form of an avatar (Ali) and a toolbox containing useful information and tools.

The survey included the evaluation of the following six dimensions: socio-demographic characteristics, downloading and general appreciation of the application, topics addressed with the chatbot and its appreciation, the toolbox and its appreciation, the perceived effects of using the application and, finally, the analysis of the request for human assistance.

1Ali is an application created by Optania. It should be noted that the research team behind this study has no commercial interest in Optania or its product Ali.

Results

Use and Overall Satisfaction with the Application

The application was relatively unknown to students.

Of the 648 participants, only a quarter (26%) had downloaded the application. The most frequently cited reason for not having downloaded it is unfamiliarity with the application (74%). Conversely, those who had downloaded the application had mainly heard about it through CEGEP advertising (42%).

The application was relatively well appreciated by users.

Analysis of the data shows that the application was generally well received. For example, almost all respondents thought the application was easy to use (98%), flexible (92%) and useful (86%).

The topics most frequently discussed with the conversational agent were stress (and anxiety) by 68% of participants, sleep (37%) and loneliness (25%) (topics highlighted in green in Figure 1). These topics were deemed useful by 84% to 86% of participants. To a lesser extent (21%), we found topics linked to academic support (“effective learning” and “distance learning”) (topics highlighted in gray in Figure 1).

Here is a more detailed breakdown of the 18 topics discussed by survey participants.

Figure 1. The 18 Topics Discussed by Survey Participants Figure 1 Title: The 18 Topics Discussed by Survey Participants Graph with two columns. The first column features the 18 topics, while the second column shows the percentage of people who have discussed the topic. Here are the 18 topics and their corresponding percentages: 1. Stress / Anxiety with 68%. This is one of the most frequently discussed topics with the conversational agent. 2. Sleep with 37%. This is one of the most frequently discussed topics with the conversational agent. 3. Loneliness with 25%. This is one of the most frequently discussed topics with the conversational agent. 4. Effective learning with 21%. This is an important topic for academic support. 5. Emotions with 21% 6. Distance learning with 21%. This is an important topic for academic support. 7. Relationships with 19% 8. Depression with 17% 9. Finance with 17% 10. Pandemic with 16% 11. Health with 11% 12. Self-esteem with 11% 13. Oral presentation with 9% 14. Employment with 17% 15. Adaptation with 6% 16. Breakup with 5% 17. Teamwork with 5% 18. Bereavement with 3% The source of the data used for this figure is the following research report: Gallais, B. & Desjardins, F. (2022). Analyse logique du déploiement d’un agent conversationnel destiné au soutien scolaire : tome 1. Satisfaction et efficacité perçue d’un agent conversationnel destiné au soutien de la santé mentale et au soutien scolaire chez les étudiantes et étudiants collégiaux. ÉCOBES – Recherche et transfert, Cégep de Jonquière.

Concerning the application’s toolbox, users appreciated its usefulness (90%) and found the information it contained satisfactory (87%) and relevant (91%).

Perceived Effectiveness as a Pedagogical, Psychosocial and Help-Seeking Support Tool

Satisfaction was high when it came to academic support.

Overall, participants agreed that they felt more pleasure and satisfaction (74%) and were more motivated (75%) to learn new knowledge in courses since using the application.

Perceived effectiveness of the tool on mental health was more moderate.

When asked whether they felt less stressed and nervous or, on the contrary, more zen and relaxed since using the application, more than one in three respondents disagreed, although a majority agreed.

From the participants’ responses, the tool seemed to be appreciated and was thought to be useful for the college community. However, the tool’s effectiveness concerning academic support seemed to be greater than for mental health.

What Can we Learn from our Results?

Is the Application a Front-Line Tool for Reducing Wait Times for Support Services in CEGEPs?

Robots cannot replace human support workers.

The massive appearance of applications and robots can give rise to a number of prejudices. However, at present, and specifically in the application assessed in the case at hand, the conversational robot’s function is not to replace support services provided by humans. Rather, it plays a metacognitive role, helping individuals to become aware of their difficulties. 

The application is a reliable source of information on mental health and educational success.

In an age of information overload on the Internet and social networks, an application for students that brings together information relevant to mental health and educational success (the toolbox) certainly has a place in today’s landscape.

Digital tools can be seen as future allies for college staff.

Support workers and teachers in CEGEPs and colleges could use this kind of application as a promotion and prevention tool for academic success, and make it a strategic complement to their own tasks and missions. Referring students to this kind of tool could help defuse certain situations and be sufficient for people who above all need to be reassured and informed (Gallais, Bikie Bi Nguema & Turcotte, 2021). However, support services must be and remain available to the student population when they feel the need for additional assistance.

Courses of Action

  • Implement winning strategies for advertising (e.g. social networks) and promoting the application in college institutions that offer this tool to their students.
  • Use the application as a tool to promote support services to students.
  • Use the conversational agent as a tool to support mental health literacy as well as to develop knowledge and different skills on the topic of mental health.
  • Promote the toolbox as a “safe” place to find information and tips.

Lines of Research

  • Replicate this survey on a larger scale (e.g. with all college institutions now using the application).
  • Refine results on user perceptions through a qualitative study.
  • Directly assess the application’s effects on mental health prevention and interventions through a study that would directly measure certain variables (e.g. stress levels, anxiety, depression, self-esteem) before and after using the application.

For Further Reading 

Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick B. M. & Househ, M. (2020). Effectiveness and Safety of Using Chatbots to Improve Mental Health: Systematic Review and Meta-Analysis. Journal of Medical Internet Research, 22(7).

Desjardins, F. & Gallais, B. (2022). Analyse logique du déploiement d’un agent conversationnel destiné au soutien scolaire : tome 2. Éléments favorables à l’intégration d’un agent conversationnel au sein des établissements collégiaux. ÉCOBES – Recherche et transfert, Cégep de Jonquière.

Nadarzynski, T., Miles, O., Cowie, A. & Ridge, D. (2019). Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: A mixed-methods study. DIGITAL HEALTH, 5, 1-12.

References

Abd-Alrazaq, A. A., Alajlani, M., Ali, N., Denecke, K., Bewick, B. M. & Househ, M. (2021). Perceptions and Opinions of Patients About Mental Health Chatbots: Scoping Review. Journal of Medical Internet Research, 23(1).

Center for Collegiate Mental Health. (2020). 2019 Annual report (publication no STA 20‑244). 

Fitzpatrick, K. K., Darcy, A. & Vierhile, M. (2017). Delivering Cognitive Behavior Therapy to Young Adults with Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial. JMIR Mental Health, 4(2).

Gallais, B., Bikie Bi Nguema, N. & Turcotte, A. (2021). Portrait de l’intervention psychosociale dans le réseau collégial québécois : point de vue des intervenants sur leur pratique, leurs collaborations et sur les programmes d’accompagnement psychologique existants. ÉCOBES – Recherche et transfert, Cégep de Jonquière.

Gallais, B., Blackburn, M.-È., Paré, J., Maltais, A. & Brassard, H. (2022). Adaptation psychologique et adaptation aux études à distance des étudiants collégiaux face à la crise de la COVID-19. ÉCOBES – Recherche et transfert, Cégep de Jonquière.

Gamble, A. (2020). Artificial Intelligence and Mobile Apps for Mental Healthcare: A social Informatics Perspective. Aslib Journal of Information Management, 72(4), 509-523.

Kretzschmar, K., Tyroll, H., Pavarini, G., Manzini, A., Singh, I. & NeurOx Young People’s Advisory Group. (2019). Can Your Phone Be Your Therapist? Young People’s Ethical Perspectives on the Use of Fully Automated Conversational Agents (Chatbots) in Mental Health Support. Biomedical Informatics Insights, 11.

Optania (n.d.). Ali, une mesure de soutien supplémentaire.

Palmer, K. M. & Burrows, V. (2021). Ethical and Safety Concerns Regarding the Use of Mental Health–Related Apps in Counseling: Considerations for Counselors. Journal of Technology in Behavioral Science, 6(1), 137-150.

Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S. & Torous, J. B. (2019). Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape. The Canadian Journal of Psychiatry, 64(7), 456-464.

Statement of responsibility

Editor: Karine Vieux-Fort  

Editorial Committee: Karine Vieux-Fort, Anouk Lavoie-Isebaert and Amélie Descheneau-Guay  

Linguistic Review: Sandrine Bourget-Lapointe  

This article is licensed under the Creative Commons BY-NC-SA 4.0 license.

ISSN 2817-2817

This document was produced with the financial support of the gouvernement du Québec, under the Canada-Québec Agreement.