The role of interactions in training with AI chatbots

Enfoque

Introduction. The recent advancements in chatbots, and especially in large language models, have the potential to improve students’ learning by supporting education and training practices. However, their actual deployment as learning tools requires further investigation, in particular relating to the different ways in which interaction with the AI system may affect participants’ learning experience and outcomes.

Objectives. The aim of this explorative study is to investigate, in the context of a single-session chatbot-mediated psychology training, whether learners’ perceptions of the chatbot and of the interaction have an influence on learning outcomes.

Methodology. A sample of 72 students and graduates in Psychology (76% female, mean age: 25±6.6 years) carried out a learning session on a cognitive-behavioural technique (ABC, Antecedent-Behaviour-Consequence) with a chatbot called Serena, and filled out questionnaires measuring both personal factors (alexithymia, metacognitive function) and variables related to the perception of the chatbot and of the interaction (likeability, perceived intelligence, perceived safety, perceived moral agency, perceived trust, positive/negative emotions). In addition, ad hoc questions were administered to gather participants’ feedback on the learning experience, and the conversation logs were reviewed to obtain a hetero-evaluation, on a scale from 1 to 5, of the competence acquired by the participants (as a proxy of the effectiveness of the training with the chatbot). Correlations and hierarchical regressions were used for the analyses.

Results. The analyses revealed two main effects. First, there was a significant correlation between participants’ perception of their knowledge of the psychological technique after the training and the extent in which they believed that Serena helped them learn it; however, both these evaluations did not correlate with the hetero-evaluation of the knowledge actually acquired. Second, participants’ perception of training efficacy was significantly predicted, after controlling for personal factors, by the intelligence attributed to the chatbot, the trust placed in it, and the positive emotions experienced during the interaction; again, however, these variables did not predict actual acquisition of the psychological technique.

Discussion. The results show a discrepancy between the level of effectiveness perceived by the participants and the learning scores assigned by an external review. Even though the actual efficacy of this single-session chatbot-mediated training is to be questioned, participants thought that they learned well because they appreciated and trusted the tool with which they carried out the task.

Conclusions. At the core of education and training does not lie the mere act of delivering knowledge, but the relationship between the trainer and the trainee; this holds especially true in psychological training. AI tools will be able to really improve education if they are used with the right awareness of the interpersonal variables at play, which can impact people’s perceptions of their learning outcomes.