Evaluation of a Supervised Learning-Based Chatbot: Impact on Satisfaction and Improvement Proposals
DOI:
https://doi.org/10.21556/edutec.2025.92.3849Keywords:
Artificial Intelligence, chatbots, supervised learning, natural language processing, user satisfactionAbstract
In a context where the speed of resolving queries is essential, higher education institutions must rethink their student support models. Technologies such as chatbots, based on natural language processing, provide agile and precise interactions, optimizing university resources. This study outlines the design, implementation, and evaluation of a chatbot trained through supervised learning, using real student FAQs. The Scrum methodology was employed to iterate and adjust the tool based on feedback, with monitoring over a year to enhance its effectiveness. The results show that the chatbot significantly reduces response times and is perceived as useful for basic queries. However, limitations were identified in its ability to handle more complex interactions, emphasizing the need for advanced language models to enable a more natural and adaptive user experience. The study concludes that, while the chatbot is a valuable tool for improving student support, achieving optimal satisfaction requires integrating continuous adjustment processes, such as fine-tuning, and developing more specialized functionalities. These enhancements could transform the use of chatbots into an effective and scalable solution within the educational sector.
Downloads
References
Aleedy, M., Atwell, E., & Meshoul, S. (2022). Using AI chatbots in education: Recent advances challenges and use case. Algorithms for Intelligent Systems (pp. 661-675). https://doi.org/10.1007/978-981-19-1653-3_50 DOI: https://doi.org/10.1007/978-981-19-1653-3_50
Ali, F., Zhang, Q., Tauni, M. Z., & Shahzad, K. (2023). Social chatbot: My friend in my distress. International Journal of Human-Computer Interaction, 1-11. https://doi.org/10.1080/10447318.2022.2150745 DOI: https://doi.org/10.1080/10447318.2022.2150745
Balderas, A., García-Mena, R. F., Huerta, M., Mora, N., & Dodero, J. M. (2023). Chatbot for communicating with university students in emergency situation. Heliyon, 9(9), e19517. https://doi.org/10.1016/j.heliyon.2023.e19517 DOI: https://doi.org/10.1016/j.heliyon.2023.e19517
Belda-Medina, J., & Kokošková, V. (2023). Integrating chatbots in education: Insights from the chatbot-human interaction satisfaction model (CHISM). International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00432-3 DOI: https://doi.org/10.1186/s41239-023-00432-3
Belkacem, A. N., & Hireche, A. (2024). Exploring student engagement and learning preferences: A comparative study between virtual- and robot-based tutors. Measurement Sensors, 101704. https://doi.org/10.1016/j.measen.2024.101704 DOI: https://doi.org/10.1016/j.measen.2024.101704
Caldarini, G., Jaf, S., & McGarry, K. (2022). A literature survey of recent advances in chatbots. Information, 13(1), 41. https://doi.org/10.3390/info13010041 DOI: https://doi.org/10.3390/info13010041
Chaves, A. P., & Gerosa, M. A. (2020). How should my chatbot interact? A survey on social characteristics in human–chatbot interaction design. International Journal of Human-Computer Interaction, 37(8), 729-758. https://doi.org/10.1080/10447318.2020.1841438 DOI: https://doi.org/10.1080/10447318.2020.1841438
Croes, E., & Antheunis, M. L. (2022). "I am in your computer while we talk to each other" a content analysis on the use of language-based strategies by humans and a social chatbot in initial human-chatbot interactions. International Journal of Human-Computer Interaction, 39(10), 2155-2173. https://doi.org/10.1080/10447318.2022.2075574 DOI: https://doi.org/10.1080/10447318.2022.2075574
García-Peñalvo, F. J. (2023). La percepción de la inteligencia artificial en contextos educativos tras el lanzamiento de ChatGPT: Disrupción o pánico. Education in the Knowledge Society, 24, e31279. https://doi.org/10.14201/eks.31279 DOI: https://doi.org/10.14201/eks.31279
Haqimi, N. A., & Kusuma, R. K. (2023). Timeline reminder system bot and telegram assistant chatbot for a university student and lecturer. Journal of Soft Computing Exploration, 4(4), 186-194. https://doi.org/10.52465/joscex.v4i4.221 DOI: https://doi.org/10.52465/joscex.v4i4.221
Huang, W., Hew, K. F., & Fryer, L. K. (2021). Chatbots for language learning—Are they really useful? A systematic review of chatbot‐supported language learning. Journal of Computer Assisted Learning, 38(1), 237-257. https://doi.org/10.1111/jcal.12610 DOI: https://doi.org/10.1111/jcal.12610
Ifelebuegu, A., Kulume, P., & Cherukut, P. (2023). Chatbots and AI in education (AIEd) tools: The good, the bad, and the ugly. Journal of Applied Learning and Teaching, 6(2). https://doi.org/10.37074/jalt.2023.6.2.29 DOI: https://doi.org/10.37074/jalt.2023.6.2.29
Isa, N. A. N. M., Jawaddi, S. N. A., & Ismail, A. (2024). Experimental evaluation of machine learning models for goal-oriented customer service chatbot with pipeline architecture. arXiv preprint. https://doi.org/10.48550/arXiv.2409.18568 DOI: https://doi.org/10.21203/rs.3.rs-4532068/v1
Keerthana, R. R., Fathima, G., & Florence, L. (2021, May). Evaluating the performance of various deep reinforcement learning algorithms for a conversational chatbot. In 2021 2nd International Conference for Emerging Technology (INCET) (pp. 1-8). IEEE. https://doi.org/10.1109/INCET51464.2021.9456321 DOI: https://doi.org/10.1109/INCET51464.2021.9456321
Labadze, L., Grigolia, M., & Machaidze, L. (2023). Role of AI chatbots in education: Systematic literature review. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00426-1 DOI: https://doi.org/10.1186/s41239-023-00426-1
Latif, E., & Zhai, X. (2024). Fine-tuning ChatGPT for automatic scoring. Computers and Education: Artificial Intelligence, 6, 100210. https://doi.org/10.1016/j.caeai.2024.100210 DOI: https://doi.org/10.1016/j.caeai.2024.100210
Lin, C., Huang, A. Y., & Yang, S. J. (2023). A review of AI-driven conversational chatbots implementation methodologies and challenges (1999–2022). Sustainability, 15(5), 4012. https://doi.org/10.3390/su15054012 DOI: https://doi.org/10.3390/su15054012
Medina, G. T., Lengua, M. A. C., & Medrano, H. V. (2024). Design of a chatbot in a mobile application for managing payments and controlling activities in a fast school organization. Indonesian Journal of Electrical Engineering and Computer Science, 35(2), 1271. https://doi.org/10.11591/ijeecs.v35.i2.pp1271-1286 DOI: https://doi.org/10.11591/ijeecs.v35.i2.pp1271-1286
Mekić, E., Jovanović, M., Kuk, K., Prlinčević, B., & Savić, A. (2024). Enhancing educational efficiency: Generative AI chatbots and DevOps in education 4.0. arXiv preprint. https://doi.org/10.48550/arXiv.2406.15382 DOI: https://doi.org/10.1002/cae.22804
Mou, Y., & Xu, K. (2017). The media inequality: Comparing the initial human-human and human-AI social interactions. Computers in Human Behavior, 72, 432-440. https://doi.org/10.1016/j.chb.2017.02.067 DOI: https://doi.org/10.1016/j.chb.2017.02.067
Nguyen, Q. N., Sidorova, A., & Torres, R. (2021). User interactions with chatbot interfaces vs. menu-based interfaces: An empirical study. Computers in Human Behavior, 128, 107093. https://doi.org/10.1016/j.chb.2021.107093 DOI: https://doi.org/10.1016/j.chb.2021.107093
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033 DOI: https://doi.org/10.1016/j.caeai.2021.100033
Pawlik, Ł., Płaza, M., Deniziak, S., & Boksa, E. (2022). A method for improving bot effectiveness by recognising implicit customer intent in contact centre conversations. Speech Communication, 143, 33-45. https://doi.org/10.1016/j.specom.2022.07.003 DOI: https://doi.org/10.1016/j.specom.2022.07.003
Pratama, Y. A., & Kristiana, T. (2023). Design of cloud-based chatbot application at Pt. Traveloka Singapore using the agile method. Jurnal Pilar Nusa Mandiri, 19(1), 19-26. https://doi.org/10.33480/pilar.v19i1.3055 DOI: https://doi.org/10.33480/pilar.v19i1.3055
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
Rojas, E. M. (2020). Machine learning: Análisis de lenguajes de programación y herramientas para desarrollo. Revista Ibérica de Sistemas e Tecnologias de Informação, 28, 586–599. https://www.proquest.com/docview/2388304894/1866F7F83224F28PQ/3?sourcetype=Scholarly%20Journals
Segovia-García, N., & Said-Hung, E. M. (2021). Factores de satisfacción de los alumnos en e-learning en Colombia. Revista Mexicana de Investigación Educativa, 26(89), 595-621. https://ojs.rmie.mx/index.php/rmie/article/view/186
Segovia-García, N. (2023). Percepción y uso de los chatbots entre estudiantes de posgrado online: Un estudio exploratorio. Revista de Investigación en Educación, 21(3), 335-349. https://doi.org/10.35869/reined.v21i3.4974 DOI: https://doi.org/10.35869/reined.v21i3.4974
Segovia-García, N., & Segovia-García, L. (2024). Chatbot de IA para prevenir el abandono de la educación superior: Una revisión de la literatura. Education in the Knowledge Society, 25, e31416. https://doi.org/10.14201/eks.31416 DOI: https://doi.org/10.14201/eks.31416
Sharma, V., Goyal, M., & Malik, D. (2017). An intelligent behaviour shown by chatbot system. International Journal of New Technology and Research, 3(4), 52-54. https://media.neliti.com/media/publications/263312-an-intelligent-behaviour-shown-by-chatbo-7020467d.pdf
Skuridin, A., & Wynn, M. (2024). Chatbot design and implementation: Towards an operational model for chatbots. Information, 15(4), 226. https://doi.org/10.3390/info15040226 DOI: https://doi.org/10.3390/info15040226
Stokel-Walker, C. (2022). AI bot ChatGPT writes smart essays — should professors worry? Nature. https://doi.org/10.1038/d41586-022-04397-7 DOI: https://doi.org/10.1038/d41586-022-04397-7
Sudiatmika, I. P. G. A., & Ariantini, M. (2021). Digital marketing chatbot using API dialog flow case study ITB Stikom Bali, Jimbaran campus. https://doi.org/10.4108/eai.19-12-2020.2309122 DOI: https://doi.org/10.4108/eai.19-12-2020.2309122
Tabone, W., & De Winter, J. (2023). Using ChatGPT for human–computer interaction research: A primer. Royal Society Open Science, 10(9). https://doi.org/10.1098/rsos.231053 DOI: https://doi.org/10.1098/rsos.231053
Torres Martínez, D. M., & Cruz Guerrero, S. A. (2020). ¿Qué tipos de agentes virtuales pueden usar las pequeñas empresas para mejorar su publicidad? Revista Vínculos, 17(2), 140-155. https://doi.org/10.14483/2322939X.16874 DOI: https://doi.org/10.14483/2322939X.16874
Uc-Cetina, V., Navarro-Guerrero, N., Martín-González, A., Weber, C., & Wermter, S. (2022). Survey on reinforcement learning for language processing. Artificial Intelligence Review, 56(2), 1543-1575. https://doi.org/10.1007/s10462-022-10205-5 DOI: https://doi.org/10.1007/s10462-022-10205-5
Zhang, R., Zou, D., & Cheng, G. (2023). A review of chatbot-assisted learning: Pedagogical approaches, implementations, factors leading to effectiveness, theories, and future directions. Interactive Learning Environments, 32(8), 4529–4557. https://doi.org/10.1080/10494820.2023.2202704 DOI: https://doi.org/10.1080/10494820.2023.2202704
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Edutec, Revista Electrónica de Tecnología Educativa
This work is licensed under a Creative Commons Attribution 4.0 International License.
By submitting the paper, the authors assign the publication rights to the journal Edutec. For its part, Edutec authorises its distribution as long as its content is not altered and its origin is indicated. At the end of each article published in Edutec, the citation procedure is indicated.
The management and editorial board of Edutec Revista Electrónica de Tecnología Educativa do not accept any responsibility for the statements and ideas expressed by the authors in their work.
Translated with www.DeepL.com/Translator (free version)
