Unveiling the Impact of Large Language Models on Student Learning: A Comprehensive Case Study

Large language models (LLMs) have achieved planetary popularity and have become accepted in higher education. On the basis of face-to-face interviews, a survey examining students' attitudes about the integration of LLM into education, and our own academic experience, we defined a realistic solution for creating assignments. It embraces essay writing as well as various aspects of computer programming. The experiments were carried out during the winter semester of academic 2023/24 at two universities from two different countries. This paper presents the experience gained in the creation of two different computer science assignments with and without the use of LLM. Comparative analysis refers on three approaches: traditional or manual assignment preparation without using any LLM; full reliance on LLMs; and a hybrid mode, depending on the amount of application of the LLM in the preparation of the assignment. The proposed solution was evaluated quantitatively, with the aim of becoming a benchmark for examining the integration of LLM studies into higher education. Findings reveal the importance of hybrid mode, as the most preferred approach among students.