Assessing the Potential of Large Language Models (LLMs) for the Automatic Generation of Solutions for Programming Exercises

Since several AI chatbots have become available to the general public, the popularity of Large Language Models (LLMs) has risen, especially among students, who have used them as educational support tools. This directly impacts the reproduction of scientific knowledge and the learning process. In this context, it is relevant to evaluate the performance and limitations of these tools in providing correct answers. This article discusses the results of C++ code solutions requested from the Meta Llama 3 model for practical programming tasks of Olimpíada Brasileira de Informática (OBI). It was demonstrated that the method still needs some correctness checker, presenting only 30.4% of correct answers in general. In addition, it was evidenced that the model’s performance deteriorates in questions with images, since the model does not support them as input, presenting a 20.3% drop in correct answers. Finally, the model appears to work better the shorter the question text, with 31.1% more correct answers for questions with up to 200 words compared to questions with more than 400 words.