Tackling Students’ Coding Assignments with LLMs

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F24%3A10485470" target="_blank" >RIV/00216208:11320/24:10485470 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1145/3643795.3648389" target="_blank" >https://doi.org/10.1145/3643795.3648389</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3643795.3648389" target="_blank" >10.1145/3643795.3648389</a>

Result language

angličtina

Original language name

Tackling Students’ Coding Assignments with LLMs

Original language description

State-of-the-art large language models (LLMs) have demonstrated an extraordinary ability to write computer code. This ability can be quite beneficial when integrated into an IDE to assist a programmer with basic coding. On the other hand, it may be misused by computer science students for cheating on coding tests or homework assignments. At present, knowledge about the exact capabilities and limitations of state-of-the-art LLMs is still inadequate. Furthermore, their capabilities have been changing quickly with each new release. In this paper, we present a dataset of 559 programming exercises in 10 programming languages collected from a system for evaluating coding assignments at our university. We have experimented with four well-known LLMs (GPT-3.5, GPT-4, Codey, Code Llama) and asked them to solve these assignments. The evaluation results are intriguing and provide insights into the strengths and weaknesses of the models. In particular, GPT-4 (which performed the best) is currently capable of solving 55% of all our exercises and achieved an average score of 86% on exercises from the introductory programming course (using the best of five generated solutions).

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach<br>I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the 1st International Workshop on Large Language Models for Code

ISBN

979-8-4007-0579-3

ISSN

—

e-ISSN

—

Number of pages

8

Pages from-to

—

Publisher name

Neuveden

Place of publication

Neuveden

Event location

Lisbon, Portugal

Event date

Apr 14, 2024

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—