AI IN WRITING ASSESSMENT: HOW RELIABLE AND CONSISTENT ARE THE SCORING AND FEEDBACK?

This study investigates the consistency and usefulness of AI-generated scores and feedback in writing assessments using ChatGPT. Employing a mixed-methods approach, 23 student texts were evaluated across three rounds. Quantitative results showed excellent score consistency, with an Intraclass Correlation Coefficient (ICC) of 0.93, indicating high reliability. Qualitative analysis revealed that ChatGPT consistently addressed five writing criteria—content, organization, vocabulary, language use, and mechanics. However, the specific focus of feedback varied between rounds, and the absence of reference to previous suggestions limited its effectiveness in supporting revision as a recursive process. While ChatGPT provides reliable scoring and thematically consistent feedback, greater continuity and follow-up are needed to enhance its pedagogical impact in writing instruction.