Criterion-based assessment of reading literacy with AI

This study explores the potential of artificial intelligence (AI) to generate criteria for criterion-based assessment of pupils' reading literacy. The main objective was to design and examine a structured set of prompts enabling an AI chatbot to produce assessment criteria that are relevant to the educational framework and judged by experts to be content-aligned, comprehensible, and didactically usable.

The study used a design-based, mixed-methods approach. A prompt engineering procedure was developed by integrating theoretical foundations of reading literacy, international literacy frameworks, and principles of criterion-referenced assessment. AI-generated criteria were created for reading-literacy milestones across four domains and evaluated by a panel of five experts using a predefined four-point rubric and accompanying qualitative comments.

The findings showed a high level of expert agreement with the AI-generated criteria. Overall, approximately 82% of the AI-generated criteria were rated as suitable or completely suitable. The highest agreement was found for comprehension and interpretation and relationship to reading, while lower agreement was observed for assessment of content and form, where the AI more often produced overly general formulations.

The results suggest that, when guided by precise prompt engineering, AI can serve as a useful assistive tool for generating draft assessment criteria that resemble expert proposals. However, the study did not test whether AI-generated and expert-generated rubrics produce comparable scoring decisions when applied to the same samples of student work. Therefore, AI-generated criteria should be used with expert review, pedagogical mediation, and local calibration.