Design and Validation of the Metacognitive Diagnostic Disruption in
AI Decision‐Making Scale ( MDD ‐ AI Scale) Among Educational Administrators in Jordan: Insights From the Network Analysis Perspective
Design and Validation of the Metacognitive Diagnostic Disruption in
AI Decision‐Making Scale ( MDD ‐ AI Scale) Among Educational Administrators in Jordan: Insights From the Network Analysis PerspectiveYusra jadallah abed Khasawneh,N. Khasawneh,M. Khasawneh
TLDR
Overall, the 42‐item MDD‐AI Scale is a valid, reliable, and innovative tool for assessing metacognitive disruptions in algorithmic decision‐making within educational management.
Abstract
In recent decades, emerging technologies—particularly Artificial Intelligence (AI)—have had a profound impact on the structure and decision‐making processes within educational systems. AI‐based tools now play a central role in data analysis, performance prediction, and personalised learning design.
This study aimed to design and validate the Metacognitive Diagnostic Disruption in AI Decision‐Making Scale (MDD‐AI Scale) for educational administrators, focusing on metacognitive disruptions—disturbances in self‐monitoring and self‐regulation processes caused by AI involvement in decision‐making.
Using an exploratory mixed‐methods design, the study combined a literature review and interviews with 20 Jordanian educational administrators to develop 52 items from an initial 92. Data from 670 administrators were analysed for validity and reliability, with exploratory graph analysis (EGA) providing a network‐based perspective.
Exploratory Factor Analysis (EFA) revealed a six‐factor structure—Metacognitive Self‐Monitoring Disruption, Metacognitive Self‐Evaluation Disruption, Cognitive Flexibility Reduction, Cognitive Overreliance on AI, Conflict Detection Shutdown, and Disruption of Metacognitive Decision Planning—explaining 67.6% of the variance. Confirmatory Factor Analysis (CFA) showed good fit (RMSEA = 0.072, CFI = 0.918), with AVE values > 0.50 confirming convergent validity. Reliability was high across all indices (
α = 0.904–0.951; ω = 0.897–0.952; CR > 0.89; ICC = 0.752–0.870). Measurement invariance across gender indicated structural equivalence. Exploratory Graph Analysis (EGA) and Random Forest Modeling (RFM) further supported robustness and predictive utility. Overall, the 42‐item MDD‐AI Scale is a valid, reliable, and innovative tool for assessing metacognitive disruptions in algorithmic decision‐making within educational management.