A Framework for Assessing AI-Based Personalised Learning in Classrooms

Thesis Abstract

This study investigates how artificial intelligence (AI)-based personalised learning (PL) interventions influence student learning outcomes, engagement, and equity in classroom settings, addressing the gap between theoretical potential and practical classroom implementation. Despite widespread deployment of adaptive feedback and data-driven pacing in educational technology, there is limited empirical consensus on the validity, reliability, and transferability of AI-driven PL systems across diverse learners and curricular contexts. The aim is to develop a rigorous framework for evaluating AI-based PL in real classrooms, encompassing effectiveness, fairness, and pedagogical alignment. Specific objectives are (1) to identify dimensions of AI-based PL that affect learning gains; (2) to design a multi-criteria assessment framework integrating cognitive outcomes, motivational processes, and equity considerations; (3) to test the framework through empirical data from heterogeneous classrooms; (4) to examine interactions between AI recommendations and teacher facilitation; and (5) to provide actionable guidelines for implementing AI-based PL with fidelity. A mixed-methods research design is employed, combining quasi-experimental and explanatory sequential components. The study is conducted in three public secondary schools within an urban district, comprising grades 9–10, with an estimated population of 1,200 students. A stratified random sample of 600 students is selected, assigning 300 to AI-based PL instruction and 300 to a control condition using conventional adaptive learning tools without AI-driven personalisation. Data collection instruments include standardized test scores in mathematics and science (pre- and post-tests), validated engagement scales (short-form, classroom-level adaptation), detailed system-logged interaction data (time-on-task, sequence of content, hint usage), and teacher interview protocols. Instrument validity and reliability are established through pilot testing (n=60) and Cronbach’s alpha coefficients above 0.80 for all scales. Additional data comprise classroom observations guided by a structured rubric to assess pedagogical alignment, computational fairness indicators, and teacher-student interactions. Data analysis uses a combination of regression analysis to estimate learning gains attributable to AI-based PL, multilevel modelling to account for nested data (students within classes within schools), and thematic analysis of teacher interviews to elucidate contextual factors influencing implementation fidelity. The analytical framework also incorporates fairness auditing methods, examining differential item functioning and group-based performance disparities across gender and socio-economic status. Model specification includes a hierarchical linear model with post-test scores as the dependent variable, fixed effects for treatment, prior achievement, gender, SES, and random effects for class and school. Mediation analyses explore whether engagement mediates the relationship between AI-based PL exposure and achievement. Robustness checks employ propensity score matching to mitigate selection bias and sensitivity analyses to assess the impact of missing data. Key expected findings include (i) statistically significant improvements in subject-specific achievement for students exposed to AI-based PL relative to controls, with effect sizes in the small-to-moderate range (Cohen’s d ? 0.25–0.40); (ii) higher engagement indicators and time-on-task in the AI condition, partially mediated by tailored feedback and adaptive pacing; (iii) positive moderation effects of teacher facilitation quality, with greater gains when teachers align AI recommendations to instructional goals; (iv) evidence on equity considerations, identifying areas where AI-based PL reduces achievement gaps for lower- SES students but also potential risk of amplification if access to devices is uneven. The study contributes to knowledge by operationalising a comprehensive, generalisable framework for evaluating AI-based PL that integrates learning analytics, pedagogy, and fairness considerations, and by providing empirical benchmarks for classroom-level impact. The main conclusion anticipates that AI-based personalisation can enhance learning outcomes and engagement when implemented with robust pedagogical alignment, transparent system design, and professional development for teachers. Recommendations include (a) establishing clear instructional objectives aligned with AI-driven nudges; (b) investing in professional development to support teachers in interpreting AI feedback; (c) implementing continuous fairness audits and device access mitigation; and (d) scaling the framework across varied subjects and school contexts to refine generalisability.