Designing AI Tutors to Enhance Science Concept Mastery in Classrooms

Thesis Abstract

The rapid integration of artificial intelligence (AI) in classroom settings offers the potential to personalize science instruction, yet existing implementations often lack rigorous evaluation of their impact on core science concept mastery across diverse learner populations. This study addresses the problem of inconsistent learning gains when AI tutors are deployed in science classrooms and investigates whether adaptive AI tutoring can reliably enhance students’ mastery of foundational science concepts, namely forces and motion, matter and energy, and ecosystems, through targeted, scaffolded interventions. The aim is to determine the effectiveness of an AI tutor in improving conceptual understanding, engagement, and progression toward scientifically literate thinking. Specific objectives include (1) to design an AI tutoring system that adaptiveizes content according to students’ misconceptions and prior knowledge; (2) to evaluate its impact on concept mastery using standardized pre- and post-tests aligned with national science standards; (3) to examine changes in student engagement and time-on-task; (4) to explore teachers’ perceived feasibility and instructional integration of AI tutoring; and (5) to identify differential effects across gender, grade level, and baseline achievement. The study adopts a quasi-experimental mixed-methods design conducted in six public middle schools over an academic semester. A total of 1,200 students in seventh and eighth grades will be selected, with 600 students assigned to the AI-tutor intervention group and 600 to traditional instruction as a control group, matched on prior achievement and school SES indicators. Data collection instruments include (a) a validated science concept mastery assessment administered at pre-test and post-test, (b) the AI-tutor system logs detailing time-on-task, hint usage, and misconception repair sequences, (c) a student engagement questionnaire adapted from established scales, and (d) semi-structured teacher interviews and classroom observation rubrics. To establish validity and reliability, the mastery assessment will undergo item analysis (point-biserial correlations, item difficulty indices) and a Cronbach’s alpha check (>0.80). The study will employ a mixed-methods analytic framework quantitative data will be analyzed via ANCOVA to compare post-test scores between groups while controlling for pre-test scores, multilevel modeling to account for nesting within classrooms, and regression analyses to identify predictors of gains (e.g., tutoring usage patterns, time-on-task). Qualitative data from interviews and observations will be analyzed using thematic analysis to elucidate moderators of effect such as instructional fidelity, teacher-student interactions, and perceived affordances of the AI tutor. The theoretical basis integrates constructivist learning theory and the ICAP (Interactive, Constructive, Active, Passive) framework, with the AI tutor designed to promote constructive and interactive engagement through adaptive feedback, Socratic questioning, and concept-repair scaffolds. The study will also reference cognitive load theory to minimize extraneous load through streamlined UI/UX and chunked content. Key expected findings include (1) statistically significant improvements in concept mastery for the AI-tutor group relative to the control group, with medium to large effect sizes (Cohen’s d 0.40–0.70) after adjusting for pre-test scores; (2) higher engagement metrics and reduced time to mastery for students receiving adaptive interventions; (3) evidence that students with stronger initial misconceptions benefit more from targeted feedback and scaffolded hints; (4) positive teacher perceptions regarding feasibility and classroom integration, albeit with identified needs for professional development and alignment with existing curricula. The study’s contribution to knowledge lies in providing robust, empirically grounded evidence on the effectiveness of AI-driven adaptive tutoring for science concept mastery in real classrooms, advancing understanding of how AI tutors can be designed to mediate conceptual change within standard curricula, and offering a model for evaluating scalable ICT-based interventions in science education. The main conclusion is that well-designed AI tutoring, grounded in constructivist and ICAP principles and implemented with fidelity, can meaningfully enhance science concept mastery and engagement. Recommendations include scalable professional development for teachers, iterative refinement of AI-tutor reasoning to address persistent misconceptions, alignment with district assessment frameworks, and longitudinal studies to assess retention and transfer of conceptual understanding beyond the intervention period.