Design and evaluation of an adaptive formative feedback system for CS education

Thesis Abstract

The study investigates how an adaptive formative feedback (AFF) system can enhance learning outcomes, code quality, and metacognitive awareness in undergraduate computer science education, addressing the persistent challenge of providing timely, targeted feedback at scale in introductory and intermediate programming courses. Despite widespread use of automated feedback tools, existing systems often deliver generic, non-adaptive guidance that fails to account for individual student trajectories, misconceptions, and varying prior knowledge. The aim is to design, implement, and evaluate an AFF system that personalizes feedback using learner models, domain ontologies, and real-time performance indicators to support novice to intermediate programmers across multiple CS modules. Specific objectives are to (1) develop a modular AFF architecture integrating automated error detection, a Bayesian learner model, and a constraint-based feedback engine; (2) implement the system within two programming courses (Python and Java) with 420 enrolled students over two consecutive offerings; (3) evaluate the impact on learning gains, programming self-efficacy, and persistence through a mixed-methods design; (4) examine the relationship between feedback timing, granularity, and student engagement; and (5) identify actionable design guidelines for scalable deployment in diverse CS education settings. A quasi-experimental and ethnographic mixed-methods approach is employed. The population comprises undergraduate CS students enrolled in two introductory programming courses at a mid-sized university. A total of 420 students (approximately 210 per course) participate, with 210 assigned to the AFF-enabled instructional condition and 210 to the standard feedback condition using existing institutional tooling. Data collection instruments include (i) learning analytics from the AFF system (error type, time-to-feedback, feedback granularity, subsequent attempt outcomes), (ii) pre- and post-tests assessing programming concepts and problem-solving skills, (iii) validated scales for programming self-efficacy and intrinsic motivation, (iv) course grade data, and (v) semi-structured interviews and focus groups with a purposive sample of 40 students and 12 instructors/teaching assistants. Instrument validity and reliability are established through pilot testing, factor analysis, and inter-rater reliability checks for qualitative coding. Data analysis employs a multi-tier strategy. Quantitative analyses begin with descriptive statistics and normality checks, followed by ANCOVA to compare post-test gains between the AFF and control groups while controlling for baseline performance, prior CS exposure, and course section. Regression analyses examine predictors of learning gains, including feedback latency, granularity, and student engagement metrics. Mediation analysis tests whether improvements in self-efficacy mediate the relationship between feedback characteristics and achievement. Qualitative data from interviews are analyzed using thematic analysis, with triangulation against system logs to interpret how students perceive, interpret, and act on adaptive feedback. A design-based research (DBR) cycle guides iterative refinement of the AFF architecture, and a user-centered evaluative framework evaluates usability, perceived usefulness, and cognitive load. Expected findings include (i) statistically significant improvements in post-test scores and course grades for the AFF group relative to controls; (ii) higher self-efficacy and intrinsic motivation associated with timely, granular, and trajectory-aware feedback; (iii) reduced cognitive load and increased persistence in subsequent modules; and (iv) qualitative evidence that adaptive feedback supports deeper conceptual understanding and debugging strategies by addressing specific misconceptions identified by the learner model. The study contributes to knowledge by integrating a Bayesian learner model with a constraint-based feedback mechanism within a scalable architecture, providing empirical evidence on the effectiveness of adaptive formative feedback in CS education and offering a transferable design blueprint for institutions seeking to deploy similar systems. The main conclusion anticipated is that an AFF system designed with principled learner modeling, timely and targeted feedback, and modular integration with existing LMS and IDE environments can produce meaningful gains in programming achievement and learner confidence. Recommendations include refining the federated data governance model for privacy-compliant analytics, extending the AFF to project-based assessments, exploring cross-language adaptability (e.g., Python, Java, C++), and developing professional development programs for instructors to maximize the effective use of adaptive feedback in large-scale CS curricula.