Design, implement, and evaluate a language-oriented chatbot for second-language learners

Thesis Abstract

This study investigates the design, implementation, and evaluation of a language-oriented chatbot to support second-language (L2) learners in acquiring practical communicative competence and intercultural pragmatic awareness. The problem addressed is the limited availability of scalable, adaptive conversational practice that integrates authentic language use with feedback on accuracy, appropriateness, and discourse management, particularly for classroom-to-real-world transfer. The aim is to develop a chatbot that (i) provides targeted lexical and grammatical practice, (ii) simulates diverse social interactions across registers, and (iii) offers real-time corrective feedback aligned with communicative competence frameworks. Specific objectives are to design a modular chatbot architecture incorporating natural language understanding, dialogue management, and adaptive feedback; to evaluate usability, learner engagement, and perceived linguistic gains; to measure objective learning outcomes through pre- and post-tests and to examine transfer to communicative performance in written and spoken tasks; and to investigate learners’ strategic adjustments and affective responses during extended interactions. A mixed-methods approach guides the inquiry. The research adopts a quasi-experimental design with two intact L2 learner groups (N = 120) drawn from two university English-as-a-foreign-language programs, assigned to a treatment condition (chatbot-assisted practice) and a control condition (traditional instructor-led practice) for 12 weeks. Data collection instruments include a standardized proficiency test (e.g., IELTS-like speaking and writing tasks) administered at baseline, mid-point, and post-intervention; a system usage log capturing interaction frequency, response latency, and error types; a validated usability questionnaire; a motivation and affect survey; and performance on two discourse tasks (role-plays and collaborative writing) scored with rubrics aligned to the Common European Framework of Reference for Languages (CEFR) descriptors. Qualitative data comprise think-aloud protocols during pilot sessions (n = 20) and semi-structured interviews with a stratified subsample of participants (n = 24) to elicit perceptions of conversational realism, feedback usefulness, and perceived learning gains. Data analysis integrates quantitative and qualitative techniques repeated-measures ANOVA and ANCOVA to assess between- and within-group changes in language proficiency and task performance; regression analysis to identify predictors of learning gains (e.g., time-on-task, feedback quality, initial proficiency); mixed-effects modeling to account for nested data (participants within groups) and time; thematic analysis of interview transcripts and think-aloud data to triangulate with quantitative outcomes. The theoretical basis draws on Activity Theory to frame dialogic interaction with the chatbot as mediated learning, and on Sociocultural Theory to situate feedback and collaboration within zone of proximal development. Additional reference is made to output processing and error-treatment theories to optimize corrective feedback. Expected findings include statistically significant improvements in targeted linguistic constructs (lexical diversity,accuracy of morphosyntax, and pragmatic appropriateness) for the chatbot group relative to controls, as well as enhanced discourse skills in role-plays and collaborative writing. Increased engagement, higher perceived usefulness, and positive affect toward autonomous practice are anticipated, alongside evidence that adaptive feedback based on learner evolving proficiency yields greater gains than static feedback. Qualitative insights are expected to reveal how learners negotiate lexical gaps, apply reformulation strategies, and transfer classroom learning to spontaneous interactions, with themes concerning perceived realism, dialogic scaffolding, and perceived social presence of the chatbot. The study contributes to knowledge by empirically validating a scalable, theory-grounded chatbot design that integrates adaptive feedback, discourse-aware evaluation, and ecologically valid tasks for L2 development. It offers a practical blueprint for deploying language-oriented conversational agents within tertiary curricula and contributes to methodological debates on rigorous assessment of technology-enhanced language learning interventions. The main conclusion is that a well-designed language-oriented chatbot can yield measurable gains in communicative competence and learner motivation when embedded in structured practice with scaffolded feedback aligned to CEFR-based targets. Recommendations include refining adaptive feedback algorithms, expanding multimodal input capabilities for richer discourse analysis, and exploring longitudinal effects across different L2 varieties and instructional contexts.