Multimodal AI for Real-Time Multilingual Communication in Classrooms

Thesis Abstract

In multilingual classroom settings, real-time multimodal translation and interpretation present opportunities to enhance comprehension and participation, yet practical deployment faces challenges of latency, accuracy, cultural nuance, and user acceptance. This study investigates the design, implementation, and evaluation of a multimodal AI system that supports real-time multilingual communication in university classrooms, aiming to improve student engagement, comprehension, and learning outcomes across language-diverse cohorts. The objectives are to (1) develop an integrated multimodal AI architecture that fuses speech, gesture, and visual context with multilingual translation and educational semantic parsing; (2) evaluate system performance in latency, translation accuracy, and appropriateness of pedagogical content across languages; (3) assess user acceptance, perceived usefulness, and interaction patterns among students and instructors; (4) examine the impact of real-time multimodal assistance on learning outcomes and participation; and (5) identify ethical, privacy, and equity implications for deployment in higher education. The methodology adopts a mixed-methods design within a pragmatic paradigm. The population comprises undergraduate and graduate classes at a large public university with multilingual student presence, sampled from three disciplines (engineering, humanities, and social sciences). A purposive sample of 18 intact classes (approximately 540 students total) will be recruited, with 9 classes assigned to an intervention condition and 9 to a control condition over a 14-week term. Data collection instruments include (i) system usage logs and performance metrics (latency, word error rate, and semantic accuracy) captured by the multimodal AI platform; (ii) classroom transcripts and dialogue act annotations for linguistic and pedagogical analysis; (iii) standardized learning assessments aligned with course objectives; (iv) pre- and post-surveys measuring perceived usefulness, satisfaction, and perceived learning gains; (v) semi-structured interviews and focus groups with students and instructors; and (vi) ethical impact audit checklists. Validity and reliability procedures incorporate triangulation across quantitative and qualitative data, instrument piloting, inter-rater reliability for manual annotations, and cross-language validation by bilingual experts. Data analysis combines quantitative and qualitative techniques descriptive statistics and inferential analyses (mixed-design ANOVA to assess time-by-condition effects on learning outcomes; hierarchical linear modeling to account for nested data structure; regression analysis to identify predictors of engagement and achievement) alongside qualitative thematic analysis of transcripts guided by grounded theory to reveal interaction patterns, discourse changes, and perceived affordances. A conceptual model is proposed wherein multimodal inputs (speech, gesture, facial expression, gaze, and visual context) feed a multilingual translation and pedagogy-aware processing module, which outputs simultaneously translated text captions, spoken language synthesis, and contextually tailored pedagogical cues, moderated by instructor interventions and classroom norms. Anticipated findings include significant improvements in comprehension and participation for multilingual students in the intervention group, moderated by language distance and prior exposure to English-medium instruction, as well as higher perceived usefulness and acceptance among instructors. The study expects to demonstrate that integrating gesture and visual context with real-time translation reduces cognitive load and increases inclusive interaction without compromising translation quality. The contribution to knowledge lies in (a) advancing an end-to-end multimodal framework for real-time multilingual classroom communication with demonstrated feasibility in real-world settings, (b) providing empirical evidence on instructional impact and user acceptance across disciplines, and (c) revealing ethical and equity considerations for scalable deployment in higher education. The main conclusion is that carefully engineered multimodal AI systems can enhance real-time multilingual?? without degrading pedagogical integrity, provided that latency remains within sub-second thresholds, translations preserve technical accuracy, and privacy safeguards are robust. Recommendations include incorporating ongoing human-in-the-loop auditing, developing language-specific calibration procedures, extending the model to include culturally responsive pedagogical cues, and establishing institutional policies for data governance, accessibility standards, and continuous professional development for instructors.