Developing a Multimodal Dialogue System for Endangered Language Revitalization

Thesis Abstract

This study investigates the development of a multimodal dialogue system designed to support the revitalization of an endangered language by enabling naturalistic spoken, gesture, and visual interactions within community-centric educational and digital archival contexts. The core problem addressed is the erosion of productive language transmission in everyday contexts, where traditional literacy- and media-based resources fail to capture the full socio-communicative fabric of the language. The aim is to design, implement, and evaluate a multimodal dialogue system that integrates speech, gesture, and gesture-enabled visual cues with culturally informed lexical and syntactic models to facilitate learner-centered practice, authentic discourse simulation, and intergenerational transmission. Specific objectives include (1) to develop an annotated multimodal corpus capturing spontaneous speech, co-speech gestures, facial expressions, and contextual visual scenes in language-use scenarios; (2) to construct a scalable dialogue management framework incorporating end-user feedback loops and a rule-based plus neural hybrid approach that respects community norms and language ideologies; (3) to evaluate linguistic accessibility, usability, and learning outcomes across diverse user groups; and (4) to assess the system’s impact on active language production and cultural knowledge retention over a 12-week pilot deployment. The methodology adopts a mixed-methods research design, combining corpus-based linguistic analysis, human–computer interaction (HCI) experimentation, and evaluative heuristics grounded in sociolinguistic theory. The study will be conducted in a language documentation setting within a community of roughly 1,200 fluent or semi-fluent speakers, with a stratified sample of 60 participants spanning ages 12–65, including 20 heritage learners and 20 adult language learners. Data collection instruments comprise (a) a multimodal corpus elicitation protocol using scripted tasks and naturalistic interaction sessions recorded via high-fidelity audio, 4K video, and depth-sensing cameras; (b) a dialogue system prototype deployed on tablets and smartphones with offline and cloud-based processing capabilities; (c) standardized usability scales (System Usability Scale, User Experience Questionnaire) and retention tests; (d) semi-structured interviews and focus groups to capture sociocultural dimensions. Validity and reliability are ensured through inter-rater reliability checks (Cohen’s kappa ? 0.80) for annotation, pilot testing of the corpus annotation schema, and triangulation across linguistic, usability, and learning outcomes. Analytical procedures include (i) thematic analysis of qualitative data to identify language-ideology considerations and user needs; (ii) variational analysis of corpus annotations to extract lexical, syntactic, and gestural alignment patterns; (iii) quantitative evaluation of learning gains using paired-sample t-tests and repeated-measures ANOVA to compare pre- and post-deployment language proficiency; (iv) mixed-effects modeling to examine factors influencing system usage and language production outcomes; (v) regression analyses to determine predictors of sustained engagement. The theoretical backbone integrates Activity Theory and Sociocultural Theory to frame user-system interactions as mediated by tools and community practices, while applying Endangered Language Documentation and Technology Acceptance models to interpret adoption dynamics. A conceptual model linking multimodal inputs, dialogue strategies, learner outcomes, and cultural reinforcement is developed and iteratively refined through pilot data. Expected findings include (a) enhanced lexical access and grammatical production in the endangered language facilitated by synchronized speech-gesture cues; (b) higher retention of cultural concepts when multimodal prompts align with community discourse practices; (c) positive usability scores and increased sustained engagement among younger users compared with constraints in older participants, mitigated through adaptive interface configurations. The study contributes new knowledge by (i) presenting an empirically validated multimodal dialogue architecture tailored for endangered language contexts, (ii) offering an annotated, publicly shareable multimodal corpus with rich metadata, and (iii) providing a scalable evaluative framework for ICT-driven language revitalization initiatives. The principal conclusion posits that culturally attuned multimodal dialogue systems can significantly augment active language production and intergenerational transmission. Recommendations include expanding corpus diversity, integrating community-led governance for content curation, prolonging longitudinal assessments beyond 12 weeks, and exploring cross-language transferability to other endangered languages with similar sociocultural ecosystems.