Design, implementation and evaluation of a multilingual phoneme-gesture alignment system

Thesis Abstract

This study addresses the challenge of aligning phonetic segments with articulatory gestures across multiple languages to improve spoken language understanding, second-language pronunciation training, and perceptual evaluation of multimodal speech. Despite substantial work on phoneme recognition and gestural annotation in monolingual contexts, there is limited cross-linguistic integration of phoneme-gesture alignment that can generalize to typologically diverse languages. The aim is to design, implement, and evaluate a multilingual phoneme-gesture alignment system that (i) learns language-agnostic gestural cues from audiovisual data, (ii) accurately maps phoneme sequences to synchronized articulatory gestures across a typologically diverse sample, and (iii) assesses system performance against human-annotated benchmarks. Specific objectives are to (a) develop a multimodal corpus comprising 150 hours of speech with synchronized articulatory gesture annotations across five languages (English, Mandarin, Spanish, Arabic, and Turkish), (b) implement a neural alignment architecture that fuses acoustic, visual, and kinematic features for phoneme-to-gesture alignment using a cross-lusion Transformer with joint CTC/attention objectives, (c) integrate a language-aware attention mechanism to handle phoneme-gesture correspondences in contexts with coarticulation and gestural variability, (d) evaluate alignment accuracy using frame-level alignment F1 scores and boundary precision-recall metrics, and (e) examine the system’s utility for pronunciation training and intelligibility assessment through a user study with 40 language learners. The methodological design combines corpus-based development with experimental evaluation. A mixed-methods approach is adopted, featuring quantitative analyses of alignment accuracy and qualitative feedback from expert annotators and language learners. The population comprises professional speakers and language learners recruited from university communities. The sample includes 20 native speakers per language for corpus collection and 40 adult second-language learners (eight per language) for user-testing. Data collection instruments include high-definition video and audio capture, electromagnetic articulography (EMA) for a subset of 20 speakers to obtain ground-truth gesture trajectories, and a refined gestural annotation scheme grounded in Articulatory Phonology. The system utilizes a multimodal encoder that processes spectrograms, lip-region video features, and EMA-derived gestural trajectories, with a sequential decoder predicting phoneme sequences and aligning them to gesture segments. Validity and reliability of instruments are established through inter-annotator agreement (Cohen’s kappa > 0.80 on gesture labels) and test-retest reliability for gestural measurements (intraclass correlation ICC > 0.85). Data analysis employs a combination of alignment metrics (frame-level F1, boundary recall, and Kendall’s tau for gestural sequence order) and statistical testing using mixed-effects models to account for language and speaker variability. The primary analytical framework is a cross-linguistic Transformer with joint Connectionist Temporal Classification (CTC) and attention objectives, augmented by a dynamic time warping-based smoothing post-processing step. Model evaluation includes ablation studies, cross-language generalization tests, and comparisons with baselines such as phoneme-only alignment and gesture-naïve audiovisual alignment. Ethical considerations encompass informed consent, data anonymization, and strict usage restrictions for EMA data. Expected findings indicate that the multilingual system can achieve frame-level alignment F1 scores exceeding 0.82 on English and exceed 0.76 on Mandarin, with meaningful performance gains (average 9–12% absolute F1 improvement) over monolingual baselines in all five languages. It is anticipated that the integration of gestural features will reduce misalignment caused by coarticulation, yielding more consistent gesture-to-phoneme mappings across languages with divergent phonotactics. The study also anticipates that learners using the system's pronunciation feedback will show statistically significant improvements in intelligibility scores (mean improvement of 0.6 on a 5-point Likert scale) after a four-week training intervention. The research contributes to knowledge by operationalizing Articulatory Phonology within a scalable, multilingual neural framework and by providing a publicly available, richly annotated multimodal corpus with cross-linguistic gesture annotations. The study concludes that robust phoneme-gesture alignment is feasible across typologically diverse languages and offers tangible benefits for pronunciation pedagogy and speech technology. Recommendations include extending the corpus to under-resourced languages, exploring real-time deployment optimizations for mobile devices, and integrating the system into language learning platforms to support adaptive feedback and intelligibility-focused assessment.