Automated Diagnos­tics of Code-Switching in Multilingual Speech Interfaces

Thesis Abstract

Multilingual speech interfaces increasingly enable real-time communication across language communities, yet code-switching—shifting between languages within a single discourse—poses substantial challenges for automatic speech recognition (ASR), natural language understanding, and user experience. This study addresses the problem that existing interfaces exhibit degraded recognition accuracy and misinterpretation of user intents when code-switching is present, leading to higher error rates, user-frustration, and reduced accessibility for multilingual populations. The aim is to develop automated diagnostics that detect, characterize, and predict code-switching phenomena in multilingual speech interfaces and to evaluate how these diagnostics can inform adaptive systems that maintain performance and usability. The specific objectives are (1) to profile the prevalence and linguistic patterns of code-switching in bilingual and multilingual user interactions with conversational agents; (2) to develop a diagnostic framework combining acoustic-phonetic cues, lexical cues, and discourse-based features to identify code-switch points and their impact on ASR and intent classification; (3) to evaluate the diagnostic framework on a diverse corpus of multilingual speech data with explicit code-switching, comparing performance across languages, domains, and user demographics; (4) to integrate the diagnostics into a prototype adaptive speech interface that dynamically adjusts recognition models and dialogue management based on detected switching; and (5) to derive guidelines for designers on mitigating code-switching-related errors and improving user satisfaction. A mixed-methods design will be employed, incorporating corpus-based quantitative analysis and qualitative error analysis. The population consists of bilingual and multilingual speakers (n=120) drawing from two high-education urban centers with Turkish–English and Hindi–English bilingual communities, sampled to reflect gender, age, and proficiency variance. Data will comprise approximately 240 hours of spoken interactions collected through controlled task-based sessions (n=4 tasks per participant) and spontaneous usage in naturalistic settings (n=2 weeks of logged interactions). Instruments include a bespoke code-switching annotation schema integrating the Linguistic Inserting and Switching (LIS) framework with a probabilistic language identification module, acoustic feature extractors (e.g., MFCC trajectories, pitch, energy), and contextual markers (dialogue act tags, reaction times). ASR systems will be configured to operate with and without dynamic model adaptation to quantify the diagnostic system’s impact. Reliability will be established via inter-annotator agreement (Cohen’s kappa ? 0.80) and test–retest stability for feature extraction. Analytical techniques will encompass a sequence of approaches (i) generalized linear mixed models (GLMMs) to examine the relationship between code-switching frequency and ASR error rates across languages and contexts; (ii) survival analysis to model the time-to-next-switch in dialogue sequences; (iii) regression analyses to identify predictors of misclassification in intent recognition when code-switching occurs; (iv) Bayesian hierarchical modeling to estimate uncertainty in language identity and switching boundaries; (v) machine learning classifiers (random forest, gradient boosting, and transformer-based encoders) trained on combined acoustic-lexical-discourse features to detect code-switch points, assess their severity, and predict degradation in downstream tasks; and (vi) thematic analysis of qualitative error logs to uncover user experience implications. The theoretical framing integrates sociolinguistic code-switching theory (Gumperz) and the principle of adaptive user interfaces within human–computer interaction, complemented by the speech processing theory of acoustic-phonetic invariance in multilingual production. Expected findings include (a) robust diagnostic indicators that reliably signal code-switch onset and type (inter-sentential versus intra-sentential) and correlate with declines in ASR accuracy and intent classification, (b) quantifiable thresholds for when model adaptation significantly improves recognition performance in switch-rich segments, and (c) evidence that user perceived usability improves when interfaces proactively adjust language models and prompt for disambiguation at switch points. The study contributes to knowledge by operationalizing a reproducible, data-driven diagnostic framework for code-switching in multilingual interfaces, integrating linguistic, acoustic, and interactional signals, and providing actionable design guidelines for adaptive conversational systems. The main conclusion is expected to indicate that early, context-aware diagnostics combined with dynamic model selection can substantially mitigate code-switching-related errors, enhancing accessibility and user satisfaction. Recommendations will address dataset diversification, real-time diagnostic latency reduction, ethical considerations for bilingual data, and guidelines for scalable integration into commercial multilingual ASR and dialogue systems.