A Multimodal Conversational Alignment Framework for Dialect Variation

Thesis Abstract

This study investigates how multimodal cues—prosody, gesture, gaze, facial expressions, and linguistic choices—facilitate conversational alignment across dialect varieties, addressing the persistent mismatch between communicative intent and perceived intelligibility in multilingual and multi-dialectary settings. The problem stems from limited integrative models that quantify how cross-dialect alignment emerges in real-time interaction when multiple modalities interact, potentially amplifying or reducing communicative friction in dyadic and small-group discourse. The aim is to develop a comprehensive Multimodal Conversational Alignment Framework (MCAF) that identifies, models, and tests the mechanisms by which speakers align their talk across dialectal differences, and to validate this framework in naturalistic and elicited conversational contexts. Specific objectives are (1) to operationalize multimodal alignment indicators across dialect varieties using a coding scheme that integrates prosodic features, facial affect, gaze patterns, hand gestures, and lexical choices; (2) to examine the temporal dynamics of alignment using fine-grained sequential analysis and time-series modeling; (3) to determine the relative contribution of each modality to alignment in spontaneous versus task-driven conversations; (4) to test the applicability of established theories of alignment and sociolinguistic expectancy, including Communication Accommodation Theory (CAT) and Dynamic Systems Theory (DST), in predicting cross-dialect convergence; and (5) to develop a predictive model capable of forecasting alignment outcomes from multimodal input and dialect distance measures. The methodology adopts a mixed-methods, multi-site design combining corpus-based analysis with controlled elicitation experiments. The population comprises adult bilinguals and bidialectal speakers (N ? 120) recruited from urban and rural communities with substantial dialectal variation. A stratified sampling approach yields equal representation of three dialect pairings with distinct sociolinguistic distances. Data collection uses (i) naturalistic conversations recorded in social interaction settings (n ? 60 dyads, each 20–30 minutes), (ii) structured interaction tasks designed to elicit negotiation and problem-solving (n ? 30 dyads), and (iii) elicitation interviews to capture subjective alignment perceptions (n ? 60 participants). Instruments include high-definition video and audio capture, immersive eye-tracking glasses for gaze data, and a synchronized motion capture system for gestural analysis, supplemented by audio-based prosodic profiling (F0, intensity, speech rate) and lexical-semantic tagging. Analytical procedures combine quantitative and qualitative techniques. Multimodal data are aligned using a unified annotation protocol incorporating a multimodal coding scheme, with reliability established via Cohen’s kappa and intraclass correlations. Temporal alignment is analyzed through Granger causality and cross-recurrence quantification analysis to detect precursor and lagged effects across modalities. A hierarchical Bayesian modeling approach assesses the probability of alignment outcomes given multimodal inputs and dialect distance, while structural equation modeling tests the mediating role of social factors (power, accommodation willingness). Regression analyses (linear, logistic) examine modality-specific contributions to alignment success, and ANOVA explores differences across dyad types (spontaneous vs. task-driven). Thematic analysis of interview data provides interpretive depth on perceived alignment quality and sociolinguistic attitudes. The study integrates theoretical lenses from Communication Accommodation Theory (CAT) and Dynamic Systems Theory (DST), situating alignment as emergent, distributed across modalities, and sensitive to contextual constraints. Key expected findings include (a) a robust set of multimodal indicators that predict alignment across dialects with high accuracy (anticipated R^2 ? 0.60 for predictive models); (b) modality interaction effects revealing that gaze and prosody jointly predict alignment more strongly in face-to-face contexts, whereas lexical alignment becomes more salient in audio-only settings; (c) dialect distance moderates the strength of multimodal alignment, with closer dialect pairs showing more rapid and stable convergence; and (d) cognitive load and task type influence the allocation of attention to different modalities, altering the alignment trajectory. The study contributes to knowledge by proposing and validating the Multimodal Conversational Alignment Framework, integrating cross-disciplinary methods to model alignment processes in dialect variation, and offering a scalable predictive tool for speech technologists and sociolinguists. Conclusions are expected to emphasize that alignment is a distributed, dynamic process shaped by the interaction of multiple modalities and sociolinguistic factors, rather than a single-channel adjustment. Recommendations include the design of inclusive communication strategies in multilingual environments, the development of multimodal diacritic tools for dialect-aware conversational agents, and the refinement of theoretical models of accommodation to account for multimodal and dynamic conditions.