Automated Multimodal Discourse Analysis for Low-Resource Languages via AI

Thesis Abstract

This study addresses the persistent challenge of automatic understanding and analysis of discourse in low-resource languages, where multimodal cues (speech, gesture, facial expression, and visual context) are underrepresented in available corpora and computational models. The aim is to develop and validate a robust multimodal discourse analysis framework that leverages AI to fuse audiovisual, textual, and contextual signals for accurate interpretation of discourse structure and pragmatic function in low-resource language communities. Specific objectives include (1) constructing an annotated multimodal corpus for three representative low-resource languages (Language A, Language B, Language C) with 600 hours of multimedia interaction data, (2) designing a cross-modal representation model that integrates speech transcripts, gesture dynamics, facial action coding, and scene context using a transformer-based architecture with modality-specific encoders, (3) developing discourse-level annotation schemes informed by Systemic Functional Grammar and Speech Act Theory, and (4) evaluating the framework against baseline unimodal and hybrid models to quantify gains in discourse recognition, polarity, stance, and illocutionary force. The methodology adopts a mixed-methods research design combining data-driven AI development with qualitative validation. The population consists of publicly available broadcast and conversational data within the target language communities, supplemented by elicitation sessions with 40 native speakers per language to obtain high-quality transcripts and pragmatic annotations. A stratified sampling approach yields 600 hours of audiovisual material, divided into 300 hours for Language A, 180 hours for Language B, and 120 hours for Language C. Multimodal data collection instruments include synchronized audio recorders, high-definition video capture, and expert-annotated ground-truth labels for discourse functions (e.g., assertive, directive, expressive), discourse markers, and context labels. To ensure reliability, inter-annotator agreement is established using Cohen’s kappa and Krippendorff’s alpha on a subset of 15% of the data. The analysis employs a multimodal deep learning framework modality-specific encoders for speech (wav2vec 2.0 fine-tuned on each language), visual features (3D-CNN for gesture and facial expression, ResNet-50 for scene attributes), and text (language-specific BERT variants). These encoders feed into a cross-modal transformer with joint attention to produce discourse-level predictions, including discourse relation classification, speech act type, and stance detection. The annotation scheme aligns with Systemic Functional Grammar for semantic roles and functional categories, complemented by Gricean maxims for pragmatic inference. Model training uses cross-entropy objectives for token- and utterance-level tasks, with auxiliary losses for alignment consistency across modalities. Evaluations compare the proposed model against baselines unimodal (speech-only, vision-only) and simple late-fusion approaches, using metrics such as macro-averaged F1, BLEU for transcript quality, and Matthews correlation coefficient for discourse labels. The study also conducts ablation analyses to determine the contribution of each modality, and ablation experiments to assess cross-language generalization via zero-shot transfer. Expected findings indicate that the integrated multimodal framework outperforms unimodal and baseline fusion models across languages, with statistically significant improvements in discourse relation accuracy (?F1 ? 12–18%), speech-act classification (?F1 ? 10–15%), and stance detection (?F1 ? 8–12%). Cross-language experiments reveal that shared representations for gesture-speech contingencies and contextual cues enhance performance in Language B and Language C when Language A data is abundant, validating the model’s capacity for transfer learning in resource-scarce settings. The research is anticipated to produce a publicly available, richly annotated multimodal corpus and an open-source framework to facilitate reproducibility and further research in low-resource discourse analysis. The study contributes to knowledge by demonstrating a scalable, cross-modal approach to discourse analysis that integrates linguistic theory with state-of-the-art AI to address resource constraints in multilingual contexts. It advances methodological practices in corpus construction, annotation schemas, and evaluation protocols for multimodal NLP in low-resource languages, and provides practical tools for applications in education, media analytics, and human–computer interaction. The main conclusion posits that multimodal fusion grounded in linguistic theory significantly enhances discourse interpretation in low-resource languages, and recommends further exploration of active learning to expand annotated data, domain adaptation for diverse communicative contexts, and ethical considerations in deploying language technologies within minority communities.