Developing AI-Enhanced Speech Recognition for Multilingual Healthcare Communication

Thesis Abstract

Effective communication between healthcare providers and patients is critical for ensuring quality healthcare outcomes, yet language barriers present significant challenges in multilingual settings, often resulting in misdiagnosis, treatment errors, and reduced patient satisfaction. The proliferation of digital communication technologies offers opportunities to mitigate these barriers through the development of robust speech recognition systems capable of functioning across multiple languages. This study aims to develop and evaluate an AI-enhanced speech recognition model tailored for multilingual healthcare communication, specifically targeting environments where language diversity complicates clinical interactions. The primary objective is to design a speech-to-text system that accurately transcribes medical dialogues in at least five widely spoken languages within the healthcare setting—namely English, Spanish, Mandarin, Arabic, and Swahili—while maintaining high levels of precision and contextual understanding. Secondary objectives include assessing the system’s performance against existing multilingual speech recognition tools, understanding the system's usability from healthcare practitioners’ perspectives, and identifying linguistic nuances affecting recognition accuracy. The research adopts a mixed-methods approach, combining quantitative development and validation of the speech recognition model with qualitative usability assessments. The quantitative component employs a participatory experimental design involving a corpus of 10,000 annotated healthcare-related speech samples collected from 200 native speakers spanning the targeted languages. Data collection utilizes a combination of in-situ recordings during simulated clinical consultations and existing open-access datasets, ensuring diversity in speaker accents, speech styles, and medical terminologies. The core model is developed using deep learning techniques, specifically leveraging transformer-based architectures integrated with transfer learning from multilingual pretrained language models such as mBERT and wav2vec 2.0. Model training involves hyperparameter tuning and cross-validation to optimize recognition accuracy while addressing language-specific phonetic and syntactic features. Evaluation metrics include Word Error Rate (WER), Sentence Error Rate (SER), and semantic accuracy scores, subjected to statistical analysis using repeated measures ANOVA to compare performance across languages and baseline systems. Additionally, error analysis identifies recurrent misrecognitions linked to linguistic complexities. The qualitative component comprises semi-structured interviews with 30 healthcare practitioners, analyzed through thematic analysis to gauge system usability, contextual appropriateness, and integration challenges within clinical workflows. Expected findings indicate that the AI-enhanced model can achieve an average WER below 10% across the five languages, outperforming existing commercial multilingual speech recognition systems by at least 15 percentage points. The model demonstrates particular strengths in handling medical jargon and contextual understanding, owing to domain-specific fine-tuning and multilingual training. Usability assessments are projected to reveal high acceptance levels among practitioners, with insights into necessary interface adjustments and integration strategies. This research contributes novel insights into the application of transformer-based transfer learning models in multilingual healthcare settings, specifically in low-resource languages such as Swahili and Arabic, where limited existing research exists. It advances understanding of linguistic challenges affecting speech recognition accuracy and demonstrates the efficacy of AI-driven solutions in enhancing communication, thereby supporting equitable healthcare service delivery. The study concludes that integrating AI-enhanced speech recognition into healthcare workflows can significantly improve communication efficiency, reduce misdiagnosis, and foster inclusivity in diverse patient populations. Based on the findings, the study recommends the deployment of tailored multilingual speech recognition systems in healthcare facilities, further research into expanding language coverage, and the development of user-centric interfaces that accommodate clinical contextual cues. Future work should also explore real-time implementation, integration with electronic health records, and continuous learning mechanisms for improving system accuracy over time.