Design and Evaluation of AI-Driven Document Automation System for SMEs

Thesis Abstract

This study investigates the design, implementation, and evaluation of an AI-driven document automation system tailored for small and medium-sized enterprises (SMEs), addressing the persistent inefficiencies and error-prone routine document workflows that constrain productivity and competitive advantage. The problem centers on manual document processing, fragmented data sources, and inconsistent compliance across business units, which collectively hinder timely decision-making and increase operational risk. The aim is to deliver a scalable, modular AI-assisted platform that automates document ingestion, classification, routing, extraction, and policy-compliant generation, while evaluating its impact on processing time, accuracy, user satisfaction, and compliance adherence. Specific objectives include (1) mapping typical SME document workflows and identifying critical bottlenecks, (2) designing an adaptable architecture incorporating natural language processing (NLP), optical character recognition (OCR), rule-based decision engines, and a lightweight RPA layer, (3) implementing a pilot system for three SME contexts (legal, procurement, and human resources) with reusable templates and domain-specific ontologies, (4) evaluating system performance against baseline manual processes using a quasi-experimental design, and (5) deriving guidelines for deployment, adoption, and governance in SME settings. A mixed-methods approach combines quantitative and qualitative techniques to provide both measurable impact and user insights. The research adopts a pragmatic research design within an interpretivist-positivist hybrid framework to balance objective performance metrics with organizational context. The population comprises SME employees involved in document-intensive processes across three pilot domains in a metropolitan business district. A purposive sample of 60 participants (20 per domain) is selected, with 30 participants representing the control group (pre-implementation processes) and 30 representing the intervention group (post-implementation using the AI-driven system). Data collection instruments include (i) time-motion logs and system-generated metadata to quantify processing time, error rates, throughput, and compliance flags; (ii) structured questionnaires measuring perceived usefulness, ease of use, and user satisfaction using established scales (Technology Acceptance Model extensions); and (iii) semi-structured interviews and focus groups to capture contextual factors, workflow fit, and perceived change in decision-making quality. Validity and reliability are ensured through instrument triangulation, pilot testing (n=10), Cronbach’s alpha checks (target >0.80 for multi-item scales), and inter-rater reliability for qualitative coding (Cohen’s kappa >0.70). Data analysis employs a combination of descriptive statistics, inferential analyses, and thematic synthesis (a) paired t-tests and ANCOVA to assess reductions in processing time and error rate while controlling for domain-specific covariates, (b) regression analysis to identify factors predicting user acceptance and system utilization, and (c) thematic analysis of interview data guided by the Diffusion of Innovations and Technology Acceptance Model frameworks to elucidate adoption barriers and enablers. An analytical framework integrates a conceptual model that maps inputs (data quality, model accuracy, governance rules), processes (OCR/NLP pipelines, rule engine, and templating), and outcomes (efficiency, accuracy, compliance, and satisfaction). The study includes a cost-benefit analysis to estimate return on investment (ROI) over a 12-month horizon, incorporating capital and operational expenditures, incremental productivity gains, and risk mitigation benefits. Expected findings anticipate significant reductions in average document processing time (20–35%), error rates (25–40%), and cycle times for approval routing, alongside improved compliance flag accuracy and higher user satisfaction scores (mean increase of 0.8–1.2 on a 5-point scale). The research contributes to knowledge by operationalizing an SME-focused AI document automation architecture, identifying pivotal design decisions (modular NLP components, templated governance, and explainable AI considerations), and providing empirical evidence on organizational impact, adoption dynamics, and governance implications in resource-constrained settings. The conclusion is anticipated to advocate a staged deployment framework, emphasizing domain-specific ontology development, change management, and continuous monitoring to sustain gains. Recommendations include developing industry-specific template libraries, establishing data quality benchmarks, and integrating the system with existing enterprise resource planning (ERP) and customer relationship management (CRM) ecosystems to maximize interoperability and value realization for SMEs.