Comparative Analysis of Edge AI Frameworks in Real-Time Inference

Thesis Abstract

Recent advances in edge computing have amplified the deployment of AI inference at the network edge, yet developers face a trade-off between latency, energy consumption, model accuracy, and resource visibility across heterogeneous edge devices and frameworks. This study addresses the problem of selecting and configuring edge AI frameworks to optimize real-time inference across diverse edge hardware, networks, and application domains. The aim is to conduct a rigorous comparative analysis of leading edge AI frameworks—TensorFlow Lite, PyTorch Mobile, OpenVINO, and Nvidia Jetson Inference—evaluating their performance, energy efficiency, and deployment simplicity in constrained environments. Specific objectives include (1) quantify latency, throughput, and end-to-end inference accuracy for common computer vision workloads (object detection and segmentation) under varying batch sizes (1–16) and input resolutions (224×224 to 640×480); (2) assess resource utilization (CPU/GPU utilization, memory footprint, and power consumption) on representative devices (Raspberry Pi 4, Nvidia Jetson Nano, and Intel NUC running Linux); (3) analyze framework-specific optimizations such as model quantization, operator fusion, and hardware acceleration; (4) evaluate ease of deployment, portability, and maintainability via a standardized benchmarking suite and developer surveys; and (5) synthesize guidelines for framework selection aligned with application requirements. The methodology adopts a mixed-methods, cross-sectional design. The population comprises widely used edge devices and AI models suitable for real-time inference. A stratified purposive sample of 60 edge deployments (15 per framework) across three device classes will be examined. Data collection will combine quantitative measurements from a controlled benchmark environment—utilizing a standardized workload suite consisting of YOLOv5n, MobileNetV3, and DeepLabV3Lite models—and qualitative inputs from semi-structured interviews with system engineers and developers. Instruments include a benchmark harness generating synthetic video streams at 15–60 frames per second, a power profiling tool (RAPL/Intel Power Gadget and NVIDIA's nvidia-smi) for energy measurement, and a survey instrument for deployment experience. Validity and reliability will be established through pilot testing (n=6), triangulation across hardware, software, and human factors, and inter-rater reliability checks for qualitative coding (Cohen’s kappa ? 0.80). Data analysis will proceed in three phases. First, descriptive statistics will profile performance metrics (latency, throughput, accuracy) and resource usage. Second, inferential analyses will apply repeated-measures ANOVA to compare frameworks across devices and tasks, with post hoc pairwise comparisons adjusted by Bonferroni correction. Third, regression analyses will model the relationship between framework optimizations (quantization level, operator fusion) and observed latency and energy consumption. A sensitivity analysis will examine the robustness of results to network variations and device thermal throttling. The study will also incorporate a theoretical lens drawn from the Technology-Organisation-Environment (TOE) framework and the Resource-Based View (RBV) to interpret framework adoption and performance in real-world edge ecosystems. Expected findings anticipate framework-dependent performance envelopes TensorFlow Lite and PyTorch Mobile may offer greater model flexibility but exhibit higher variance in latency on CPU-bound devices; OpenVINO is expected to yield superior inference speed on Intel-based hardware due to optimized kernels; Nvidia Jetson Inference is anticipated to deliver best end-to-end performance on GPU-enabled platforms but with higher energy costs. Quantization and hardware acceleration are likely to yield substantial reductions in latency and energy, particularly for smaller models. The study will contribute to knowledge by providing a rigorous, end-to-end comparative benchmark and actionable decision criteria for practitioners selecting edge AI frameworks for real-time inference under resource constraints. Conclusions will articulate a framework-selection decision model balancing latency, energy efficiency, accuracy, and deployment complexity, along with a set of best-practice recommendations for pre-deployment profiling, model optimization, and hardware-software co-design. Recommendations will include framework-specific guidance for quantization schemes, operator support priorities, and device-architecture alignment to maximize real-time inference performance, with implications for edge AI deployment standards and future research on unified benchmarking methodologies.