Comparative Analysis of Edge AI Acceleration on Embedded Devices

Thesis Abstract

Edge artificial intelligence (AI) has emerged as a pivotal capability for delivering low-latency, privacy-preserving inference on resource-constrained embedded devices. The study addresses the performance and energy efficiency gaps among contemporary edge AI acceleration strategies, particularly contrasting on-device inference across CPU, GPU, and dedicated neural processing unit (NPU) accelerators under varying workloads and hardware configurations. The aim is to determine how different acceleration approaches influence latency, throughput, energy consumption, and model accuracy in real-world embedded environments, and to identify trade-offs that inform design choices for constrained-edge deployments. Specific objectives include (1) benchmarking a representative set of deep neural networks (ResNet-18, MobileNet-V2, and TinyYOLOv3) on three embedded platforms (Raspberry Pi 4 with Arm CPU, Nvidia Jetson Nano with integrated GPU, and Google Coral Dev Board with Edge TPU) under uniform software stacks, (2) quantifying latency, frames-per-second, energy-per-inference, and thermal throttling across varying batch sizes (1, 2, 4, 8), (3) evaluating the impact of model quantization, pruning, and operator fusion on accuracy and inference speed, (4) analyzing architectural and software-level factors—memory bandwidth, cache behavior, and runtime optimizations—using instrumentation from Valgrind and NVIDIA Nsight, (5) applying theoretical lenses from the Resource-Based View (RBV) and the Theory of Constraints to interpret performance outcomes, and (6) proposing a decision framework for selecting edge acceleration configurations in typical IoT ecosystems. The methodology adopts a mixed-methods design framed as a cross-sectional comparative study. The population comprises commercially available embedded platforms and widely used neural network models suitable for edge inference. A purposive sampling of three hardware platforms and three CNN architectures results in nine primary configurations, with reproducible runs (n=30 per configuration) to ensure statistical robustness. Data collection employs automated benchmarking scripts to record latency, throughput, energy-per-inference (measured via hardware power monitors and the Integration of Power Data Acquisition), peak memory usage, and thermal readings, complemented by qualitative logs from runtime profiler outputs. Validity and reliability are ensured through calibration of power measurement instruments, repeated trials, and cross-validation of results using both open-source (TensorFlow Lite, PyTorch Mobile) and vendor-specific runtimes. Data analysis employs statistical methods including repeated-measures ANOVA to compare acceleration strategies across devices and networks, post-hoc Tukey tests for pairwise differences, and multivariate regression to model the relationship between model size, precision (FP32, int8, int4), and performance metrics. Where applicable, a non-parametric Kruskal-Wallis test is used to handle non-normal distributions. The study also integrates qualitative interpretation of system-level constraints to contextualize quantitative results. Expected findings indicate that dedicated accelerators (Edge TPU) yield the lowest latency and energy per inference for compact networks (MobileNet-V2) while maintaining acceptable accuracy, whereas CPU-based inference on constrained devices benefits from optimized runtimes and contextual batching in moderate workloads. GPU-based acceleration (Jetson Nano) is anticipated to outperform other platforms on larger models (ResNet-18) in terms of raw throughput but at higher energy costs, with thermal throttling becoming a critical factor beyond 4-inference batch sizes. Quantization and pruning are expected to improve latency and energy efficiency with marginal model accuracy loss for simple models, but more pronounced accuracy degradation for high-capacity networks unless retraining is employed. Theoretical interpretation through RBV will frame platform-specific capabilities as strategic resources, while the Theory of Constraints will illuminate bottlenecks in data movement, memory bandwidth, and kernel launch overhead. The study contributes to knowledge by delivering a comprehensive, empirically grounded framework for selecting edge acceleration configurations in heterogeneous embedded ecosystems, and by characterizing the interaction between hardware architecture, software optimization, and model compression techniques. The main conclusion anticipates that no single configuration is universally optimal; instead, performance is contingent on the model complexity, workload characteristics, and thermal management. Recommendations include guidelines for platform-model pairing, advisable quantization and pruning strategies aligned with target accuracy budgets, and a decision framework for practitioners to optimize for latency, energy efficiency, or accuracy given specific application constraints, along with directions for future work on adaptive runtime orchestration and co-design of models and hardware accelerators for edge AI.