Edge-Aware Vision Transformer for Real-Time Embedded Systems

Thesis Abstract

The rapid proliferation of edge devices in autonomous systems and smart cameras demands vision models that deliver high accuracy under real-time constraints with limited computational resources. Current offloading approaches incur latency, privacy, and reliability concerns, while traditional lightweight models often struggle with robustness in diverse environments. This study addresses the problem of deploying high-performing vision transformers at the edge by introducing an Edge-Aware Vision Transformer (EAVT) architecture optimized for real-time embedded systems. The aim is to design, implement, and evaluate an edge-aware transformer model that leverages sparse attention, dynamic routing, and hardware-conscious quantization to achieve sub-50 ms inference on a representative edge platform without sacrificing accuracy on standard benchmarks. The specific objectives are (1) to formulate an architecture that integrates edge-dedicated attention mechanisms with a compact transformer backbone suitable for embedded GPUs and MCUs; (2) to develop a training pipeline incorporating edge-augmented data augmentation, curriculum learning, and knowledge distillation from a high-capacity teacher model; (3) to implement the model on a development board with a constrained memory budget (?512 MB) and measure real-time performance under varying workloads; (4) to evaluate robustness to domain shifts, occlusion, and varying lighting conditions using established datasets; and (5) to compare against state-of-the-art lightweight vision transformers and conventional CNN baselines across accuracy, latency, energy consumption, and memory footprint. The methodology adopts a design, implementation, and evaluation research framework. A cross-sectional empirical study will be conducted with a purposively constructed dataset comprising 50,000 labeled images drawn from public benchmarks (e.g., ImageNet-1K subset, COCO) augmented with synthetic edge-specific variations. The population includes edge-enabled compute platforms, notably an NVIDIA Jetson Nano and an ARM-based MCU evaluation kit. A two-stage sampling strategy will be employed (a) data sampling to ensure balanced class representation and diverse environmental conditions, and (b) hardware sampling to assess portability across devices. Data collection instruments encompass (i) a custom edge-accelerated training pipeline using PyTorch with quantization-aware training and sparse-attention modules, (ii) low-level performance profiling tools (nvidia-smi, perf, and vendor SDK profilers), and (iii) energy measurement apparatus including a high-precision power meter for on-board consumption. Data analysis proceeds along quantitative and qualitative dimensions. For performance evaluation, descriptive statistics summarize inference latency, frame-rate, memory usage, and energy per inference. Inferential analyses include repeated-measures ANOVA to compare EAVT against baselines across five edge devices, with post-hoc Tukey tests for pairwise comparisons. Robustness assessments employ accuracy vs. occlusion level and illumination variance analyses using regression models to quantify sensitivity. Model interpretability will be supported by attention-map visualizations and gradient-weighted class activation mapping (Grad-CAM) to demonstrate region-focused processing on edge scenes. The study will also conduct ablation experiments to isolate the impact of (a) edge-aware sparse attention, (b) dynamic routing, and (c) quantization strategies. A theoretical framing will draw on the Information Bottleneck principle and Human-in-the-Loop theory for practical deployment considerations, complemented by the Dynamic Vision Transformer (DVT) paradigm as a baseline. Expected findings indicate that the EAVT achieves top-1 accuracy within 2–3 percentage points of a heavier transformer model on edge-compatible benchmarks while reducing average latency to 32–45 ms per frame on Jetson Nano and 28–40 ms on the ARM MCU board. Energy per inference is anticipated to decrease by 25–40% relative to full-precision baselines, with memory footprint under 512 MB. The approach is expected to demonstrate superior robustness to occlusion and lighting variation due to edge-aware routing and curriculum-based training. Findings will confirm the viability of deploying high-performance vision transformers on constrained devices, maintaining accuracy suitable for real-time decision-making in surveillance, robotics, and smart-camera applications. The study contributes to knowledge by delivering a novel edge-aware transformer design that bridges the gap between high accuracy and real-time embedded feasibility, providing a reproducible training-and-deployment pipeline for edge platforms. It informs hardware-software co-design paradigms for future edge AI systems and offers practical guidelines for balancing latency, energy, and accuracy in resource-constrained environments. The conclusion emphasizes the practical viability of EAVT and recommends extending the framework to multi-sensor fusion scenarios and real-world deployments in autonomous edge ecosystems.