Embedded AI is a systems engineering problem

Deploying AI on embedded hardware is a systems integration problem. Model quality matters, but production viability also depends on the device's memory map, power delivery, thermal design, scheduler, and compiler. Many of those constraints are fixed by the hardware and product requirements before a model reaches the target.

This essay focuses on deployment-oriented embedded AI: moving a selected model from a development environment to reliable inference on a microcontroller or edge SoC. It does not argue that model development is easy. It argues that model accuracy alone does not determine whether an embedded deployment will fit, meet its latency requirements, or stay within its power and thermal limits.

Model availability is only a starting point

Model and runtime options for embedded deployment are broad. Google ships Gemma 3n in mobile-first variants supported by llama.cpp, Ollama, MLX, and Google's own stack. Meta's ExecuTorch 1.0 supports LLM, vision, and speech workloads with a 50 KB base runtime spanning microcontrollers to high-end phones. Its model support is broad, but export and backend compatibility are not automatic. They depend on the exported graph, operator coverage, memory requirements, and the delegates available on the target. A PyTorch model may still require graph changes, a different quantization path, or CPU fallback for unsupported partitions.

A reference demo may run on a development kit with wall power, generous cooling, and an unrestricted power mode. A production target may run in a sealed enclosure, from a battery, beside control and communication tasks with latency requirements. It also needs artifacts that a manufacturing line can reproduce. Closing that gap requires explicit memory budgets, quantization strategy, scheduling, power limits, and toolchain control.

The model is one workload within that system. Deployment engineering determines whether it can coexist with the rest of the product.

Memory budgets constrain deployment

At the small end of embedded machine learning, a recent survey of TinyML and tiny deep learning puts the typical envelope at 32 to 512 kB of SRAM, flash often under 1 MB, and power budgets in the milliwatt range. Its canonical TinyML definition is under 1 mW.

Microcontroller deployments usually budget two separate memory pools with different failure modes. Flash holds model weights and program code. SRAM holds activations, the tensor arena, and the rest of the firmware's runtime state. A practical STM32 deployment guide offers one planning heuristic for that workflow: keep model weights under roughly 50% of flash and peak activation memory under about 70% of SRAM, leaving room for firmware, drivers, and the RTOS. Those percentages are a starting point for STM32 planning, not universal limits. Parameter count alone is insufficient because peak activation memory can be the tighter constraint. A model with modest weights can still fail to fit because of a wide intermediate tensor. Memory planning, operator ordering, buffer reuse, and arena sizing are therefore first-class deployment tasks rather than compiler footnotes.

Development boards can hide a mismatch. An evaluation kit may have external PSRAM and more flash than the production board selected for volume. A model tuned against the development kit's memory map may not fit the production bill of materials. The hardware requirements should record the memory budget, and candidate models should be checked against the production memory map early.

Memory hierarchy also shapes deployment at the SoC scale. Gemma 3n is a public example: its E2B variant has 5 billion raw parameters and a memory footprint comparable to a traditional 2-billion-parameter model, and it can run in as little as 2 GB of memory. Per-Layer Embeddings allow the layer embeddings to be loaded and computed on the CPU, so approximately 2 billion core transformer parameters need to reside in accelerator memory. The split shows how model architecture and memory placement can be designed together for an edge target.

Quantization is part of deployment design

A converter may expose quantization as a flag, but selecting a format is a deployment decision that needs task-specific validation. The llama.cpp project documents more than twenty quantization formats, from Q8_0 through the k-quants to sub-2-bit IQ variants. Its Llama 3.1 8B table lists about 15 GiB at F16 and about 4.6 GiB at Q4_K_M, a reduction of roughly a factor of three. That reduction can determine whether a model fits. Lower precision can also reduce quality, so the tradeoff needs to be measured on production-representative task data. Methods such as GPTQ and AWQ, along with GGUF importance-matrix calibration, address limitations of naive rounding.

NPUs add backend-specific constraints. Mobile NPUs often favor fixed integer datapaths, and recent work on on-device LLM inference argues for fully static quantization because dynamic rescaling can be expensive on NPU silicon. Deployment may require INT8 or INT4 graphs, calibration data that resembles production inputs, and selected layers kept at higher precision. Changes to those choices should trigger an accuracy regression.

Qualcomm reports that on the Snapdragon 8 Elite Gen 4, conventional LLMs running on the Hexagon NPU instead of the CPU load 30 to 75% faster and decode 2 to 4x more tokens per second, with memory footprint reduced by up to 47%. Those hardware-specific results illustrate why quantization and backend compilation are part of deployment work.

Power and scheduling bound runtime behavior

Peak throughput does not establish sustained performance under a product's thermal and power limits. NVIDIA's 8 GB Jetson Orin Nano module offers 7 W and 15 W modes, a 25 W mode, and an uncapped MAXN SUPER mode delivering 67 sparse INT8 TOPS. The same documentation says the module reduces frequency when its power exceeds the thermal design budget and recommends a custom power mode tuned to the workload. The 67 TOPS figure is therefore mode-specific. Sustained throughput needs measurement in the intended enclosure and ambient conditions.

Scheduling constraints matter on both microcontrollers and shared SoCs. Inference may run for long intervals and can disturb cache locality, but the effect depends on the model, runtime, backend, and competing tasks. On an embedded device, inference may share compute resources with motor control, sensor sampling, and communication stacks that have latency requirements. An RTOS design should establish task priority, watchdog behavior, and whether the selected execution path can be preempted at a useful granularity. Preemption behavior is backend-specific: the scheduler may preempt a CPU thread, while an accelerator delegate may expose coarser execution boundaries. If an operator or delegated subgraph has no useful yield point, higher-priority work can be delayed.

On embedded Linux, frequency governors and thermal management may reduce clock rates during inference. On battery devices, wake frequency, state transitions, and sleep residency can matter as much as arithmetic throughput. A faster inference path that wakes the SoC for every sensor event may consume more energy than a slower one with a lower duty cycle. These behaviors require measurement on the target; a model card does not describe them.

Toolchain compatibility is part of deployment

The available edge runtimes cover a broad range of frameworks and targets, but compatibility remains target-specific. Google's LiteRT, the runtime formerly named TensorFlow Lite, provides conversion paths from TensorFlow, PyTorch, and JAX. It dispatches work to GPUs and NPUs from Qualcomm, MediaTek, Intel, and Google Tensor, and runs in more than 100,000 applications. ExecuTorch 1.0 covers twelve-plus hardware backends, including CoreML, Vulkan, Arm Ethos-U NPUs, and Cadence DSPs. Meta uses it across Instagram, WhatsApp, Quest 3, and Ray-Ban smart glasses.

At the bare-metal end, microTVM compiles models ahead of time into C and can link against CMSIS-NN kernels. A 2025 survey of the field describes LiteRT for Microcontrollers as the standard choice for running neural networks on an MCU and notes ExecuTorch's traction on Ethos-U parts. It also identifies the tradeoff in vendor tools such as STM32Cube.AI: they can be fast, but they tie a deployment to the vendor's silicon and tooling.

Two compatibility risks need early testing. The first is operator coverage. A model may convert successfully while an unsupported operator remains on the CPU instead of the NPU. If delegate logs and profiling are not checked, that fallback can be missed. Model architecture should therefore be evaluated against the selected backend's operator coverage and partitioning behavior.

The second risk is artifact reproducibility. Embedded products often need to reproduce model binaries after the original deployment. When Google renamed TensorFlow Lite, it retained the .tflite format. Format stability helps, but converter, runtime, delegate, and compiler versions still need to be pinned with the deployment artifacts.

A deployment workflow

A deployment workflow starts with a target budget for peak SRAM, flash, sustained power, worst-case latency, and thermal limits. Candidate models should be measured on production-representative hardware, power, and cooling rather than only on a development kit. Quantization belongs in model validation, with regression tests on task data. The model architecture and toolchain should be selected together so the full graph, memory plan, and delegate partitioning can be tested before the deployment design is fixed.

A common failure pattern is selecting the model before checking it against the target system. Establishing the constraints first may add work early, but it reduces the risk of late changes to the model graph, firmware schedule, cooling design, or hardware.

Model updates do not automatically drop into an embedded product. Each candidate still needs validation for export, operator coverage, memory, accuracy, latency, power, and thermal behavior. Integration time depends on what changed. For deployment-oriented embedded AI, the memory map, power envelope, scheduler, and toolchain are as consequential as the model itself.