Artificial intelligence is rapidly moving beyond cloud infrastructure and into embedded devices at the edge. Wearables, hearables, smart-home products, industrial sensors, healthcare devices, and other endpoints are increasingly expected to capture and interpret data in real time and make decisions locally.
This shift can reduce latency, limit unnecessary wireless data transmission, and enable intelligent functions even when cloud connectivity is unavailable. But it also poses a difficult challenge for embedded designers: adding more intelligence without compromising battery life, thermal limits, or device size.
For power-constrained products at the edge, evaluating a system-on-chip (SoC) to meet customer expectations requires more than comparing peak AI performance. The end-to-end path from sensor input to inference result matters—including compute architecture, memory movement, software, power management, and the application’s actual operating duty cycle.
Increasingly, efficient Edge AI requires designing hardware and software as a unified system.
Look Beyond Peak AI Performance
AI accelerators are often characterized by their maximum computational throughput. While these specifications provide useful information about processing capability, they don’t necessarily indicate how efficiently an embedded AI application will run.
Real workloads contain much more than neural-network inference.
A device may first collect information from a microphone, camera, inertial measurement unit, or other sensor. That data may need filtering, transformation, or feature extraction before it reaches the AI model. Model weights and intermediate results must move through memory, and additional processing may occur after inference. Depending on the result, the device may activate another peripheral, store information, or establish a wireless connection.
Every step consumes energy.
For battery-powered Edge AI, a more meaningful question is therefore not simply how many operations an SoC can perform per second, but how much energy the complete system requires to turn real-world sensor data into a useful result.
Match Compute Resources to the Workload
Embedded AI workloads are rarely uniform, making heterogeneous computing an important part of Edge AI SoC design.
General-purpose CPUs provide flexibility for application logic and control. Vector-processing capabilities can accelerate signal-processing and machine-learning operations, while dedicated acceleration can provide additional performance for specific AI workloads.
The objective is not to run every operation on the highest-performance processing resource. Instead, the system should use the appropriate resources for each stage of the workload and activate higher-performance capabilities only when needed.
Consider an always-on audio application. The device may spend most of its operating life monitoring an audio signal and performing relatively lightweight processing. A more computationally demanding AI workload might only need to run when an event of interest is detected.
Keeping high-performance processing resources active continuously would waste energy. An architecture that can scale processing capability with workload demand and quickly return unused resources to lower-power states can be more efficient over the device’s complete operating cycle.
This distinction matters most for endpoints expected to run continuously from a limited energy source.
Memory Is Part of the AI Power Budget
Compute is only one part of Edge AI efficiency. Moving data also requires energy.
Neural networks depend on model weights, activations, sensor data, and intermediate results moving between memory and processing resources. As models become more sophisticated, memory capacity, bandwidth, and data movement can become significant contributors to overall system energy consumption.
This means an efficient AI engine can lose part of its advantage if the architecture continually moves large amounts of data or relies heavily on external memory.
Developers should therefore consider whether the model and its working data fit within available memory, how often information must move between processing resources, and how efficiently those resources can access the data they need.
For example, Ambiq’s Apollo510 SoC integrates up to 4 MB of non-volatile memory and up to 3.75 MB of TCM and low-leakage system RAM for code and data. Its Arm Cortex-M55 application processor with Helium technology can operate at up to 250 MHz. This combination provides developers with local compute and memory resources for increasingly sophisticated, embedded workloads while maintaining a focus on energy efficiency.
Software techniques such as quantization, model optimization, efficient kernels, and memory planning can further reduce computational and memory requirements. The goal is not simply to execute individual operations faster, but to minimize the energy required by the complete processing path.
Software Becomes Part of the Architecture
This is where the distinction between an AI-capable SoC and a practical Edge AI development platform becomes important.
Optimizing a few computationally intensive neural-network operations does not necessarily optimize the entire model. Production workloads can contain activation functions, reductions, padding, reshaping, normalization, preprocessing, and other operations surrounding the headline matrix multiplication and convolution workloads.
If those supporting operations execute inefficiently, they can become bottlenecks in latency and energy consumption.
The software stack therefore has an increasingly important role in extracting efficiency from Edge AI hardware. Optimized kernels can improve common operations, while runtimes and compilation tools determine how models map onto the underlying architecture.
Runtime choice can itself become an engineering tradeoff. A flexible runtime can simplify development and make it easier to update or change models. An ahead-of-time approach can instead specialize code around a particular model before deployment, potentially reducing runtime overhead and providing tighter control over memory, latency, and energy.
Neither approach is universally correct. The right choice depends on the product’s requirements.
Connecting Efficient Silicon with the AI Software Stack
Ambiq’s approach to Edge AI illustrates this hardware/software relationship.
At the silicon level, the Apollo510 is built around an Arm Cortex -M55 processor with Helium technology and Ambiq’s proprietary Subthreshold Power Optimized Technology (SPOT) platform. For typical AI inference workloads, Apollo510 has demonstrated up to 10 times higher performance and three times lower energy consumption compared with the previous-generation Apollo4 Plus.
The HELIA AI ecosystem extends this focus on efficiency into the software stack.
At its foundation, heliaCORE provides an optimized neural-network kernel layer for Apollo SoCs. More than 200 accelerated operators are optimized for Apollo devices, with validation spanning more than 40 AI models, 53 operator types, 247 distinct operators, and 963 operator instances. That represents approximately 12 times more operator instances than those covered by MLPerf Tiny.
This broader validation matters because real-world models extend beyond a small set of computationally intensive kernels. Operations surrounding the main computations can consume meaningful processing time and energy, making optimization across the complete model important for production deployments.
Above this foundation, heliaRT provides a runtime-oriented deployment path, while heliaAOT uses ahead-of-time compilation to provide greater optimization around a defined workload.
Ambiq’s neuralSPOT SDK provides an open-source, real-time, OS-agnostic development environment for building and deploying AI applications across Apollo SoCs, with support for TFLM, heliaRT, and heliaAOT workflows.
Together, these layers illustrate an important evolution in embedded AI: the SoC and AI software stack increasingly need to be considered as parts of the same architecture.
Evaluating an Edge AI Platform
As Edge AI SoCs become more capable, developers need to evaluate platforms against the requirements of the actual application rather than relying on a single performance specification.
Useful questions include:
- How much energy does the complete workload consume rather than the AI accelerator alone?
- Can the model and working data fit within the available memory architecture?
- How efficiently are both major AI kernels and supporting operators executed?
- Can compute resources and peripherals be activated only when required?
- How quickly can the system complete a workload and return to a low-power state?
- Does the software environment provide an efficient path from model development to deployment?
- Can developers optimize memory footprint, latency, and energy as product requirements evolve?
These questions shift the focus from theoretical peak performance toward the efficiency of the complete application.
Delivering More Intelligence per Unit of Energy
Edge AI models will continue to become more capable, but the physical constraints of endpoint devices will remain. Wearables will still require long battery life. Compact products will still have thermal limitations. Always-on sensors will still need to minimize energy consumption between meaningful events.
The next stage of Edge AI will therefore not be defined by silicon performance alone.
Efficient compute provides the foundation, but memory architecture, optimized operators, runtime and compilation choices, power management, and development tools determine how effectively that capability can be translated into a real product.
For designers of power-constrained Edge AI systems, the most useful measure of progress may not be how much AI computation a device can perform, but how much useful intelligence it can deliver within the available energy.
That is where hardware and software co-design becomes essential—and where the power budget becomes a defining architectural consideration for the next generation of Edge AI.
About the author
Charlene Wan is Vice President of Corporate Marketing and Investor Relations at Ambiq, where she leads the company’s global corporate communications, brand strategy, public relations, analyst relations, and investor engagement programs. Since joining Ambiq in 2020, she has played a pivotal role in strengthening the company’s market position as a leader in ultra-low power semiconductors and edge AI technologies, supporting private fundraising initiatives, and helping guide Ambiq through its successful initial public offering in 2025 and transition to the public markets. With more than two decades of leadership experience spanning semiconductors, consumer electronics, storage, and software, Charlene specializes in corporate positioning, strategic communications, and investor relations that enhance brand value, build stakeholder trust, and support long-term business growth.
