Hardware accelerators and edge deployment
LLMs and VLMs do not only run on data-center GPUs. This session builds one simple model of any device (memory capacity, memory bandwidth, compute and power), uses it to compare GPUs, TPU-style systolic arrays, FPGAs (FlightLLM), phone NPUs and SRAM-heavy chips, and then turns to the edge: what fits on a phone or a Jetson, which runtimes are used (llama.cpp, MLX, ExecuTorch, MLC LLM), and which optimizations from the survey matter most there.
What you will learn
- Describe any device with four numbers (capacity, bandwidth, compute, power) and use them to predict fit, decode speed, prefill speed and energy per token
- Why decode at batch size 1 is memory-bound on every accelerator, from a phone to an H100, using the ridge point
- How a systolic array (TPU, many NPUs) multiplies matrices cycle by cycle, and why it is nearly idle during decode
- FlightLLM on FPGAs: the configurable sparse DSP chain, always-on-chip decode with mixed precision, length-adaptive compilation, and how to read its 6.0× energy claim
- Why moving bytes dominates energy, and what that means for bits per weight and on-chip memory
- The edge runtimes: llama.cpp and GGUF, MLX and unified memory, ExecuTorch, MLC LLM, and LoRA adapters on devices
- Which survey techniques matter most at the edge, worked through for a VLM on a phone
Why this matters
Until now, most of this track assumed one kind of machine: a data-center GPU with fast HBM memory and hundreds of watts of power. But many LLMs and VLMs run somewhere else: on phones, laptops, cars, robots, cameras, and on special chips in data centers that are not GPUs. On these machines, the rules of the game are the same, but the numbers are very different.
Three things limit any device that runs a language model:
- Memory capacity. The weights and the KV cache must fit. A phone with 8 GB of memory, shared with the operating system and other apps, cannot hold an 8B model even in 4-bit form.
- Memory bandwidth. In the decode phase, every new token reads every weight once (session 82). So the bytes per second the memory can deliver set the tokens per second.
- Power. A phone can spend a few watts, a car computer tens of watts, an H100 up to 700 W. Moving a byte from memory costs much more energy than one multiply-add, so the power budget is mostly a budget for moving bytes.
The efficient-inference survey of Zhou et al. has a short section on hardware accelerator design (its section VI-C). It names FPGA designs such as FACT, ALLO, DFX and FlightLLM, and reports that FlightLLM gets 6.0× higher energy efficiency and 1.8× better cost efficiency than an NVIDIA V100S for Llama2-7B, and 1.2× higher decode throughput than an A100 on a newer FPGA. [Certain] Chapter 7 of the VLM book ends with edge deployment: MLX, llama.cpp, mobile runtimes and PEFT adapters. [Certain]
This session connects the two worlds. First we build one simple model of any device (capacity, bandwidth, compute, power). Then we look at the main kinds of accelerators (GPU, TPU-style systolic arrays, FPGAs, NPUs and SRAM-heavy chips), study FlightLLM in depth, and finish with the runtimes people really use at the edge and the survey techniques that matter most there. The previous session, session 139, was about serving software on servers; the next one, session 141, turns these numbers into capacity plans.
One model for every device: capacity, bandwidth, compute, power
We can describe any device for LLM inference with four numbers: memory capacity \(C\) (GB), memory bandwidth \(B\) (GB/s), peak compute \(F\) (FLOP/s or integer operations per second), and power \(P\) (W). With these four numbers we can answer most deployment questions on paper, before buying hardware.
Does it fit?
The model needs memory for its weights and for the KV cache (session 80):
- Read the symbols
- \(N\)
- number of weights (parameters)
- \(b_w, b_{kv}\)
- bits per weight and bits per KV-cache number (16 for BF16; 4.5 for llama.cpp’s Q4_0 format, which stores a 16-bit scale per block of 32 four-bit weights)
- \(L, h_{kv}, d_h\)
- number of layers, number of key-value heads, size of each head
- \(s\)
- number of tokens in the context; the factor 2 is for keys and values
Example: Llama 3.1 8B has 8.03 billion weights, 32 layers and 8 KV heads of size 128. At 4.5 bits the weights take 4.52 GB. The BF16 KV cache needs \(2 \times 32 \times 8 \times 128 \times 2 = 131{,}072\) bytes (128 KiB) per token, so 0.54 GB for 4,096 tokens. Total: 5.05 GB.
How fast can it decode?
At batch size 1, each decode step reads all weights and the whole KV cache once. If memory is the bottleneck, the time per token is at least
where \(W_{\text{read}}\) is the bytes of weights read per token. (For a model with a separate output head this is all weights except the input embedding table, from which only one row is read. For a model with tied embeddings, the same matrix is read as the output head, so it is all weights.) This is an upper bound on speed: real systems reach a fraction \(\eta\) of \(B\), often 50% to 80%. [Likely]
How fast can it read the prompt?
The prefill phase processes all \(n\) prompt tokens at once. Each weight is used for \(n\) tokens, so it is limited by compute, not memory:
(2 FLOPs per weight per token, one multiply and one add; attention adds a term that grows with \(n^2\), small for short prompts.)
Which one is the limit? The ridge point
The arithmetic intensity of a piece of work is the number of operations it does per byte it reads from memory. The device has its own ratio, the ridge point of the roofline model (session 82):
If \(I_{\text{work}} \lt I^{*}\), the work is memory-bound; if \(I_{\text{work}} \gt I^{*}\), it is compute-bound. Decode at batch size 1 with 4.5-bit weights has \(I = 16/4.5 = 3.56\) FLOP/byte. An H100 SXM has about 989 dense BF16 TFLOP/s (NVIDIA lists 1,979 with 2:4 sparsity) and 3.35 TB/s, so \(I^{*} \approx 295\). [Certain] A phone NPU, with the effective numbers we derive below, has \(I^{*} \approx 240\). Decode is 67 to 83 times below the ridge on these two. A prompt of 829 tokens at 4.5 bits has \(I \approx 2 \cdot 829 \cdot 8 / 4.5 \approx 2{,}950\), far above. So, on every device:
- decode speed is set by memory bandwidth (bytes per token / \(B\)),
- prefill speed is set by compute (FLOPs / \(F\)),
- and the model must fit in capacity \(C\).
Figure 1 applies equations 140.1 and 140.2 to seven real devices.
How to read Figure 1
Click a row to see its numbers. With Llama 3.1 8B at 4.5 bits, the model needs 5.05 GB: it does not fit on the phone (4 GB usable), but it fits on the Jetson Orin Nano, where the bound is 21.4 tokens/s. The same model on an H100 is bounded at 704 tokens/s: the ratio is just the ratio of bandwidths, 3,350/102. Switch to BF16: the weight bytes grow 3.6× (the KV cache does not change), so every speed drops about 3.3× at this context, and the model fits on fewer devices. Raise the context to 32K: the KV cache grows to 4.3 GB and pushes the model out of the small devices. Choose the 70B model: only the Mac, the Jetson AGX and the H100 can hold it at 4.5 bits, and even the Mac is bounded at 13.5 tokens/s.
Apple described its 2024 on-device language model: about 3 billion weights, compressed with a mixed 2-bit and 4-bit scheme to an average of 3.7 bits per weight; on an iPhone 15 Pro, a time to first token of about 0.6 ms per prompt token and a generation rate of 30 tokens/s. [Certain]
Memory: \(3 \times 10^9 \times 3.7 / 8 = 1.39\) GB of weights.
Effective bandwidth: \(30\ \text{tokens/s} \times 1.39\ \text{GB} = \mathbf{41.6}\ \textbf{GB/s}\). If the phone’s memory delivers 51.2 GB/s at peak [Likely], that is 81% of peak: decode runs close to the bandwidth limit, as equation 140.2 predicts.
Effective compute: each prompt token costs \(2 \times 3 \times 10^9 = 6 \times 10^9\) FLOPs; in 0.6 ms that is 10 TFLOP/s (or 10 trillion integer operations per second). A 1,000-token prompt takes about 0.6 s.
Ridge point: \(10^{13} / 41.6 \times 10^{9} \approx 240\) operations per byte. Decode at 3.7 bits has intensity \(16/3.7 = 4.3\). So decode uses only a few percent of the NPU’s compute; it waits for memory.
We will reuse these two effective numbers (41.6 GB/s and 10 TFLOP/s) as our “phone” in Figure 4.
The accelerator zoo
An accelerator is a chip designed to run some kind of work much faster or with less energy than a general CPU. For LLMs, five families matter. They differ in where the weights live, how the math units are arranged, and how flexible they are.
| Family | Math units | Where the weights live | Strength | Weakness |
|---|---|---|---|---|
| GPU (NVIDIA, AMD) | Thousands of simple cores plus tensor cores (small matrix units) (session 42) | HBM (data center) or GDDR (consumer), next to the chip | Flexible, huge software ecosystem, very high compute and bandwidth | High power and price; low-bit and irregular formats need special kernels |
| TPU-style systolic array (Google TPU and many NPUs) | A large grid of multiply-accumulate units that pass data to neighbours | HBM (TPU) or the device’s DRAM (NPU) | Very efficient dense matrix multiply | Poor use of the grid for skinny shapes (decode, batch 1); fixed data types |
| FPGA (AMD/Xilinx Alveo, Versal) | Reconfigurable logic and thousands of DSP blocks | HBM and DDR on the card; on-chip SRAM blocks | Custom data paths for sparsity and odd bit widths; low power | Low clock (hundreds of MHz), hard to program, long compile times |
| Mobile and laptop NPU (Apple Neural Engine, Qualcomm Hexagon, and others) | Fixed matrix engines for INT8/INT4/FP16 | The device’s shared LPDDR memory | Best energy per operation on the device; frees the CPU and GPU | Static shapes, limited operators, vendor-specific tools |
| SRAM-heavy chips (Groq LPU, Cerebras wafer-scale engine) | Many matrix units with huge on-chip SRAM | On-chip SRAM, spread over many chips | Very high bandwidth per chip, so very fast decode | SRAM is small per chip: a large model needs many chips [Likely] |
All five follow the same physics: moving data costs more than computing on it. Han et al. give the classic table for a 45 nm chip: a 32-bit float multiply costs 3.7 pJ, a 32-bit float add 0.9 pJ, a 32-bit read from on-chip SRAM 5 pJ, and a 32-bit read from off-chip DRAM 640 pJ. [Certain] One DRAM read costs as much energy as about 170 float multiplies. Newer chips and memories use less energy per operation, but the gap between moving and computing remains large. [Likely] Each family is a different answer to this one problem.
Systolic arrays: how TPUs and many NPUs multiply matrices
A systolic array is a grid of \(R \times C\) small processing elements (PEs). Each PE can do one multiply-accumulate (MAC) per clock cycle. Data enters at the edges and moves one PE per cycle, like blood pumped through the body (hence “systolic”). Each number read from memory is used by many PEs, so the array needs little memory bandwidth per operation. The first Google TPU had a \(256 \times 256\) grid: 65,536 8-bit MAC units with a peak of 92 trillion operations per second, and 28 MiB of on-chip memory. [Certain]
In the output-stationary version, PE \((i, j)\) keeps the running sum \(C_{ij}\). Row \(i\) of \(A\) enters from the left, delayed by \(i\) cycles; column \(j\) of \(B\) enters from the top, delayed by \(j\) cycles. Then at cycle \(t\), PE \((i,j)\) receives exactly \(A_{ik}\) and \(B_{kj}\) for the same \(k\):
- Read the symbols
- \(M, K, N\)
- shapes: \(A\) is \(M \times K\), \(B\) is \(K \times N\), the output \(C\) is \(M \times N\)
- \(t\)
- clock cycle, starting at 0; PE \((i,j)\) does a MAC when \(0 \le k \lt K\)
- \(T\)
- cycles until the last PE finishes (the wave needs \(M - 1 + N - 1\) extra cycles to cross the grid)
- \(U\)
- utilization: useful MACs divided by MACs the \(R \times C\) array could do in \(T\) cycles
- input: \(A\) (\(M \times K\)), \(B\) (\(K \times N\)), array of \(R \times C\) PEs with \(M \le R\), \(N \le C\)
- set every accumulator \(C_{ij} \leftarrow 0\)
- for \(t = 0\) to \(M + N + K - 3\) do
- left edge: PE \((i, 0)\) receives \(A_{i,\,t-i}\) if \(0 \le t - i \lt K\), else 0
- top edge: PE \((0, j)\) receives \(B_{t-j,\,j}\) if \(0 \le t - j \lt K\), else 0
- every PE in parallel: \(C_{ij} \leftarrow C_{ij} + a \cdot b\); pass \(a\) to the right neighbour and \(b\) to the neighbour below // one cycle
- return \(C\) // read the accumulators out of the grid
Step through Figure 2. Then switch to “decode (1 row)” and see the main weakness of a big matrix engine for LLM decode.
How to read Figure 2
At cycle 0 only the top-left PE works: it multiplies \(A_{00} = 1\) by \(B_{00} = 2\). The work moves as a diagonal wave. At cycle 2 six PEs work (all with \(i + j \le 2\)); at cycle 6 only the bottom-right one. After 7 cycles (\(3 + 3 + 3 - 2\)) the grid holds \(C = [[4,1,6],[9,7,5],[2,2,4]]\), and 27 of the 63 possible MACs were done (43%). For a big \(K\) the start and end waves matter less and utilization goes near 100%. In decode mode only the first row of PEs ever works, because \(A\) has one row (one token): 9 of 45 possible MACs, 20%. A \(128 \times 128\) array doing a decode GEMV with \(K = 4096\) would be below 1% busy.
This is why NPUs that are excellent at prefill are often idle during decode: a single token gives a matrix-vector product, and a matrix-vector product reads each weight for only one MAC. Decode needs bandwidth, not a big grid. The fix on the software side is more rows: batching (session 84), or several tokens per weight read, as in speculative decoding (session 128).
FPGAs and FlightLLM
An FPGA (field-programmable gate array) is a chip whose logic can be rewired after it is made. It contains lookup tables (LUTs, small programmable logic), flip-flops, small SRAM blocks (BRAM and the larger URAM on AMD/Xilinx chips) and hard DSP blocks (fixed multiply-add units; on Xilinx chips they are called DSP48). You describe a circuit, and the tools map it onto these parts. The circuit runs slowly (FlightLLM runs at 225 MHz, against about 1 to 2 GHz for GPUs), but it does exactly what you designed, with no wasted work. [Certain]
Why try FPGAs for LLMs? FlightLLM (Zeng et al., FPGA 2024) points at three problems GPUs have with compressed LLMs at batch size 1: [Certain]
- Sparsity. GPUs speed up only one structured pattern, 2:4 sparsity (2 non-zero weights in every group of 4; see session 114). The paper notes that 75% unstructured sparsity gives almost no end-to-end speed-up on GPUs.
- Mixed precision. Using 2, 3, 4 and 8 bits in different parts of the model saves memory, but GPU memory access and SIMD units want all weights in the same format.
- Bandwidth use. In decode, existing designs move small activation vectors to and from off-chip memory again and again, and use only 29% to 43% of the memory bandwidth.
FlightLLM answers with three ideas.
1. A configurable sparse DSP chain
DSP blocks are fastest when they are chained: each one adds its product to the sum coming from its neighbour, with no extra logic. But a fixed chain assumes dense data. FlightLLM splits a long chain into small DSP groups and adds a sparse multiplexer (a switch that picks which input goes into each DSP), so that only non-zero weights are multiplied. A configurable path between groups and a reduction node let the same hardware serve dense and sparse matrices. Each DSP48 packs two INT8 MACs. [Certain]
The sparsity pattern is N:M: in every block of \(m\) weights, only \(n\) are kept (we use lower-case letters, because \(N\) is already the number of weights). FlightLLM uses \(m = 16\) and \(n \in \{0, 2, 4, 8, 16\}\), so different blocks can have different sparsity, but each block is regular enough for hardware. In N:M mode a chain produces \(n\) outputs per cycle instead of one. [Certain] The paper reports 1.6× higher computation efficiency with block-wise and N:M sparsity. [Certain] Storage per original weight, if each kept value of \(b\) bits needs a \(\log_2 m\)-bit index:
For 2:4 sparsity with 4-bit values: \(\tfrac{1}{2}(4 + 2) = 3\) bits per original weight. For 8-bit values: 5 bits. The index bits matter: sparsity saves less memory than it seems.
2. Always-on-chip decode with mixed precision
In decode, the activations are vectors (one token), small enough to stay in on-chip SRAM. FlightLLM fuses all layers of one decode step: the output of one layer stays in the on-chip buffer as the input of the next, and only the final result goes back to off-chip memory. Element-wise operations (SiLU, additions) run right after the matrix-vector product that feeds them; Softmax and LayerNorm wait for a full vector. [Certain] This raises the off-chip HBM bandwidth use from 35.6% to 65.9%. [Certain]
The weights use mixed precision: a gradient-based importance score gives each weight group 3, 4 or 5 bits, for an average of 3.5 bits per weight with 8-bit activations. A dequantization unit expands the packed weights to INT8 just before the DSPs. [Certain] Large one-time reads (weights, KV cache) come from HBM (high bandwidth); small, frequent reads come from DDR (lower latency). [Certain]
- input: token embedding \(\mathbf{x}\) in the on-chip buffer; weights (mixed 3/4/5-bit, N:M sparse) and KV cache in HBM
- for layer \(\ell = 1\) to \(L\) do
- stream \(W_Q, W_K, W_V\) of layer \(\ell\) from HBM; dequantize to INT8; sparse MV on the DSP chain // activations never leave the chip
- append \(\mathbf{k}, \mathbf{v}\) to the KV cache in HBM; stream the cache and compute attention
- Softmax, norm, SiLU and adds in the special function unit, fused with the MVs
- stream the FFN weights; sparse MV; residual add on chip
- write only the final logits (or the next token) to off-chip memory
3. Length-adaptive compilation
An FPGA accelerator runs a list of instructions made by a compiler. Each prompt length and each decode position needs different instructions. For Llama2-7B on the U280, the paper estimates about 1.67 TB of instructions to cover all lengths from 1 to 2,048 tokens: far more than the card’s 32 GB of DDR. FlightLLM lets nearby lengths share one instruction file (for example, lengths 1 to 16 use the file for 16), with finer steps in decode, where memory traffic grows with length. This brings the size to 4.77 GB; merging the load and store instructions for 8 HBM channels into one brings it to 3.25 GB, about 500× smaller. [Certain] NPUs face the same problem: they like static shapes, and a growing KV cache is not static (Apple, for example, mentions work to “enable efficient Key-Value (KV) cache update” on its neural engines). [Certain]
What FlightLLM measured
| Platform | Compute units | Memory | Bandwidth |
|---|---|---|---|
| NVIDIA V100S (12 nm) | 640 tensor cores | 32 GB | 1,134 GB/s |
| NVIDIA A100 (7 nm) | 432 tensor cores | 80 GB | 1,935 GB/s |
| Xilinx Alveo U280 (16 nm) | 9,024 DSPs | 8 GB HBM + 32 GB DDR | 460 + 38 GB/s |
| Xilinx Versal VHK158 (7 nm) | 7,392 DSPs | 32 GB HBM + 32 GB DDR | 819 + 51 GB/s |
(Table 2 of the paper. [Certain]) The U280 results are measured on real cards; the VHK158 results come from a cycle-accurate simulator. On Llama2-7B and OPT-6.7B at batch size 1, FlightLLM on the U280 has lower end-to-end latency than the V100S with vLLM and SmoothQuant, 6.0× higher energy efficiency than the V100S and 4.2× higher than the A100 (tokens per joule, on average), and 1.8× and 1.4× better cost efficiency (tokens per second per dollar). The simulated VHK158 has 1.2× higher throughput than an A100. Bandwidth use in decode is 65.9% on the U280 and 64.8% on the VHK158, against 42.5% (naive) and 65.5% (optimized) on the V100S, and 28.6% and 57.4% on the A100. [Certain]
Figure 1 of the paper summarises: the V100S runs at about 45 tokens/s at about 220 W and costs about $12K; the U280 runs at about 55 tokens/s at about 45 W and costs about $8K. [Certain]
Tokens per joule: \(45 / 220 = 0.205\) for the GPU, \(55 / 45 = 1.22\) for the FPGA. Ratio: \(1.22 / 0.205 = 5.98 \approx 6.0\times\).
Tokens per second per dollar: \((55 / 8000) / (45 / 12000) = 1.83 \approx 1.8\times\).
So most of the gain is in power (45 W against 220 W), not in speed (55 against 45 tokens/s). An FPGA at 225 MHz is not a faster chip; it wastes less energy on bytes and on work it does not need.
Energy: the real budget at the edge
A phone does not have a fan, and its battery holds a fixed number of joules. So on the edge the most useful unit is tokens per joule, or its inverse, energy per token:
- Read the symbols
- \(\text{bytes}_{\text{DRAM}}, \text{bytes}_{\text{SRAM}}\)
- bytes of weights read per token from off-chip memory and from on-chip SRAM
- \(e_{\text{DRAM}}, e_{\text{SRAM}}\)
- energy per byte read: 160 pJ and 1.25 pJ in the 45 nm table (640 pJ and 5 pJ per 32 bits)
- \(e_{\text{MAC}}\)
- energy per multiply-add: 4.6 pJ for a 32-bit float multiply plus add, an upper bound; low-bit integer math costs less
- \(P\)
- power the device can spend on the model (a few watts on a phone)
Han et al. used the same table to make the point for mobile devices: a network with 1 billion connections running 20 times per second needs \(20 \times 10^9 \times 640\ \text{pJ} = 12.8\ \text{W}\) just for DRAM access. [Certain] Figure 3 does the same for an LLM.
How to read Figure 3
With the 8B model at 4.5 bits, all weights from DRAM: 0.675 J for DRAM reads and 0.037 J for arithmetic per token, 0.712 J in total, so 1.40 tokens per joule; a 5 W budget allows at most 7.0 tokens/s. DRAM is 95% of the energy. Lower the bits: the DRAM part shrinks in proportion, the MAC part does not. Move SRAM to 100% (like Groq, Cerebras or FlightLLM’s on-chip buffers, if the model could fit): the DRAM part disappears and the energy drops to 0.042 J per token. Now arithmetic dominates. Lower the DRAM energy to 20 pJ/byte: DRAM still dominates for the 8B model.
Three lessons follow. First, bits per weight is an energy knob, not only a memory knob: 4.5-bit weights cut energy per token by about 3.4× against BF16 for the 8B model. Second, the model size you can afford on a phone is limited by power as much as by memory: a 3B model needs about 2.3 times less energy per token than an 8B model. Third, sustained power is lower than peak power: a phone that gets hot slows its chips down (thermal throttling), so a benchmark of the first 10 seconds can overstate the speed of a long session. [Likely]
Edge runtimes: llama.cpp, MLX and mobile stacks
A runtime is the software that loads a model file and runs it on a device. On servers we used vLLM, TensorRT-LLM and similar engines (session 131). On the edge, different tools win, because the goals are different: batch size 1, small memory, many kinds of hardware, and no Python in the final app.
llama.cpp and GGUF
llama.cpp aims to “enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware”. It is a plain C/C++ program “without any dependencies”; it treats Apple silicon as “a first-class citizen” (ARM NEON, Accelerate and Metal), supports AVX, AVX2, AVX512 and AMX on x86, and offers 1.5-bit to 8-bit integer quantization. [Certain] Its list of backends includes CUDA (NVIDIA), HIP (AMD), Metal (Apple), Vulkan (most GPUs), SYCL (Intel GPUs), OpenCL for Adreno phone GPUs, Hexagon for Snapdragon, CANN for Huawei Ascend NPUs, and WebGPU. [Certain] Models are stored in one GGUF file: weights in a chosen quantization, plus the tokenizer and the chat template. It can split a model between CPU and GPU when the GPU is too small (session 130). [Certain]
MLX and unified memory
MLX is Apple’s “array framework for machine learning on Apple silicon”. Its API follows NumPy; arrays “live in shared memory”, so operations can run on the CPU or the GPU “without transferring data”; computation is lazy (“arrays are only materialized when needed”). [Certain] Apple chips have unified memory: CPU, GPU and Neural Engine share one pool of LPDDR memory. This removes the PCIe bottleneck of the previous session: there is nothing to offload. A MacBook Pro with an M4 Max has up to 128 GB of unified memory at over half a terabyte per second, so a 70B model at 4 bits fits (Figure 1). [Certain] The trade-off: 546 GB/s is less than a fifth of an H100’s bandwidth, so decode is several times slower. The mlx-lm package adds loading, quantization, generation and LoRA fine-tuning of LLMs on top of MLX. [Likely]
Mobile stacks
- ExecuTorch: PyTorch’s “on-device AI across mobile, embedded and edge” runtime. The model is exported ahead of time and parts of the graph are handed (“delegated”) to backends: XNNPACK on CPUs, Vulkan GPUs, Qualcomm, MediaTek, Samsung Exynos and Arm accelerators on Android, Core ML on Apple devices, and Arm Ethos-U and Cortex-M on microcontrollers. [Certain]
- MLC LLM: “a machine learning compiler and high-performance deployment engine for large language models”. It compiles one model for Metal (Mac, iPhone), Vulkan, CUDA, ROCm, OpenCL (Android Adreno and Mali GPUs), and WebGPU in the browser. [Certain]
- Vendor stacks: Core ML on Apple devices, Qualcomm’s AI Engine tools for the Hexagon NPU, Google’s LiteRT (formerly TensorFlow Lite) and its LLM inference API, and ONNX Runtime. The VLM book also shows transformers.js for running models in the browser. [Likely]
Adapters: one base model, many tasks
A phone cannot store a separate 3B model for each feature. The usual answer is one frozen base model plus small LoRA adapters (session 57) that are swapped at run time. Apple’s on-device model does this: adapters of rank 16, stored in 16 bits, take “10s of megabytes” each, against about 1.4 GB for the base model. [Certain] The VLM book recommends the same pattern for edge VLMs. [Certain]
- input: device \((C, B, F, P)\), latency targets for time to first token and tokens/s, typical prompt (text and image tokens) and output length
- pick the largest model whose quality is good enough; compute memory with equation 140.1 at 4 bits; if it does not fit in usable memory, pick a smaller model or a lower bit width
- check decode: \(\eta B / (\text{bytes per token}) \ge\) target tokens/s (equation 140.2) // use \(\eta \approx 0.7\) to 0.8
- check prefill: \(\text{FLOPs} / (\eta F) \le\) target time to first token (equation 140.3); for VLMs, count image tokens and the vision encoder
- check energy: \(P / E_{\text{token}}\) (equation 140.7) and the sustained (not peak) power
- if prefill is too slow: fewer visual tokens, shorter prompts, prefix caching; if decode is too slow: fewer bits, speculative decoding, smaller model
- choose a runtime that supports the device’s fastest unit (NPU, GPU) and the model’s operators; export, quantize, then measure on the real device, including after several minutes (heat)
Which survey techniques matter most at the edge
The survey sorts techniques into data-level, model-level and system-level (session 83). At the edge, the batch is 1, memory is small, and power is tight. That changes the ranking a lot. Our ranking below is a judgment based on the bandwidth and energy math above, not a measured result. [Likely]
| Technique (survey level) | Edge value | Why |
|---|---|---|
| Weight quantization to 4 bits or less (model) | Highest | Cuts memory, decode time and energy per token in proportion to bits (140.1, 140.2, 140.7) |
| Small models: distillation, pruning, efficient design (model) | Highest | Fewer weights means fewer bytes for everything; see session 119 and MobileLLM |
| Input compression, including fewer visual tokens (data) | High for VLMs | Prefill is compute-bound on the edge; image and video tokens dominate it (Figure 4) |
| KV-cache quantization and efficient attention (GQA) (model) | High for long context | The KV cache competes with weights for small memory (session 110) |
| Speculative decoding (system) | Medium to high | Decode is far below the ridge, so verifying several tokens per weight read is almost free; but a draft model costs memory |
| Operator fusion and graph optimization (system) | Medium | Fewer memory round trips, as in FlightLLM’s on-chip decode (session 127) |
| N:M sparsity (model) | Depends on hardware | Helps only if the chip has sparse units; index bits reduce the memory saving (140.6) |
| Offloading to flash (system) | Niche | For models larger than DRAM; flash is slow (session 130) |
| Continuous batching, paged KV memory, distributed serving (system) | Low | Built for many users on a server; one user on a phone has nothing to batch |
Speculative decoding needs one more formula. With draft length \(\gamma\), acceptance rate \(\alpha\) and a draft that costs a fraction \(c\) of a target pass (session 128):
With \(\alpha = 0.7\), \(\gamma = 4\) and a light draft head (\(c = 0.1\)): \(2.77 / 1.4 = 1.98\times\). With Llama 3.2 1B as the draft for Llama 3.2 3B, \(c\) is about 0.38 (the ratio of their bytes), and the speed-up falls to about 1.1×. On the edge, the draft must be tiny.
For a VLM, the time to first token is the vision encoder plus the language model’s prefill:
- Read the symbols
- \(N_{\text{enc}}, P_{\text{img}}\)
- vision encoder weights; number of image patches it processes (729 for one \(384 \times 384\) image with 14-pixel patches)
- \(n_{\text{vis}}\)
- visual tokens passed to the language model, after any merging or pruning (session 183)
- \(L, d\)
- layers and attention width of the language model; the \(s^2\) term is causal attention in prefill
- \(F_{\text{eff}}\)
- effective compute of the device (10 TFLOP/s for our phone)
Figure 4 puts the techniques together for a VLM on a phone: a 3B language model (Llama 3.2 3B shape) with a 0.4B vision encoder (SigLIP so400m size, kept in 16 bits), and the effective phone numbers from the worked example.
How to read Figure 4
Defaults (1 image, 4.5-bit weights): 729 visual tokens, time to first token 0.60 s, decode 21.8 tokens/s, 2.71 GB. Switch to BF16: 7.33 GB, over budget, and decode falls to 6.4 tokens/s; weight quantization is the first thing to do. Switch to 8 video frames: 5,832 visual tokens and 4.88 s before the first word. Now keep 25% of the visual tokens: 1.51 s. Token reduction changes decode speed only a little, but prefill a lot. Turn on speculative decoding: decode doubles to about 43 tokens/s, prefill does not change. Each technique fixes one bottleneck, so the order depends on which bottleneck hurts.
PyTorch: a systolic array, N:M sparsity and an edge budget
The first program simulates Algorithm 1 cycle by cycle and checks the result against A @ B. It also measures utilization for a full tile and for a decode GEMV on a \(16 \times 16\) array (equation 140.5).
import torch torch.manual_seed(0) def systolic_matmul(A, B, array=(16, 16)): """Output-stationary systolic array: PE(i,j) keeps C[i,j]. Row i of A enters from the left, delayed by i cycles; column j of B enters from the top, delayed by j cycles. Values move one PE per cycle.""" M, K = A.shape K2, N = B.shape assert K == K2 C = torch.zeros(M, N, dtype=A.dtype) a_reg = torch.zeros(M, N, dtype=A.dtype) # value each PE passes to the right b_reg = torch.zeros(M, N, dtype=A.dtype) # value each PE passes down cycles, busy = M + N + K - 2, 0 for t in range(cycles): a_in = torch.zeros(M, N, dtype=A.dtype) b_in = torch.zeros(M, N, dtype=A.dtype) a_in[:, 1:] = a_reg[:, :-1] # shift right b_in[1:, :] = b_reg[:-1, :] # shift down for i in range(M): # feed the left edge k = t - i a_in[i, 0] = A[i, k] if 0 <= k < K else 0 for j in range(N): # feed the top edge k = t - j b_in[0, j] = B[k, j] if 0 <= k < K else 0 C += a_in * b_in # every PE does one MAC i_idx = torch.arange(M)[:, None]; j_idx = torch.arange(N)[None, :] k_now = t - i_idx - j_idx # which k reaches PE(i,j) now busy += int(((k_now >= 0) & (k_now < K)).sum()) a_reg, b_reg = a_in, b_in assert M <= array[0] and N <= array[1] # the tile must fit the array return C, cycles, busy / (array[0] * array[1] * cycles) A = torch.randint(-3, 4, (4, 6)) B = torch.randint(-3, 4, (6, 5)) C, cycles, util = systolic_matmul(A, B, array=(4, 5)) print("equal to A @ B:", torch.equal(C, A @ B), "| cycles:", cycles, "| PE utilization:", f"{util:.1%}") C, cycles, util = systolic_matmul(torch.randint(-3, 4, (16, 512)), torch.randint(-3, 4, (512, 16))) print("batch of 16 rows, K=512 -> cycles:", cycles, "| PE utilization:", f"{util:.1%}") C, cycles, util = systolic_matmul(torch.randint(-3, 4, (1, 512)), torch.randint(-3, 4, (512, 16))) print("decode GEMV, 1 row, K=512 -> cycles:", cycles, "| PE utilization:", f"{util:.1%}")
Output (PyTorch 2.8 on CPU):
equal to A @ B: True | cycles: 13 | PE utilization: 46.2% batch of 16 rows, K=512 -> cycles: 542 | PE utilization: 94.5% decode GEMV, 1 row, K=512 -> cycles: 527 | PE utilization: 6.1%
The array is exact. With a full 16-row tile and a long \(K\), it is 94.5% busy; with one row (one decode token), 6.1%. The second program is FlightLLM’s compute idea in miniature: prune to 2:4, store only the kept values and their positions, compute a matrix-vector product with a “sparse mux”, and add low-bit quantization (equation 140.6).
import math, torch torch.manual_seed(0) def prune_n_m(W, n=2, m=4): """Keep the n largest-magnitude weights in every group of m (along a row).""" rows, cols = W.shape g = W.reshape(rows, cols // m, m) idx = g.abs().topk(n, dim=-1).indices.sort(dim=-1).values # positions kept, 0..m-1 vals = torch.gather(g, -1, idx) return vals, idx # compressed form def sparse_matvec(vals, idx, x, m=4): """What one N:M lane does: pick n of every m inputs with a mux, then MAC.""" rows, groups, n = vals.shape xg = x.reshape(groups, m) # inputs grouped like the weights picked = torch.gather(xg.expand(rows, groups, m), -1, idx) # the "sparse mux" return (vals * picked).sum(dim=(-1, -2)), rows * groups * n # result, MACs done def quant_sym(v, bits): """Symmetric per-row integer quantization; returns dequantized values.""" q = 2 ** (bits - 1) - 1 s = v.abs().amax(dim=(-1, -2), keepdim=True) / q return torch.round(v / s).clamp(-q, q) * s W = torch.randn(64, 256) x = torch.randn(256) vals, idx = prune_n_m(W) W_sparse = torch.zeros_like(W).reshape(64, 64, 4).scatter(-1, idx, vals).reshape(64, 256) y, macs = sparse_matvec(vals, idx, x) print("2:4 sparse matvec equals masked dense:", torch.allclose(y, W_sparse @ x, atol=1e-5), "| MACs:", macs, "of", W.numel()) for bits in (8, 4, 3): yq, _ = sparse_matvec(quant_sym(vals, bits), idx, x) err = ((yq - y).norm() / y.norm()).item() stored = 0.5 * (bits + math.log2(4)) # n/m * (value bits + index bits) print(f"2:4 + INT{bits}: {stored:.1f} stored bits per original weight, rel. error vs 2:4 FP32 {err:.3f}")
Output:
2:4 sparse matvec equals masked dense: True | MACs: 8192 of 16384 2:4 + INT8: 5.0 stored bits per original weight, rel. error vs 2:4 FP32 0.007 2:4 + INT4: 3.0 stored bits per original weight, rel. error vs 2:4 FP32 0.119 2:4 + INT3: 2.5 stored bits per original weight, rel. error vs 2:4 FP32 0.255
Half the MACs, exact result. The error column compares only quantized against unquantized sparse weights on random data; a real model’s quality must be measured on tasks. Per-row scales are crude; FlightLLM chooses 3, 4 or 5 bits per group by importance to keep 3-bit damage where it hurts least. The third program is the planner behind Figure 4 (equations 140.2, 140.8 and 140.9), in plain Python:
# An edge VLM budget: 3B language model (Llama 3.2 3B shape) + 0.4B vision encoder # on a phone. F and BW are *effective* numbers derived from Apple's reported # on-device results (0.6 ms per prompt token, 30 tokens/s for ~3B at 3.7 bits). N, L, KVH, HD, D = 3.2127e9, 28, 8, 128, 3072 F, BW, BUDGET_GB = 1.0e13, 41.6e9, 4.0 N_ENC, PATCHES, TXT, OUT = 0.4e9, 729, 100, 100 def plan(frames=1, keep=1.0, w_bits=4.5, kv_bits=16, spec=False, alpha=0.7, gamma=4, c=0.1): nv = round(PATCHES * frames * keep) # visual tokens the LM sees s = nv + TXT # prompt length flops = 2 * N_ENC * PATCHES * frames + 2 * N * s + 2 * L * s * s * D ttft = flops / F # prefill is compute-bound w = N * w_bits / 8 kv_tok = 2 * L * KVH * HD * kv_bits / 8 t_tok = (w + kv_tok * (s + OUT / 2)) / BW # decode is bandwidth-bound speed = 1.0 if spec: # draft cost c, acceptance alpha speed = (1 - alpha ** (gamma + 1)) / (1 - alpha) / (gamma * c + 1) mem = (w * (1.1 if spec else 1) + 2 * N_ENC + kv_tok * (s + OUT)) / 1e9 return nv, round(ttft, 2), round(speed / t_tok, 1), round(mem, 2), mem <= BUDGET_GB print("frames keep bits kv spec -> visual tokens, TTFT s, tokens/s, memory GB, fits") for cfg in [(1, 1.0, 16, 16, False), (1, 1.0, 4.5, 16, False), (8, 1.0, 4.5, 16, False), (8, 0.25, 4.5, 8, False), (8, 0.25, 4.5, 8, True)]: print(cfg, "->", plan(*cfg))
Output:
frames keep bits kv spec -> visual tokens, TTFT s, tokens/s, memory GB, fits (1, 1.0, 16, 16, False) -> (729, 0.6, 6.4, 7.33, False) (1, 1.0, 4.5, 16, False) -> (729, 0.6, 21.8, 2.71, True) (8, 1.0, 4.5, 16, False) -> (5832, 4.88, 16.7, 3.3, True) (8, 0.25, 4.5, 8, False) -> (1458, 1.51, 21.9, 2.7, True) (8, 0.25, 4.5, 8, True) -> (1458, 1.51, 43.4, 2.88, True)
Read the rows from top to bottom as a deployment story: BF16 does not fit; 4.5 bits fits and gives about 22 tokens/s; video makes the first answer take almost 5 s; keeping a quarter of the visual tokens and 8-bit KV brings it to 1.5 s; speculative decoding doubles the decode speed.
Where this is used
| Application | Typical hardware and runtime | What limits it |
|---|---|---|
| Phone assistants (writing tools, summaries, smart replies) | NPU plus GPU of the phone; vendor runtimes; one ~3B base model with LoRA adapters | Memory (shared with apps), energy, heat; Apple reports 30 tokens/s for its ~3B model [Certain] |
| Local chat and coding assistants on laptops | llama.cpp with GGUF, MLX on Macs | Decode bandwidth: tokens/s ≈ bandwidth / model bytes (Figure 1) |
| Robots and drones (vision-language-action models) | Jetson-class modules (15–60 W), TensorRT or llama.cpp | Latency per control step and power; prefill of camera images |
| Smart cameras: on-device video classification, action recognition, key-moment detection | NPU in the camera or a Jetson; small vision encoder, VLM only on events | Visual tokens per frame, sustained power; run the VLM on few, selected frames |
| Private video search and Video-RAG on a phone or laptop | Embedding model on the NPU for indexing; small VLM to answer | Indexing energy (many frames) and time to first token for the answer (Figure 4) |
| Video fingerprinting and near-duplicate detection at the edge | Small encoders on NPUs, compact embeddings | Throughput per watt; no LLM decode needed |
| Low-latency cloud inference on special chips | FPGAs (FlightLLM-style), SRAM-heavy chips | Cost and power per token at batch size 1; model must fit on-chip or in HBM |
| Browser demos and offline web apps | WebGPU through MLC LLM or transformers.js | Browser memory limits and download size |
Common mistakes
- Comparing devices by TOPS for decode. Decode at batch size 1 is memory-bound (equation 140.4). A chip with 10× more TOPS and the same bandwidth decodes at the same speed. Compare bandwidth for decode and compute for prefill.
- Counting total RAM as usable. On a phone, the operating system and other apps keep a large part of memory, and the app can be killed if it uses too much. Plan with the usable memory, plus the KV cache, plus the vision encoder.
- Forgetting the vision side of a VLM. The vision encoder needs memory, and image or video tokens can make prefill take seconds. On the edge, visual token count often matters more than the language model size for time to first token.
- Using peak power and peak speed. Phones slow down when they get hot. Measure tokens/s after minutes of use, not only in the first seconds.
- Expecting sparsity to save what it promises. 50% sparsity does not halve memory: the indices cost bits (equation 140.6), and without sparse hardware units it may give no speed-up at all.
- Reading accelerator claims without the baseline. “6× more efficient” depends on the GPU generation, the software, the batch size and the compression of each side. FlightLLM compares 3.5-bit sparse weights on an FPGA with INT8 on a V100S.
- Using a big draft model for speculative decoding on a phone. A draft that costs 38% of the target leaves almost no gain (equation 140.8), and it takes memory.
- Assuming any runtime runs on the NPU. Many runtimes run on the CPU or GPU only, or fall back to the CPU for unsupported operators. Check which unit really runs each layer.
Interview questions
Can Llama 3.1 8B run on a phone with 8 GB of memory? Show the math.basic
Why is decode memory-bound on every device, from phone to H100?basic
What is a systolic array, and why is it inefficient for decode?core
Explain FlightLLM’s three main ideas.core
FlightLLM is 6× more energy efficient than a V100S. Where does that come from, and how would you challenge the comparison?advanced
What is the role of an NPU in a phone, and what are its limits for LLMs?core
Why does unified memory on Apple silicon change the offloading story?core
Which survey techniques would you apply first for an on-device VLM, and in which order?core
How do you estimate energy per token, and what does it tell you?advanced
When does N:M sparsity help at the edge, and how much memory does 2:4 really save?advanced
Speculative decoding on a phone: is it worth it?advanced
Exercises
- Llama 3.2 1B in Q8_0 (8.5 bits) with a 4,096-token BF16 KV cache on a Jetson Orin Nano (102 GB/s): memory needed and the decode bound?
- The Jetson AGX Orin lists 275 TOPS and 204.8 GB/s. What is its ridge point? Is decode at 4.5 bits compute-bound or memory-bound on it?
- A \(128 \times 128\) output-stationary systolic array runs a GEMV with \(K = 4096\) (\(M = 1\), \(N = 128\)). How many cycles and what utilization? And with \(M = 128\)?
- How many bits per original weight does 2:4 sparsity with 8-bit values need? And 8:16 sparsity with 4-bit values?
- With Figure 3’s defaults, compute the energy per token and the bound at 5 W for the 3B model at 4.5 bits. Then assume half of the weight reads come from SRAM.
- Check the 1.8× cost-efficiency claim of FlightLLM from its approximate prices and speeds.
- Speculative decoding with \(\alpha = 0.8\), \(\gamma = 4\), \(c = 0.1\): what speed-up does equation 140.8 give?
Answerscheck
Key takeaways
- Every device is four numbers: capacity, bandwidth, compute, power. Capacity decides if the model fits (140.1), bandwidth sets decode speed (140.2), compute sets prefill speed (140.3), and power bounds tokens per second through energy per token (140.7).
- Decode at batch size 1 is memory-bound everywhere: its intensity, \(16/b_w\), is far below every device’s ridge point (140.4). Compare devices by bandwidth for decode.
- Systolic arrays (TPUs, many NPUs) are superb for large matrix products but nearly idle for one-token decode (140.5).
- FlightLLM uses FPGA features for compressed LLMs: a configurable sparse DSP chain for N:M sparsity, always-on-chip decode with 3.5-bit mixed precision, and length-adaptive compilation; 6.0× energy efficiency over a V100S, mostly from lower power.
- Moving bytes dominates energy. Fewer bits, fewer weights, and keeping data on chip are the levers.
- At the edge, the most valuable techniques are small models, 4-bit weights, fewer visual tokens for VLMs, KV compression for long context, and speculative decoding with a tiny draft (140.8, 140.9). llama.cpp, MLX, ExecuTorch and MLC LLM are the common runtimes.
Key formulas
References for this session
- Zeng et al. (2024). FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs
- ggml-org (code). llama.cpp
- Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 7: Deploying Models for Inference at Scale
- Zhou, Ning, Hong et al. (2024). A Survey on Efficient Inference for Large Language Models
- Jouppi, Young, Patil, Patterson et al. (2017). In-Datacenter Performance Analysis of a Tensor Processing Unit
- Han, Pool, Tran & Dally (2015). Learning both Weights and Connections for Efficient Neural Networks
- Apple Machine Learning Research (2024). Introducing Apple’s On-Device and Server Foundation Models
- Apple (ml-explore) (code). MLX: an array framework for machine learning on Apple silicon
- PyTorch (code). ExecuTorch: on-device AI across mobile, embedded and edge for PyTorch
- MLC AI (code). MLC LLM: universal LLM deployment engine with ML compilation
- Liu, Zhao, Iandola, Lai, Tian, Fedorov, Xiong, Chang, Shi, Krishnamoorthi, Lai, Chandra (2024). MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Meta AI (2024). Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
- Zhai et al. (2023). Sigmoid Loss for Language Image Pre-Training