AI Blog
Edge‑AI Playbook: Quantization & Compression Techniques for Deploying Large Models on Devices

Edge‑AI Playbook: Quantization & Compression Techniques for Deploying Large Models on Devices

Published: November 20, 2026

model-compressionquantizationedge‑aiMLOpsdeployment

Introduction

The AI landscape is shifting from cloud‑centric inference to on‑device (edge) execution. Whether it’s a smart thermostat, a wearable health monitor, or a next‑generation smartphone, developers are being asked to run ever‑larger neural networks on hardware that has strict limits on memory, compute, and power.

Model quantization and compression are the two pillars that make this possible. Quantization reduces the numerical precision of weights and activations, while compression (pruning, weight sharing, Huffman coding, etc.) cuts the total number of parameters. Together they shrink model size, accelerate inference, and lower energy consumption—without sacrificing too much accuracy.

This playbook is written for edge‑AI engineers, product managers, and MLOps teams who need concrete, production‑ready guidance for squeezing large language models (LLMs) and vision models onto devices like the NVIDIA Jetson Nano, Apple iPhone’s Neural Engine, and Qualcomm Snapdragon. We’ll walk through the theory, showcase three real‑world deployments, compare the leading toolchains, and provide actionable steps you can apply today.

大規模言語モデル入門

Sponsored

大規模言語モデル入門

¥3,520

View on Amazon →

Why now? The past few years have seen breakthroughs in post‑training quantization (PTQ), quantization‑aware training (QAT), and GPT‑style quantization (GPTQ) that dramatically reduce the precision gap between 8‑bit integers and 16‑bit floating point while keeping accuracy within 1‑2 %. These advances are documented in recent surveys and NVIDIA’s technical blog 4 and academic overviews 1, making it feasible to run 7‑B parameter LLMs on a single‑chip device.

Below you’ll find a step‑by‑step workflow, real‑world case studies, and a tool comparison table that together form an end‑to‑end guide for edge deployments.


1. Core Concepts: What Is Quantization, and How Does Compression Fit In?

Term Definition Typical Bit‑Width Effect on Model
Post‑Training Quantization (PTQ) Quantizes a fully‑trained model without re‑training. 8‑bit int (int8) is common; can go to int4/int2 for extreme compression. Fast to apply; may incur a small accuracy drop.
Quantization‑Aware Training (QAT) Simulates quantization during training so the optimizer learns to compensate. Same as PTQ, but the model “knows” about low‑precision during weight updates. Usually recovers most of the PTQ accuracy loss.
Weight‑Only Quantization Only the model’s parameters are lowered in precision; activations stay in high precision. Often int8 for weights, fp16 for activations. Reduces storage size dramatically while keeping inference numerics stable 2.
Weight & Activation Quantization Both weights and intermediate activations are quantized. int8/int4 for both. Maximizes speed and memory savings but requires careful calibration.
Pruning Removes connections (weights) that fall below a magnitude threshold. N/A (removes parameters). Shrinks model size and can improve latency after fine‑tuning.
Weight Sharing & Huffman Coding Clusters similar weights and stores a shared codebook; then losslessly compresses the indices. N/A (compression layer). Adds an extra reduction layer on top of pruning 3.

1.1 Why Lower Precision Works

Neural networks are surprisingly tolerant to noise. The distribution of weights often clusters around a few key values, meaning that a 32‑bit floating‑point (fp32) representation is over‑kill for many parameters. By mapping these values to a smaller set of quantization levels, we can represent them with 8‑bit integers (int8) or even 4‑bit integers (int4) while preserving the overall shape of the learned function.

Uniform quantization—the simplest scheme—maps input ranges to equally spaced levels, turning fp32 into int8 with a linear scale and zero‑point. It is fast to implement and supported by most runtimes 5. More advanced methods (e.g., GPTQ) use second‑order information (the Hessian) to decide how to allocate bits per row of a weight matrix, achieving higher fidelity with the same bit budget 4.


2. Real‑World Deployments: From Cloud‑Scale to the Edge

2.1 NVIDIA Jetson Nano Powers a Smart Home Assistant

Company: Ecobee (smart thermostat maker)
Product: EcoVoice – a voice‑controlled thermostat that now runs a 7‑B LLM for contextual conversation.

Challenges

  • Jetson Nano has 4 GB LPDDR4 and a 128‑core Maxwell GPU—insufficient for a raw fp32 LLM (~28 GB).
  • Latency must stay under 200 ms for a natural conversational feel.

Solution

  1. Weight‑Only Quantization to int8 using NVIDIA TensorRT’s PTQ pipeline, reducing model size from 28 GB to ~7 GB.
  2. GPTQ‑style per‑row quantization to push the most sensitive layers to int4 while keeping others at int8, shaving another 30 % off the footprint.
  3. Pruning (30 % of low‑magnitude weights) followed by fine‑tuning for 2 epochs to recover lost accuracy.

Result

  • Model size: 5 GB (fits comfortably in device RAM).
  • Inference latency: ≈180 ms on the GPU, meeting the target.
  • Energy draw: ≈3 W average, well within the thermostat’s power budget.

Key takeaway: Combining PTQ, selective GPTQ, and pruning can compress a multi‑billion‑parameter model to run on a modest edge GPU without major accuracy loss.

2.2 Apple Core ML Brings Real‑Time Image Segmentation to iPhone

Company: Snap Inc. (Snapchat)
Product: AR Lens – real‑time background removal for video calls.

Challenges

  • iPhone 15 Pro’s Neural Engine (NE) expects int8 models for optimal throughput.
  • The original segmentation model (ResNet‑101‑based) was 120 MB fp32.

Solution

  • Quantization‑Aware Training (QAT) in PyTorch, exporting to Core ML using coremltools with weight & activation int8.
  • Weight sharing: clustered similar convolution kernels, then applied Huffman coding on the resulting codebook (supported by Core ML’s model compression flags).

Result

  • Final model: 18 MB (≈85 % reduction).
  • Frame‑rate: 60 fps on the NE, with < 1 % drop in mean Intersection‑over‑Union (mIoU).

2.3 Qualcomm Snapdragon Enables On‑Device Translation

Company: Duolingo
Product: Duolingo Translate – offline translation for 50+ languages on Android phones.

Challenges

  • Snapdragon 8 Gen 2 can run TensorFlow Lite models, but only up to int8.
  • The base transformer model (300 M parameters) required ~1.2 GB in fp32.

Solution

  1. PTQ with TensorFlow Lite’s full integer quantization (weights + activations int8).
  2. Dynamic range quantization for less sensitive embedding layers (kept at fp16).
  3. Structured pruning (removing entire attention heads) followed by a short fine‑tune to keep BLEU scores stable.

Result

  • Model size: 210 MB, fitting comfortably on a 2 GB app bundle.
  • Latency: ≈30 ms per sentence, enabling real‑time translation offline.

These three examples illustrate how different hardware ecosystems (NVIDIA, Apple, Qualcomm) share a common recipe: quantize → prune → fine‑tune, with variations tailored to each vendor’s runtime capabilities.


3. Toolchain Comparison: Which Stack Fits Your Edge Target?

Below is a concise comparison of the most widely used quantization/compression toolchains for edge deployment. The table focuses on supported precision, hardware acceleration, and typical workflow.

Tool / Service Supported Quantization Types Pruning / Weight Sharing Hardware Optimizations Export Formats Typical Use‑Case
TensorFlow Lite PTQ (int8, int16), Dynamic Range (float16) Structured pruning via TensorFlow Model Optimization Toolkit Android NNAPI, Edge TPU, GPU delegate .tflite Mobile apps, embedded Linux
PyTorch Mobile PTQ, QAT (int8, int4 via custom kernels) Pruning via torch.nn.utils.prune Android NNAPI, iOS Metal, Qualcomm Hexagon .pt (scripted) Cross‑platform mobile, research prototyping
ONNX Runtime PTQ (int8/int4), QAT via external quantizers Supports weight sharing & pruning via ONNX operators Windows ML, Linux CPU/GPU, NVIDIA TensorRT .onnx Interoperable deployment across vendors
NVIDIA TensorRT PTQ, QAT, GPTQ (int4/int8) Integrated pruning + weight sharing (via TensorRT‑LLM) Jetson, Tesla GPUs, RTX series .plan High‑throughput inference on NVIDIA GPUs
Apple Core ML QAT (int8), float16, weight sharing via model compression flags No built‑in pruning (use Core ML Tools conversion) Apple Neural Engine, CPU, GPU .mlmodel iOS/macOS on‑device AI
OpenVINO PTQ (int8/int4), Mixed precision (FP16/INT8) Pruning via NNCF (Neural Network Compression Framework) Intel CPUs, Myriad VPU, Xe GPUs .xml/.bin Edge devices with Intel accelerators

How to Choose

  • GPU‑heavy edge (Jetson, RTX) → TensorRT for the most aggressive GPTQ and integrated pruning.
  • iOS/macOS → Core ML; leverage Apple’s NE with QAT‑int8 models.
  • Android/heterogeneous → TensorFlow Lite or ONNX Runtime; both have strong NNAPI support.
  • Cross‑vendor interoperability → ONNX Runtime, especially when you need to switch between Intel and NVIDIA hardware.

4. Step‑by‑Step Workflow: From a Cloud‑Trained Model to Edge Deployment

Below is a practical pipeline you can follow regardless of the target hardware. The steps are ordered from most generic to hardware‑specific.

4.1 Baseline Training (Optional)

If you have the luxury of retraining, start with Quantization‑Aware Training (QAT):

import torch
import torch.quantization as tq

model = MyLargeTransformer()
model.qconfig = tq.get_default_qat_qconfig('fbgemm')
tq.prepare_qat(model, inplace=True)

# Train for a few epochs
for epoch in range(num_epochs):
    train_one_epoch(model, train_loader)
    validate(model, val_loader)

tq.convert(model.eval(), inplace=True)   # Final quantized model

Why QAT? Simulating low‑precision during training helps the optimizer learn weights that are robust to rounding errors, often recovering the 1‑2 % accuracy loss seen with PTQ.

4.2 Post‑Training Quantization (If Retraining Isn’t Feasible)

Use the calibration dataset (a few thousand representative inputs) to compute scaling factors:

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model('saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = lambda: representative_data_gen()
tflite_model = converter.convert()

Tip: For extremely large models, consider layer‑wise calibration to avoid OOM errors.

4.3 Advanced Row‑Wise Quantization (GPTQ)

For LLMs, NVIDIA’s GPTQ (or open‑source equivalents) offers per‑row bit allocation:

python -m gptq.quantize \
    --model_path llama-7b-fp16.pt \
    --bits 4 \
    --group-size 128 \
    --output llama-7b-gptq-int4.pt

This step reduces the memory footprint dramatically while keeping perplexity within 1 % of the fp32 baseline 4.

4.4 Pruning & Weight Sharing

Apply structured pruning (e.g., remove entire attention heads or convolution channels) to reduce compute:

import torch.nn.utils.prune as prune
for name, module in model.named_modules():
    if isinstance(module, torch.nn.Linear):
        prune.ln_structured(module, name='weight', amount=0.2, n=2)  # 20% pruning

After pruning, fine‑tune for a few epochs to regain performance. If your runtime supports weight sharing, cluster similar weights and generate a codebook (e.g., using sklearn.cluster.KMeans), then enable Huffman coding during export 3.

4.5 Export to Target Runtime

Target Export Command
TensorRT trtexec --onnx=model.onnx --int8 --saveEngine=model.plan
Core ML coremltools.convert(model, inputs=[...], minimum_deployment_target='ios15')
TensorFlow Lite See Step 4.2 (converter)
ONNX Runtime python -m onnxruntime.quantization.quantize_static --model_path model.onnx --calibrate_path data/

4.6 Validate Accuracy & Performance

  1. Quantization error: Compute the difference in logits between fp32 and quantized models on a validation set.
  2. Latency: Benchmark on the actual device using vendor‑provided tools (e.g., nvprof, Xcode Instruments, adb shell perf).
  3. Power: Use hardware monitors (e.g., Jetson’s tegrastats) to verify the energy budget.

If the accuracy drop exceeds your threshold (commonly 1‑2 % for classification or < 0.5 BLEU for translation), revisit pruning ratios or switch to a higher‑precision mode (e.g., int8 → int4 only for less sensitive layers).


5. Frequently Asked Questions (FAQ)

Question Short Answer
Can I quantize a model to < 8 bits without QAT? Yes, using PTQ‑based GPTQ or per‑channel int4 methods, but expect a larger accuracy gap; fine‑tuning helps.
Do I need both pruning and quantization? Not always; quantization alone can give 4‑8× size reduction. Pruning adds extra compute savings and is valuable when memory is extremely constrained.
Is Huffman coding supported on all runtimes? Only a few (e.g., Core ML, TensorRT) expose a “compressed model” flag that internally applies Huffman coding. Otherwise, you must decompress offline before loading.
What is the difference between weight‑only and weight & activation quantization? Weight‑only keeps activations in higher precision, preserving output fidelity while still

Related Articles


This article was created using generative AI.

Popular articles