6 Smart Model Compression Techniques to Cut AI Costs and Lag

Have you ever built a machine learning model, only to discover it was too large to deploy? It could be too slow, or it wouldn't run on the right system. Whatever the case, you weren't alone; many developers face this same machine learning optimization challenge. The good news is that there are ways to make these models more efficient before deployment. Model compression helps reduce the size and speed of AI models without sacrificing accuracy. This article will show you how to compress your model to run faster, cost less, and work efficiently on any device, including edge and mobile. If you want to simplify the process of getting your newly compressed model running well on your system of choice, AI inference APIs can help. These tools automate and accelerate the integration of AI models into applications so they can make accurate predictions quickly, no matter the deployment environment.

What is Model Compression?

Model compression is techniques that reduce machine learning models' size and computational requirements without significantly compromising accuracy. Its essentials include:

• Improving inference speed
• Lowering energy consumption
• Enabling deployment on edge devices

In recent years, the push for better neural network models has resulted in larger model sizes. We’ve seen model parameter counts grow from the millions to the billions, with associated growth in deployment costs.

While these models are “better” in that they beat others’ benchmarks on various natural language processing (NLP) tasks, the question must be: What if a better model means a smaller model that performs just as well?

Model Compression Defined

The perspective of creating just-as-good smaller models can be referred to under the blanket term model compression. Model compression aims to achieve a simplified model from the original without significantly diminishing accuracy. A simplified model is reduced in size and/or latency from the original.

Specifically, a size reduction means that the compressed model has fewer and/or smaller parameters and, thus, uses less RAM when run. This is desirable because it frees up memory for other application parts.

Benefits of Reduced Latency

A latency reduction is a decrease in the time it takes for the model to make a prediction or inference based on an input to the trained model. This typically translates to lower energy consumption at runtime. Model size and latency often go hand in hand because larger models require more memory accesses to run.

Both types of reduction are desirable for deploying models in space, where computing resources face strict size and power constraints.

Recognizing When You're Ready for Model Compression

Before blindly implementing model compression techniques, checking whether they will accomplish your goals is a good idea. Some key indicators that compression techniques may be of benefit include:

• Models are running in real time
• Initial models are large
• Server costs for model inference are high

If any of the above factors are bottlenecks to utilizing a neural network for a task, model compression may resolve them.

Where in the ML Lifecycle Does Model Compression Occur?

First things first. To start, we must understand where the model compression step is to occur within the lifecycle of Machine Learning. Once this is identified, we can implement various model compression techniques, including:

• ONNX conversion
• Quantization
• Distillation

Figure 1: The Machine Learning lifecycle

The ML Lifecycle

The classic ML lifecycle focuses on the cycle of data acquisition, model training, and model deployment. Between steps 5 and 6, as shown in Figure 1, is the opportunity to add a step for implementing model compression techniques. Let’s explore the costs and benefits of this extra step.

What Are the Costs Associated with Model Compression?

Even when it’s beneficial, compression is not free. Costs of implementing it include:

Increased Deployment Complexity

After implementing various model compression techniques, there is more to keep track of, namely the original trained model and the compressed models. We must choose the model to deploy and spend time making this choice.

Decreased Accuracy

Some model compression techniques result in a loss of accuracy (nevertheless, this is measured). This cost has an obvious counterpart: the benefits of the model compression technique may outweigh the accuracy loss.

Compute Cost

While model compression reduces the compute resources required for inference, the compression itself may be computationally expensive. Notably, distillation introduces an additional iterative training step.

Your Time

Adding a step to the lifecycle requires an investment of your time.

Related Reading

• LLM Inference Optimization
• Real-Time Machine learning
• How to Improve Machine Learning Models

6 Popular Model Compression Techniques

1. Pruning: Cutting the Fat for a Leaner Network

Pruning involves removing connections between neurons or entire neurons, channels, or filters from a trained network, which is done by zeroing out values in its weights matrix or removing groups of weights entirely; for example, to prune a single connection from a network, one weight is set to zero in a weights matrix, and to prune a neuron, all values of a column in a matrix are set to zero.

The Rationale for Pruning

The motivation behind pruning is that networks tend to be over-parametrized, with multiple features encoding nearly the same information. Based on the removed network component, pruning can be divided into two types: unstructured pruning involves removing individual weights or neurons, and structured pruning involves removing individual weights or neurons.

In contrast, structured pruning involves removing entire channels or filters. We’ll examine these two types individually, as their implementations and outcomes differ.

Unstructured Pruning

By replacing connections or neurons with zeros in a weights matrix, unstructured pruning increases the network's sparsity, i.e., its proportion of zero to non-zero weights. Various hardware and software, like TensorFlow Lite and Caffe, are specialized to load and efficiently perform operations on sparse matrices.

Latency Reduction in Pruned Models

TensorFlow Lite and Caffe can significantly reduce latency in pruned models compared to their original dense representations. For example, Caffe can apply a mask to pruned parameters that causes them to be skipped over during network operation, reducing the amount of FLOPs and, thus, the power and time required to make an inference.

Depending on the degree of sparsity and the method of storage used, pruned networks can also take up much less memory than their dense counterparts.

Memory Efficiency of Sparse Networks

But what is the criterion for deciding which weights should be removed? One standard method, magnitude-based pruning, compares the weights’ magnitudes to a threshold value. Han et al.'s highly cited 2015 paper prompted widespread adoption of this approach. In their implementation, pruning is applied layer-by-layer.

Magnitude-Based Pruning Criterion

First, a predetermined “quality parameter” is multiplied by the standard deviation of a layer’s weights to calculate the threshold value, and weights with magnitudes below the threshold are zeroed. After all layers are pruned, the model is retrained so that the remaining weights can adjust to compensate for those removed, and the process is repeated for several iterations.

Iterative Pruning and Retraining

The researchers used this method to prune four different model architectures, two pre-trained on the MNIST dataset and two on the ImageNet dataset. We can see the effects of their unstructured pruning approach visualized in the following image, which shows the sparsity pattern of the first fully-connected layer for one of the MNIST networks, where the blue dots represent non-zero parameters:

MNIST Dataset and Input Processing

The MNIST dataset consists of 28×28 pixel images of handwritten digits from 0 to 9, like the one above, and models are trained to classify these digits. To be input into a neural network, the image is flattened by concatenating the rows of pixels end-to-end from top to bottom, resulting in a vector of 784 values.

Pruning Effects on an MNIST Network

The first layer of the network is fully connected with 300 neurons. As we can see, the digits tend to be oriented in the center of the image; thus, pixels around the edges are less consequential to the classification task, and connections to them get pruned more heavily. For the entire network, pruning decreased the number of non-zero weights and FLOPs required by a factor of 12 with no drop in predictive performance.

For the AlexNet and VGG-16 models trained on ImageNet, the parameters were reduced from 61 million to 6.7 million and from 138 million to 10.3 million, respectively, with no decrease in classification accuracy.Han et al. showed that unstructured pruning can yield some truly impressive degrees of compression when implemented correctly. They note that “the storage requirements of AlexNet and VGGNet are small enough that all weights can be stored on-chip, instead of off-chip DRAM, which takes orders of magnitude more energy to access.”

Hardware Limitations of Unstructured Pruning

Nevertheless, they also acknowledge the “limitation of general purpose hardware on sparse computation” and, thus, use specialized software to work around this problem. Indeed, unstructured pruning’s reliance on specially designed software or hardware to handle network sparsity to speed up computations is a significant limitation, one that is not faced by taking a structured approach.

Structured Pruning

Unlike unstructured pruning, structured pruning does not result in weight matrices with problematic sparse connectivity patterns because it involves removing entire blocks of weights within given weight matrices. This means the pruned model can be run using the same hardware and software as the original.

Structured Pruning and Data-Driven Methods

While we are now looking at groups of weights to remove at the channel or filter level, magnitude-based pruning can still be applied by ranking them according to their L1 norms, for example. More intelligent, “data-driven” approaches have also been proposed to achieve better results.

Pruning Agents for Performance-Size Tradeoff

Huang et al., for example, were the first to integrate into the pruning process to control the tradeoff between network performance and size in their 2018 paper. Their algorithm outputs a set of filter “pruning agents”—each a neural network corresponding to a convolutional layer of the network—and an alternative, pruned version of the original model, which is initialized to be the same as the original.

Training Pruning Agents with a Drop Bound

The pruning agents maximize an objective parameterized by a “drop bound” value, which is defined as the maximum allowed drop in performance between the original and the pruned model, forcing the agents to keep performance above a specified level. For each convolutional layer, a pruning agent is trained by evaluating the effects of pruning combinations of filters within that layer.

To do so, it removes specific filters from the alternative model and compares this model’s performance on an evaluation set to that of the original, learning which modifications will increase the network’s efficiency while still adhering to accuracy constraints.

Once the agent for one layer is trained and filters for that layer have been optimally removed, the entire pruned model is retrained to adjust for the changes, and the process repeats for the next convolutional layer.

Pruning Degree Across VGG-16 Layers

The following plot shows the degree of pruning achieved with this approach, with drop bound b = 2 on the layers of a VGG-16 model trained on the CIFAR 10 dataset. The greater degree of pruning of higher layers indicates they tend to contain more unnecessary filters than the initial ones.

Performance Comparison with Magnitude-Based Pruning

We can also see the quantitative compression results of pruning this model using different drop bound values. The accuracy values in parentheses correspond to models pruned to the same ratio. Still, with a magnitude-based approach, the “data-driven” implementation has superior performance.Similarly, impressive results have been achieved through various structured pruning methods, establishing it as a popular model compression technique. Nevertheless, pruning in general still has a few potential drawbacks.

For example, it’s generally unclear how well given methods generalize across different architectures, and the pruning process tends to involve a lot of fine-tuning that can act as a barrier to implementation and generalization.

Efficient Architectures and Further Reading

In many cases, using a more efficient architecture may be more effective than pruning a suboptimal one. If you’re interested in taking a closer look at the diverse range of approaches to pruning models, I suggest checking out this blog or the paper “What is the State of Neural Network Pruning?” written in 2019.

2. Quantization: Downsize Model Weights

While pruning compresses models by reducing the number of weights, quantization consists of decreasing the size of the present weights. Quantization is generally the process of mapping values from a large set to values in a smaller set, meaning that the output consists of a smaller range of possible values than the input, ideally without losing too much information.

Image Compression as an Analogy

We can think about this in the context of image compression, as with the images of Einstein above. In the leftmost representation, each pixel value is represented by 8 bits and thus can take on 256 shades of gray. The number of possible shades is halved in each successive image.

The Goal of Model Quantization

We aren’t able to detect much difference between the first four images. This represents the goal of model quantization: to reduce the precision of network components to the extent that it is lighter but without a “noticeable” difference in efficacy. Intuitively, neural networks can deal with some information loss.

They are trained to cope well with high noise levels in their inputs, and lower precision calculations can essentially be seen as another noise source.

Adaptive Quantization Based on Data Distribution

How might a mapping between different precisions (i.e., a quantization scheme) work? In a neural network, a particular layer's weights or activation outputs tend to be normally distributed within a specific range, so ideally, a quantization schema takes advantage of this fact and adapts to fit each layer’s distribution.

For example, model weights are typically stored as 32-bit floating-point numbers, and a common approach is to reduce these to 8-bit fixed points (though some techniques have even gone as far as to represent them with single bits!), for which there are 2⁸ = 256 possible values.

Uniform Quantization Using Min-Max Range

In the simplest case, we can take the min and max weights of a layer, divide the range between the two into 255 evenly-spaced intervals, and bin the weights according to the interval edge they are closest to:

Quantizing the weights in this way would reduce their memory footprint by 4x. Many other, more complex techniques achieve lower information loss, such as an implementation by Han et al. that involves k-means clustering. 

Combined Pruning and Quantization for High Compression

Specifically, their compression pipeline combines weight pruning, quantization, and Huffman coding, and we can see in the following graph how pruning and quantization of AlexNet trained on ImageNet work together to achieve a model that is 3.7% of its original size with no loss in accuracy:

In addition to weights, neural networks also consist of operators, such as:

• Matrix multiplication
• Convolution
• Activation functions
• Pooling that must also be quantized to deal with lower precision weights

While the general idea is the same, quantizing operators is trickier than quantizing pre-trained weights because they have to deal with potential bit overflows and unseen inputs, making determining a schema difficult.

Software Support for Operator Quantization

Luckily, many deep learning softwares, such as TensorFlow, PyTorch, and MXNet, have functionality for quantizing commonly-used operators (if you’re interested, this blog provides a high-level explanation of TensorFlow’s process), which can yield networks with significantly diminished latency given the increased efficiency of lower precision operations.

The Practical Challenges of Quantization

In practice, quantization can be challenging to implement because it requires having a decent understanding of hardware and bit-wise computations. It turns out that deciding upon the degree of precision to compress a model to is not as simple as picking a random number of bits; you have to make sure the hardware you’re using is even capable of handling numbers of that size, let alone doing so efficiently.

Hardware Dependence vs. Generalizable Compression

In reviewing the literature, I found that the savings offered by many state-of-the-art quantization implementations are tied to the features of the hardware being used. Nevertheless, other means of compression that generalize well across many kinds of hardware, such as low-rank factorization.

3. Low-Rank Approximation: Taking an Efficient Shortcut

Deep neural networks tend to be over-parametrized, with many similarities or redundancies occurring between different layers and/or channels. For example, the image above shows filters of a convolutional layer learned for an image segmentation task, many of which look nearly identical.

Low-Rank Approximation for Efficient Layers

And since low-rank matrices encode redundant information, we can use them to approximate these redundant layers of a network, which is precisely what low-rank approximation does. To understand the intuition behind achieving this, let’s look at a relatively simple approach from Jaderberg et al. that takes advantage of cross-filter redundancy.

The following represents a convolution operation with N 2D filters, W, on a single-channel feature map, Z, to output an N-channel feature map.

Approximating Filters with Lower Rank Combinations

The goal is to reduce the computational complexity of these operations by approximating the N whole rank filters with a linear combination of M lower rank filters, where M<N, based on the assumption that there is redundancy between the N filters. Specifically, Jadenberg et al. impose that the new filters rank 1.

If a matrix has rank 1, it means that if we select one of its columns, all the other columns will be some multiple of that column. Thus, rank one matrices can be factorized or decomposed to be products of a column vector and a row vector, a property called separability, for example:

• [[1, 4, 5], [[1], [[1, 4, 5]]
• [2, 8, 10], = [2], *
• [3, 12, 15]] [3]]

By imposing separability of the M basis filters and adding a layer to approximate the N output channels of the original convolution operation from the compressed basis, we arrive at a transformed operation that looks like this:

Computational Efficiency of Separable Filters

Low-rank approximation aims to approximate a layer's numerous redundant filters using a linear combination of fewer filters, such as those in image b. Compressing layers reduces the network’s memory footprint and the computational complexity of convolutional operations and can yield significant speedups.

Specifically, the filters in b are of lower rank than those in a. The rank of a matrix is defined as the dimension of the vector space spanned by its columns, which is equal to its number of linearly independent columns. To illustrate, say we have the following matrix:

• [1, 0, -1]
• [-2, -3, 1]
• [3, 3, 0]

Understanding Matrix Rank Through Linear Dependence

If we look at the first two columns, we see that they are linearly independent because neither can be computed from the other using linear operations, so the matrix has to have a rank of at least 2. But if we look at all three columns, we see that if we subtract the first from the second, we get the third. Thus, all three are not linearly independent, and the matrix only has a rank of 2.

Redundancy Between Filters and Channels

Intuitively, if an element of a matrix can be computed using others present, there is some information redundancy in the matrix. While individual filters and channels in a network tend to be of full rank, indicating no redundancy within themselves, there is typically significant redundancy between different filters or channels.

Low-Rank Approximation Exploits Redundancy

While it may look more complex, it is a more computationally efficient operation if M is made to be sufficiently small. The separable bases for these modified layers are optimized asynchronously by minimizing the reconstruction error between the original layer's output and the approximation layer's output.

Jaderberg et al. use a similar approach to exploit redundancy between channels. With this approach, they achieve a 4.5× speedup of a shallow network trained for a text classification task with only a 1% drop in classification accuracy.

Introduction to SVD in Low-Rank Approximation

Suppose you take a look at the literature on low-rank approximation. In that case, you’ll likely come across the terms SVD (“Singular Value Decomposition”), Tucker decomposition, and CP (“Canonical Polyadic”) decomposition, so it bears briefly explaining the distinction between these. SVD stands for “Singular Value Decomposition” and is a matrix factorization.

Generalizing SVD to Tensors

In the context of approximating a matrix M with another matrix M’ of a lower rank r as we’ve discussed, the optimal M’ is equal to the SVD of M constrained by r. Tucker and CP decomposition are ways of generalizing SVD to tensors. As you likely know, a tensor is a multi-dimensional array.
Tucker decomposition decomposes a tensor into a set of matrices and one small core tensor. CP is a special case of Tucker decomposition that decomposes a tensor into a sum of rank-1 tensors, though the literature often refers to them as two separate approaches.

We can see how they both work in 3D space below:Both have benefits and drawbacks, which you can read more about here, and are popular approaches to low-rank approximation of deep neural networks. Kim et al., for example, achieve substantial compression using Tucker decomposition. They take a data-driven approach to determining the ranks the compressed layers should have and then perform Tucker decomposition according to these ranks.

They apply this compression to various models for image classification tasks and run them on a Titan X and a Samsung Galaxy S6 smartphone, which notably has a GPU with 35× lower computing ability and 13× smaller memory bandwidth than Titan X.

Performance of Low-Rank Approximation on Mobile Deployment

The following table shows the compressed models’ drops in performance and reductions in size and FLOPs from the original, as well as their time and energy consumption to process a single image. With these results, Kim et al. demonstrate that low-rank approximation is an effective means of compression to achieve significant size and latency reductions in deep neural networks for their potential deployment on mobile devices.

And as I mentioned before, a considerable upside to low-rank approximation as a compression technique is that it does not require specialized hardware since it simplifies model structure by reducing parameter count.

4. Knowledge Distillation: Transferring Knowledge from One Model to Another

The need to compress neural networks arises because training and inference are fundamentally different tasks. During training, for example, a model does not have to operate in real time and does not necessarily face restrictions on computational resources, as its primary goal is simply to extract as much structure from the given data as possible. Still, latency and resource consumption become of concern if it is to be deployed for inference.

The Motivation for Model Distillation

As a result, trained models are often larger and slower than ideal for inference; hence, we have to develop ways to compress them. To address this problem, Cornell researchers proposed in 2006 the idea of transferring the knowledge from a large trained model (or ensemble of models) to a smaller model for deployment by teaching it to mimic the larger model’s output, a process that Hinton et al. generalized in 2014 and gave the name distillation.

In short, knowledge distillation is motivated by the idea that training and inference are different tasks, and a different model should be used for each.In Hinton et al.’s implementation, the small “student” network learns to mimic the large “teacher” network by minimizing a loss function in which the target is based on the distribution of class probabilities outputted by the teacher’s softmax function.

To understand this, let’s review how the softmax function works:

It takes in a logit for a particular class, z_i, and converts it to a probability by dividing it by the sum of the logits of all j classes. Here, T is the temperature, for which higher values correspond to “softer” outputs. T is usually set to 1, which results in “hard” outputs in which the correct class label is assigned a probability close to 1, indicating near certainty in this prediction. In contrast, the others are assigned probabilities close to 0.

Hinton suggests that the distribution of the incorrect class labels holds valuable information about the data that can be learned from, as they describe in the context of a handwritten digit classification task:“One version of a 2 may be given a probability of 10^-6 of being a 3 and 10^−9 of being a 7 whereas for another version it may be the other way around. This is valuable information that defines a rich similarity structure over the data (i. e. it says which 2’s look like 3’s and which look like 7’s)…”

Softening Teacher Outputs Through Distillation

To capitalize upon this information, Hinton et al. raise the temperature T in the teacher network’s softmax function to soften the distribution of probabilities over the class labels in a process they call distillation, as demonstrated in the following:The student network is then trained to minimize the sum of two different cross-entropy functions: one involving the original challenging targets obtained using a softmax with T=1 and one involving the softened targets.

Distillation Improves Smaller Network Performance

Hinton et al. demonstrate the effectiveness of this technique by training a teacher network with two hidden layers of 12,000 units and a student network with two hidden layers of only 800 units. On a test set, the teacher achieves 67 test errors, and the student achieves 74, compared to the 146 test errors made by a network of the same size as the student but that was trained without distillation.

Leveraging Intermediate Teacher Layers in FitNets

After Hinton et al.'s initial breakthrough of training the student to imitate a teacher's softmax outputs, researchers Romero et al. found that the student can also use information from the teacher’s intermediate hidden layers to improve final performance. Specifically, they propose what they call “FitNets,” which are student networks that are thinner but deeper than their teachers.

Their increased depth allows them to generalize well, while their small widths make them compact. Romero et al. introduce one “guided layer” in the middle of the student network that is tasked with learning from one “hint” layer in the middle of the teacher network.

FitNets Achieve Speedup and Accuracy Gains

The table below demonstrates the speedup, reduced parameter count, and generally increased accuracy that the FitNets of varying sizes achieve compared to their teacher architecture on the CIFAR-10 image classification dataset. Further, their performance was on par with state-of-the-art methods, of which the highest accuracy at the time was 91.78%.

Proliferation of Knowledge Distillation Methods

Impressive results have spurred research on various methods for transferring knowledge from teacher to student networks. Indeed, the field of research on knowledge distillation has become so broad and specialized in some areas that it is difficult to evaluate the overall efficacy of general approaches against one another.

User Decisions in Implementing Distillation

A drawback of knowledge distillation as a compression technique is that the user must make many decisions up front to implement it; for example, unlike the other compression techniques we’ve discussed, the compressed network doesn’t need a structure similar to the original.

Flexibility and Further Reading on Distillation

Nevertheless, this also means that knowledge distillation is flexible and can be adapted to various tasks. If you want to learn more about different knowledge distillation techniques, this 2020 paper comprehensively reviews state-of-the-art approaches for vision-based tasks.

5. Neural Architecture Search (NAS): Using Algorithms to Design Efficient Models

The reality is that multiple compression techniques we’ve discussed could be applied in a given scenario. When you consider their combinations, the number of possible architectures to explore expands dramatically. The hard part is, of course, choosing the optimal one. This is where Neural Architecture Search (NAS) comes in.

Automating Model Design

In the most general sense, it is a search over a set of decisions that define the different components of a neural network—it is a systematic, automated way of learning optimal model architectures. The idea is to remove human bias from the process to arrive at novel architectures that perform better than human-designed ones.Obviously, for any given task, there are infinite possible model architectures to explore and many different approaches to searching through them.

A 2019 survey categorizes different NAS techniques according to the following three dimensions:

1. Search Space

The possible architectures that can be discovered in the search. We’re aware that networks can consist of a whole host of operations, such as:

• Convolutions
• Pooling
• Concatenation

Defining the Neural Architecture Search Space

Activation functions and defining a search space impose constraints on how and which components can be combined to generate the networks searched through. For example, defining a search space might include restricting the number of possible convolutional layers the network can have or requiring it to repeat a general pattern of operations several times.

Imposing such restrictions inevitably introduces some human bias into the search process. Even with these restrictions, the search space will still be huge, as is the point of NAS—to discover and evaluate architectures outside what we might typically construct.

2. Search Strategy

The strategy followed to guide the exploration of the search space, i.e., what determines the following architectures to explore in the search. While this could be random, it’s unlikely given the vast scope of a given search space that an optimal architecture will be stumbled upon by random chance, so the search strategy typically takes into account the performance of previously explored architectures.

Examples of Neural Architecture Search Strategies

For example, “evolutionary” algorithms are based on maintaining an evolving population of candidate architectures throughout the training process. There are also reinforcement-learning-based strategies where an agent is trained to estimate the performance of architectures on unseen data.

3. Performance Estimation Strategy

How is the performance of candidate models on unseen data estimated? Performance feedback is used to optimize the search strategy. This can be as simple as training each network from scratch and measuring their accuracy on a validation set, but this can be very costly; thus, strategies that involve parameter sharing between many models or that use low-fidelity approximations of data or the networks themselves, for example, have been developed.

As you can imagine, NAS is a vast field of research, so here, we will focus on its use for compressing pre-trained models. In 2018, He et al. introduced AutoML for Model Compression (AMC), a technique that uses a reinforcement learning search strategy to compress pre-trained networks layer-by-layer.

Specifically, AMC’s search space is parametrized by a user-defined, hardware-based constraint such as:

• Maximum latency
• Model size
• Number of FLOPS

Reinforcement Learning for Hardware-Aware Compression

The agent proceeds through the network one layer at a time, outputting a compression ratio based on the layer's composition and adhering to the hardware constraint. After all layers are pruned according to their ratios via a structured or unstructured approach, the validation accuracy of the compressed model is computed without fine-tuning and used as the agent’s reward.

Forgoing fine-tuning after compressing the model is a tactic He et al. use in their performance estimation strategy, which allows for faster policy exploration.

AMC Outperforms Rule-Based Pruning Strategies

AMC’s learning-based pruning was tested against handcrafted, rule-based approaches on various models and exhibited superior performance. For example, the following table shows the results of AMC’s compression with a 50% FLOP or 50% latency constraint on MobileNet as compared to a rule-based strategy that prunes the network to 75% of its original parameters.

As we can see, the networks pruned with AMC achieve nearly the same accuracy as the original MobileNet with reduced latency and memory size, as opposed to the rule-based pruning technique.

Bayesian Optimization for Architecture Search

More recently, in 2019, Cao et al. used Bayesian Optimization to search for an optimal compressed architecture. In their implementation, they continuously add to a set of models based on an original network but have been compressed by randomly removing or shrinking layers and adding skip connections.

Iterative Model Generation and Evaluation

These architectures are evaluated based on the reduced parameter count and accuracy they achieve compared to the original. Over several epochs, the models are sampled and used to learn to predict to what extent new, randomly generated compressed models will outperform the current best model of the set in terms of compression and accuracy.

Efficient Architecture Exploration Through Prediction

This way, the randomly generated architectures don’t have to go through the time-intensive process of being evaluated on data, and more architectures can be explored instead. Based on these predictions, the most promising of the newly generated models are evaluated on data and added to the growing set of models, and the process continues. The set's best model in terms of compression and accuracy is selected.

The following table demonstrates Cao et al.’s approach's effectiveness in compressing three architectures for image classification on the CIFAR-100 dataset compared to a random NAS approach and to Ashok et al’s state-of-the-art “N2N” technique.

The Complexity and Barrier to Entry of NAS

We’ve seen two demonstrations of NAS’s effectiveness at achieving more compact models. The possibilities are endless with NAS, but successful implementation relies on picking a search space, search strategy, and performance estimation strategy suited to the problem, which can require a decent amount of domain knowledge. As a compression technique, it poses a barrier to entry.

Making NAS More Accessible with AutoML

Luckily, software does exist that abstracts away some of the complexity to make NAS more accessible. For example, Google’s Cloud AutoML makes the process as simple as inputting a training dataset and letting its algorithm search for an optimal set of building blocks for the network, a baseline architecture that the user can fine-tune to optimize further.

This is done within its GUI to make the process accessible even to inexperienced programmers. Auto-Keras is an open-source NAS package that takes a different approach. The user specifies the high-level architecture of the model, and the algorithm searches for the configuration details.

6. Low Matrix Factorization: Improving Speed and Performance

Model accuracy and performance highly depend on proper factorization and rank selection. The main challenge in the low-rank factorization process is its more complex implementation, which is computationally intensive. Overall, factorization of the dense layer matrices results in a smaller model and faster performance than full-rank matrix representation.

The Importance of Model Compression for Edge AI

Due to Edge AI, model compression strategies have become incredibly important. These methods complement one another and can be used across stages of the entire AI pipeline. Popular frameworks like TensorFlow and Pytorch now include techniques like Pruning and Quantization. Eventually, the number of methods used in this area will increase.

Related Reading

• LLM Latency
• Inference Time
• AI Inference vs. Training

Challenges of Model Compression in AI

Accuracy Tradeoffs: The Balancing Act of Model Compression

Model compression offers significant advantages, but it also comes with important trade-offs. For starters, there may be accuracy tradeoffs when compressing the original model. While compressing the model, it's crucial to maintain accuracy. This balance might be challenging to achieve in all scenarios.

In some cases, keeping the original accuracy may be impossible, especially when compressing the model drastically.

Compatibility: Ensuring Smooth Migration Between Original and Compressed Models

Another challenge involves the compatibility of the original AI model with the compressed one. Sometimes, compatibility between the original AI model and the compressed one is challenging, which can pose a difficulty when migrating the models. In some scenarios, the architecture of the compressed model may differ significantly from that of the original model, which can complicate the transfer process.

Increased Complexity: The Hidden Costs of Model Compression Techniques

Specific model compression techniques can increase algorithmic and implementation complexity. While these methods can effectively reduce model size, they can also lead to more complicated algorithms that can be difficult to interpret. Further, the increased complexity can result in longer training and require specialized knowledge to implement and fine-tune.

Limited Influence: Users May Have Little Control Over the Compression Process

Lastly, users of compressed models exert little influence on the compression process and results. This limited control can lead to potential misalignment with their specific needs. In some cases, compressed models may not perform as expected or introduce unwanted biases that can adversely affect predictions.

Model Compression Implementation

Implementing model compression in AI necessitates a thorough understanding of the models and a meticulous selection of the appropriate compression technique. Post this, a careful evaluation concerning the model’s performance before and after compression is crucial to ensure that the trade-off between computational efficiency and performance is balanced.

To guarantee overall efficacy, the compressed models should be periodically monitored for possible biases and unwarranted deviations.

A Holistic Approach to Model Compression

Thus, while model compression creates a pathway towards making AI models more efficient and accessible, a well-rounded approach to implementing it ranges from profiling the model's computation and communication footprints to selecting the appropriate method of compression that aligns with the model's requirements, and continuous monitoring and evaluation of the system's performance post-compression.

Start Building with $10 in Free API Credits Today!

Inference delivers OpenAI-compatible serverless inference APIs for top open-source LLM models, offering developers the highest performance at the lowest cost in the market. Beyond standard inference, Inference provides specialized batch processing for large-scale async AI workloads and document extraction capabilities designed explicitly for RAG applications.

Start building with $10 in free API credits and experience state-of-the-art language models that balance cost-efficiency with high performance.

Related Reading

• Machine Learning Optimization
• Artificial Intelligence Optimization
• Latency vs. Response Time