2.5 Determining Quantizer parameters
The quantizer parameters can be determined using several criteria. For example, Ten-
sorRT [11] minimizes the KL divergence between the original and quantized distribu-
tions to determine the step size. In this work, we adopt simpler methods. For weights,
we use the actual minimum and maximum values to determine the quantizer parame-
ters. For activations, we use the moving average of the minimum and maximum values
across batches to determine the quantizer parameters. For post training quantization
approaches, one can improve the accuracy of quantized models by careful selection of
quantizer parameters.
2.6 Granularity of quantization
We can specify a single quantizer (defined by the scale and zero-point) for an entire
tensor, referred to as per-layer quantization. Improved accuracy can be obtained by
adapting the quantizer parameters to each kernel within the tensor [17]. For example,
the weight tensor is 4 dimensional and is a collection of 3 dimensional convolutional
kernels, each responsible for producing one output feature map. per-channel quantiza-
tion has a different scale and offset for each convolutional kernel. We do not consider
per-channel quantization for activations as this would complicate the inner product
computations at the core of conv and matmul operations. Both per-layer and per-
channel quantization allow for efficient dot product and convolution implementation
as the quantizer parameters are fixed per kernel in both cases.
3 Quantized Inference: Performance and Accuracy
Quantizing a model can provide multiple benefits as discussed in section 1. We discuss
multiple approaches for model quantization and show the performance impact for each
of these approaches.
3.1 Post Training Quantization
In many cases, it is desirable to reduce the model size by compressing weights and/or
quantize both weights and activations for faster inference, without requiring to re-train
the model. Post Training quantization techniques are simpler to use and allow for
quantization with limited data. In this section, we study different quantization schemes
for weight only quantization and for quantization of both weights and activations. We
show that per-channel quantization with asymmetric ranges produces accuracies close
to floating point across a wide range of networks.
3.1.1 Weight only quantization
A simple approach is to only reduce the precision of the weights of the network to 8-
bits from float. Since only the weights are quantized, this can be done without requiring
any validation data (See figure 2). A simple command line tool can convert the weights
from float to 8-bit precision. This setup is useful if one only wants to reduce the model
8
我的内容管理
展开
