![](https://csdnimg.cn/release/download_crawler_static/21044249/bg10.jpg)
• The Python API (recommended)
• Command line script
The post training quantization using the Python API is described in this chapter. The documentation useful for model conversion
and quantization is available here:
• Python API documentation: https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/lite/TFLiteConverter
• Guide for model conversion: www.tensorflow.org/lite/convert
• Guide for model quantization:https://www.tensorflow.org/lite/performance/post_training_quantization
The guides on TensorFlow page usually covers the most up to date version of TensorFlow, which might be different
from the version available in the NXP eIQ. To see what features are available, check the corresponding API for the
specific version of the TensorFlow or TensorFlow Lite.
NOTE
The current version of the TensorFlow Lite available in the NXP eIQ is 2.4.0. It is recommended to use the TensorFlow Lite
converter from corresponding TensorFlow version. The TensorFlow Lite runtime should be compatible with models generated
by previous version of TensorFlow Lite Converter, however this backward compatibility is not guaranteed. Usage of successive
version of TensorFlow Lite converter shall be avoided.
The 2.4.0 version of the converter has the following properties:
• In the post training quantization regime, the per-channel quantization is the only option. The per-tensor quantization is
available only in connection with quantization aware training.
• Input and output tensors quantization is supported by setting the required data type in inference_input_type
and inference_output_type.
• TOCO or MLIR based conversions are available. This is controlled by the experimental_new_converter attribute. As TOCO
is becoming obsolete, MLIR-based conversion is already set by default in the 2.4.0 version of the converter.
MLIR converter uses dynamic tensor shapes, what means the batch size of the input tensor is unspecified. Dynamic tensor
shapes are not supported, by the GPU and NPU hardware accelerators and this shall be turned off. Standard installation
of TensorFlow does not provide API to control the dynamic tensor shape feature, but can be deactivated in the tensorflow
instalation, as follows. Locate the <python-install-dir>/site-packages/tensorflow/lite/python/lite.py file and
change the private method TFLiteConverterBase._is_unknown_shapes_allowed(self) to return False value, as follows:
def _is_unknown_shapes_allowed(self):
# Unknown dimensions are only allowed with the new converter.
# Return self.experimental_new_converter
# Disable unknown dimensions support.
return False
MLIR is a new NN compiler used by TensorFlow, which supports quantization. Before MLIR, quantization was
performed by TOCO (or TOCO Converter), which is now obsolete. See https://www.tensorflow.org/api_docs/
python/tf/compat/v1/lite/TocoConverter. For details about MLIR, see https://www.tensorflow.org/mlir.
NOTE
Do not use the dynamic range method for models being run on NN accelerators (GPU or NPU). It converts only the
weights to 8-bit integers, but retains the activations in fp32, which results in the inference running in fp32 with an
additional overhead for data conversion. In fact, the inference is even slower compared to a fp32 model, because
the conversion is done on the fly.
NOTE
For the full-integer post training quantization, a representative dataset is needed. The proper choice of samples in representative
dataset highly influences the accuracy of the final quantized model. The best practices for creating the representative dataset are:
NXP Semiconductors
TensorFlow Lite
i.MX Machine Learning User's Guide, Rev. LF5.10.9_1.0.0, 31 March 2021
User's Guide 16 / 88