Skip to content

Machine Learning

The DART-MX95 supports running machine learning workloads on multiple processing units. Models can run on the CPU, be offloaded to the GPU for parallel acceleration, or use the dedicated Neural Processing Unit (NPU) for higher performance and lower power consumption during inference.

For general information about getting started with machine learning, refer to NXP's i.MX Machine Learning User's Guide:

This guide demonstrates how to run a TensorFlow Lite model on the i.MX 95 using NXP's Neutron Delegate, as described in the i.MX Machine Learning User's Guide.

Running a TensorFlow Lite model

NXP provides example TensorFlow Lite models for evaluation and testing. Before these models can run on the i.MX 95 with hardware acceleration, they must first be converted using the eIQ Neutron SDK.

As described in the i.MX Machine Learning User's Guide:

For the offline compilation, the model should be converted through the eIQ Neutron SDK first. In the converted model, the neutronGraph node is already generated. The neutron-delegate only captures the neutronGraph node and offloads the work to Neutron-S.

This conversion process prepares the model for execution on the NPU by inserting the required neutronGraph node, which allows the Neutron delegate to offload supported operations to the Neutron-S accelerator.

The following steps describe how to convert the model using the eIQ Neutron SDK. Please refer to the Yocto BSP Compatibility Matrix for the appropriate eIQ Neutron SDK version to download.

Prepare an Ubuntu 20.04 machine

The eIQ Neutron SDK currently supports Ubuntu 20.04 as the recommended host environment. You can use any Ubuntu 20.04 system, but the example below demonstrates how to quickly create a clean virtual machine using Multipass.

multipass launch 20.04 --name eiq-focal --cpus 8 --mem 8G --disk 60G

Obtain the eIQ Neutron SDK

This guide was tested using eIQ Neutron SDK 2.2.2

Download the eIQ Neutron SDK from NXP: EIQ-NEUTRON-SDK-2.2.2

sha256sum eiq-neutron-sdk-linux-2.2.2-ext.zip
39a8168408280bf33808dd89caeeeb55df7b79d7080782f52f4ad5444921ae1b  eiq-neutron-sdk-linux-2.2.2-ext.zip

If using Multipass, copy the downloaded eIQ Neutron SDK archive to your VM and open a shell in the VM:

multipass transfer eiq-neutron-sdk-linux-2.2.2-ext.zip eiq-focal:
multipass shell eiq-focal

Extract the eIQ Neutron SDK

unzip eiq-neutron-sdk-linux-2.2.2-ext.zip

After extraction, the SDK binaries, including neutron-converter, are available under: eiq-neutron-sdk-linux-2.2.2/bin.

Obtain a sample model from the target

Copy a known-good sample model from your running DART-MX95 to the VM. The example below uses the MobileNet v1 quantized model shipped in Variscite's recovery SD card.

scp [email protected]:/usr/bin/tensorflow-lite-2.19.0/examples/mobilenet_v1_1.0_224_quant.tflite .

Replace 192.168.1.124 with the IP of your target.

Convert the model offline with Neutron

Run the Neutron converter from the eIQ Neutron SDK. This generates a Neutron-enabled .tflite that contains a neutronGraph node which the delegate will offload to Neutron-S on the i.MX 95.

Note: The converter depends on libortools.so and its associated shared object (.so) dependencies. When deploying neutron-converter, ensure that all the provided .so files are available in the system's library search path. This can be achieved, for example, by adding the directory containing the .so files to the LD_LIBRARY_PATH environment variable.

LD_LIBRARY_PATH=$PWD/eiq-neutron-sdk-linux-2.2.2/lib eiq-neutron-sdk-linux-2.2.2/bin/neutron-converter \
    --input  mobilenet_v1_1.0_224_quant.tflite \
    --output mobilenet_v1_1.0_224_neutron.tflite \
    --target imx95

Converting model with the following options:
  Input  = mobilenet_v1_1.0_224_quant.tflite
  Output = mobilenet_v1_1.0_224_neutron.tflite
  Target = imx95

Starting Tile scheduling. This might take a while.
[===============================================================================>] 99 %
Starting TCM allocation. This might take a while.
[================================================================================] 100 %
Starting microcode generation. This might take a while.
[================================================================================] 100 %
Statistics for NeutronGraph "subgraph_030":
  Operators:
    Number of Neutron operators = 29
    Number of builtin operators = 44
  Memory:
    Inputs        = 150,528 (bytes)
    Microcode     = 23,944 (bytes)
    Weights       = 4,329,648 (bytes)
    Kernels       = 11,088 (bytes)
    Outputs       = 381,913 (bytes)
    Scratch       = 380,912 (bytes) (Allocation efficiency: 1)
    Total data    = 913,353 (bytes) (Inputs + Outputs + Scratch)
    Total weights = 4,364,680 (bytes) (Microcode + Weights + Kernels)
    Total size    = 5,278,033 (bytes) (All)
  Latency:
    Cycle estimation   = 1,066,681 (cycles)
    Latency estimation = 1.067 (ms) (@ 1000.000 MHz)

Overall statistics for graph "":
  Operators:
    Number of operators after import    = 31
    Number of operators after optimize  = 47
    Number of operators after extract   = 4
      Number of Neutron graphs          = 1
      Number of operators total         = 47
      Number of operators converted     = 44
      Number of operators NOT converted = 3
    Operator conversion ratio           = 44 / 47 = 0.93617
    Operators converted                 = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,
  Memory:
    Total data    = 532,448 (bytes) (Inputs + Outputs + Intermediate Variable Tensors)
    Total weights = 4,364,688 (bytes) (Weights)
    Total size    = 4,897,136 (bytes) (All)
  Latency:
    Cycle estimation   = 1,066,681 (cycles) (NPU only)
    Latency estimation = 1.067 (ms) (@ 1000.000 MHz) (NPU only)

Conversion time:
  Optimization = 3.04734 (seconds)
  Extraction   = 0.0209273 (seconds)
  Generation   = 7.39501 (seconds)
  Total        = 10.4633 (seconds)

Boot OS with a Kernel DTB file supporting Neutron NPU

At the U-Boot shell, set fdt_file to the DTB that configures and reserves the memory pool for the Neutron NPU. For example, on the DART-MX95 with a Sonata board:

u-boot=> setenv fdt_file imx95-var-dart-sonata-neutron.dtb
u-boot=> boot

Copy the converted model back to the target

Transfer the converted model to your i.MX95 home directory.

scp mobilenet_v1_1.0_224_neutron.tflite [email protected]:

Run inference on the target using the Neutron delegate

Use the label_image example with the external delegate and the converted model.

cd /usr/bin/tensorflow-lite-2.19.0/examples
./label_image \
    --external_delegate_path=/usr/lib/libneutron_delegate.so \
    -m ~/mobilenet_v1_1.0_224_neutron.tflite

INFO: Loaded model /root/mobilenet_v1_1.0_224_neutron.tflite
INFO: resolved reporter
INFO: EXTERNAL delegate created.
INFO: NeutronDelegate delegate: 1 nodes delegated out of 4 nodes with 1 partitions.
INFO: Neutron delegate version: v1.0.0-f24d08e5, zerocp enabled.
INFO: Applied EXTERNAL delegate.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: invoked
INFO: average time: 1.56 ms
INFO: 0.756863: 653 military uniform
INFO: 0.145098: 907 Windsor tie
INFO: 0.0156863: 458 bow tie
INFO: 0.0117647: 466 bulletproof vest
INFO: 0.00784314: 835 suit

Conclusion

This flow demonstrates offline compilation with the eIQ Neutron SDK on a host PC and hardware-accelerated inference on i.MX 95 using the Neutron delegate. Convert once on the host, deploy the Neutron-enabled model to the target, and run using --external_delegate_path=/usr/lib/libneutron_delegate.so for acceleration.