MRTech solutions for NVIDIA Jetson platform

Jetson is an advanced series of high-performance edge-computing modules from NVIDIA. This platform is designed to accelerate computing in smart machines and AI applications, enabling users to develop and deploy breakthrough products across multiple industries.

Each Jetson module carries a powerful system-on-chip (SoC) from NVIDIA, integrated with an ARM architecture CPU, and designed to be energy efficient and have a compact form factor. So, the architecture and functionality of modules like Jetson Orin or Xavier are ideal for using them together with the high-performance image processing IFF SDK offered by MRTech SK. The following benchmarks for Jetson Orin and descriptions of use cases with other Jetson modules confirm this very well.

Let’s remember that MRTech IFF SDK is a powerful cross-platform toolkit that facilitates the development of high-performance machine vision and image processing applications. The SDK pre-processes images received from the camera (or disc) and delivers them to the target application in the required format and in an impressively efficient manner.

To learn more about MRTech IFF SDK, please visit our company’s Github account, where documentation and the SDK sample applications are now available.

When necessary, we can provide you with a demo version of the SDK on request. Please specify the target system configuration in the request.

Jetson Orin

Jetson Xavier

Jetson TX2

IFF SDK for Jetson

AGX Orin and Orin NX series

 

NVIDIA describes its Jetson Orin modules as the most powerful computers for energy-efficient autonomous vehicles and next-generation machine vision systems. The MRTech team verified this by benchmarking the performance of AGX Orin and Orin NX using the latest version of our IFF SDK and its most general sample application farsight.

The application enabled the following image processing pipeline:

  • acquisition of 8/10/12-bit images (full frame or ROI) from XIMEA camera,
  • color pre-processing on GPU including black level subtraction, histogram calculation, white balance, demosaicing (HQLI, L7, DFPD, MG algorithms at choice), color correction, gamma, and image format conversion,
  • automatic control of exposure time and white balance,
  • H.265 encoding and RTSP streaming over the network to a receiving computer.

Orin testing configuration

Image source: XIMEA CB120-CM-X8G3 camera

Transmitting computer: AGX Orin 32GB or Orin NX 16 GB that is emulated with AGX Orin devkit

  • Jetpack 5.1.2 [L4T 35.4.1]
  • IFF SDK v1.6, farsight sample application

Receiving computer: MSI Raider laptop with NVIDIA GPU

  • Windows 11 OS
  • MRTech DragonNest application to receive, decode, render on a screen

AGX Orin: Full HD image @ 8-bit

  • Image processing pipeline:
    – input image format RAW8
    – color pre-processing with fast HQLI demosaic (5×5)
    – auto-exposure and auto-white balance
    – H.265 encoding with 104 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – Full HD (1920×1080 ROI) image @ 8-bit @ 250 FPS
    – processing latency on AGX Orin: 6.4 ms
    – AGX Orin GPU avg. load: 30%
    – full G2G latency: 20-30 ms

AGX Orin: Full HD image @ 12-bit

  • Image processing pipeline:
    – input image format RAW16
    – color pre-processing with high-quality MG demosaic (23×23)
    – auto-exposure and auto-white balance
    – H.265 encoding with 104 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – Full HD (1920×1080 ROI) image @ 12-bit @ 250 FPS
    – processing latency on AGX Orin: 7.4 ms
    – AGX Orin GPU avg. load: 45%
    – full G2G latency: 20-30 ms

AGX Orin: 4K UHD image @ 8-bit

  • Image processing pipeline:
    – input image format RAW8
    – color pre-processing with fast HQLI demosaic (5×5)
    – auto-exposure and auto-white balance
    – H.265 encoding with 83 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – 4K UHD (3840×2160 ROI) image @ 8-bit @ 100 FPS
    – processing latency on AGX Orin: 17.6 ms
    – AGX Orin GPU avg. load: 20%
    – full G2G latency: 40-50 ms

AGX Orin: 4K UHD image @ 12-bit

  • Image processing pipeline:
    – input image format RAW16
    – color pre-processing with high-quality MG demosaic (23×23)
    – auto-exposure and auto-white balance
    – H.265 encoding with 83 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – 4K UHD (3840×2160 ROI) image @ 12-bit @ 100 FPS
    – processing latency on AGX Orin: 21.4 ms
    – AGX Orin GPU avg. load: 57%
    – full G2G latency: 50-60 ms
    See below Orin and laptop screenshots for this case.

Orin NX: Full HD image @ 8-bit

  • Image processing pipeline:
    – input image format RAW8
    – color pre-processing with fast HQLI demosaic (5×5)
    – auto-exposure and auto-white balance
    – H.265 encoding with 50 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – Full HD (1920×1080 ROI) image @ 8-bit @ 180 FPS
    – processing latency on Orin NX: 8.5 ms
    – Orin NX GPU avg. load: 33%
    – full G2G latency: 20-30 ms

Orin NX: Full HD image @ 12-bit

  • Image processing pipeline:
    – input image format RAW16
    – color pre-processing with high-quality MG demosaic (23×23)
    – auto-exposure and auto-white balance
    – H.265 encoding with 50 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – Full HD (1920×1080 ROI) image @ 12-bit @ 180 FPS
    – processing latency on Orin NX: 10.8 ms
    – Orin NX GPU avg. load: 64%
    – full G2G latency: 30-40 ms

Orin NX: 4K UHD image @ 8-bit

  • Image processing pipeline:
    – input image format RAW8
    – color pre-processing with fast HQLI demosaic (5×5)
    – auto-exposure and auto-white balance
    – H.265 encoding with 50 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – 4K UHD (3840×2160 ROI) image @ 8-bit @ 60 FPS
    – processing latency on Orin NX: 27.5 ms
    – Orin NX GPU avg. load: 28%
    – full G2G latency: 50-60 ms

Orin NX: 4K UHD image @ 12-bit

  • Image processing pipeline:
    – input image format RAW16
    – color pre-processing with high-quality MG demosaic (23×23)
    – auto-exposure and auto-white balance
    – H.265 encoding with 50 Mbps bitrate
    – RTSP streaming over network
  • Results:
    – 4K UHD (3840×2160 ROI) image @ 12-bit @ 60 FPS
    – processing latency on Orin NX: 36.2 ms
    – Orin NX GPU avg. load: 78%
    – full G2G latency: 60-70 ms

AGX Xavier and Xavier NX series

The predecessor of the Jetson Orin family was the Jetson Xavier series, with which NVIDIA has five different production modules in two form factors (AGX and NX) based on the same architecture.

Our company has carried out several successful machine vision projects with Xavier modules, see some use cases below.

Multi-camera systems with AGX Xavier

for remotely operated machines

Hardware options:

  • 2 or 3 or 4 cameras depending on the required configuration:
    – PCIe FireFly camera XIMEA MX124CG-SY-FF
    – PCIe FireFly camera XIMEA MX031CG-SY-FF
    – USB3 camera XIMEA MC031CG-SY
    – USB3 camera XIMEA MC051CG-SY
  • Jetson AGX Xavier 32GB module
  • GPU desktop as a receiving station

Image processing pipeline:

  • 8/10/12-bit acquisition
  • full color pre-processing cycle
  • H.264/H.265 encoding, RTSP streaming

Operating results:

  • Processing up to 1 GPixel per second in total
  • From 20 to 40 ms Xavier processing latency for each camera

WebRTC system with Xavier NX

for streaming images from a MV camera to a client browser

Hardware:

  • XIMEA MX031CG-SY-X2G2 camera with 3.1 MPixel resolution and PCIe interface
  • Jetson Xavier NX 16GB module
  • Auvedia carrier board

Image processing pipeline:

  • full frame 12-bit image acquisition
  • color pre-processing on GPU
    – including DFPD (11×11) quality demosaic
  • automatic control of exposure time and white balance
  • H.265 encoding and streaming using WebRTC protocol to show images in the client web browser
  • HTTP interface to remotely control acquisition and processing parameters at runtime

Operating results: 2058 x 1544 @ 60 FPS via browser

Edge multi-sensor system with AGX Xavier

for medical applications

System hardware:

  • MIPI sensor array (32x 24 MPixel) placed on a specially designed board connected to the carrier
  • AGX Xavier 64GB with ConnectTech Rogue AGX101 carrier board

Software application:

  • custom image processing pipeline
  • image transfer to the host computer
  • acquisition, processing, laser light control
  • API for remote control

Operating results:

  • simultaneous image acquisition from selected four sensors
  • 5 GB/s total bandwidth

4K stereo MIPI camera based on Xavier NX

for entertainment

System hardware:

  • two MIPI camera modules with Sony IMX477 sensor
  • NVIDIA Jetson Xavier NX 16GB with Auvidea carrier board
  • desktop or laptop as a receiving station

Software configuration:

  • IMX477 sensor driver operating in 3840×2160 @ 10-bit @ 60 FPS mode
  • crop to get two 1920×2160 images
  • color pre-processing of cropped images on GPU
    – including DFPD (11×11) quality demosaic
  • H.265 encoding and RTSP streaming

Operating results:

  • 4K stereo side-by-side images: 2x 1920×2160 @ 60 FPS

Jetson TX2 / TX2i / TX2 NX modules

NVIDIA launched its Jetson TX2 in 2017; before that, the TX1 was started in 2015. The TX2 series is still popular in the market, although new projects tend to focus on more modern NVIDIA architectures.

MRTech has completed many machine vision projects based on TX1/TX2 modules and these solutions, developed technologies, and experience gained in their implementation allowed us to launch our IFF SDK in early 2022.

Some examples of our solutions with different modifications of TX2 are given below.

High-resolution camera with TX2

for embedded application

System hardware:

  • PCIe XIMEA camera MX500CG-CM-X4G2 (48 MP)
  • industrial Jetson TX2i module with CT Elroy carrier

Customized processing pipeline:

  • 8/10/12 acquisition
  • RAW processing
  • custom color pre-processing
    – including demosaic of varying quality and performance
  • H.265 encoding
  • RTSP multi streaming

Processing results: 47.5 MP, 7920 x 6004 @ 5 FPS

Two camera-headset with TX2

for medical equipment

System hardware:

  • Two USB3 subminiature MV cameras XIMEA MU181CR-ON
    Jetson TX2 module with Auvidea J121 carrier board
  • desktop or laptop as a receiving station

Image processing pipeline:

  • image acquisition from two cameras with 2×2 binning
  • color pre-processing with DFPD (11×11) quality demosaic
  • H.265 encoding
  • recording of compressed images
  • RTSP streaming
  • manual and HTTP remote control interfaces

Operating results: 2x 4.5M 2448×1840 @ 30 FPS

 

Multi-camera systems with TX2

for drones

MRTech has participated in the development of several vision systems for drones, where we were responsible for delivering camera images to the user’s application on the ground. In our experience, the Jetson TX2 resources allow it to be used with up to three cameras of different resolutions. A typical example of such a camera configuration is two Full HD cameras (2x 1920×1080 @ 30 FPS) and one WQHD camera (2560×1440 @ 30 FPS) working simultaneously.