MRTech solutions for NVIDIA Jetson platform

Embedded Vision and NVIDIA Jetson computers

We introduce compact comprehensive solutions based on the NVIDIA Jetson supercomputer modules for multi-camera embedded, 3D Vision, airborne and other mobile applications.

It means imaging systems which rely on small feedback time, like autonomous cars, copters or drones, mobile robots that should be remotely controlled via Ethernet, wireless network, radio-connection, etc., and when just one camera can’t be enough to get full info from controlling objects.

As general, the solutions include XIMEA cameras, NVIDIA computer hardware and MRTech SK software.

 

Applications

  • Drones and UAV, Aerial mapping system
  • Autonomous Robotic Systems
  • Industrial Embedded Imaging
  • 3D Vision

Multi-camera configuration

In total, up to eight high-performance cameras can be used and connected to Jetson in a single system, including:

  • XIMEA PCIe cameras of xiX series
  • XIMEA USB3 cameras of xiC and xiQ series
  • USB3 cameras from other manufacturers
  • Cameras can have a resolution up to 50 megapixels, and their number is mainly determined by the performance of the chosen NVIDIA Jetson module

Hardware Configuration

  • NVIDIA Jetson TX1 / TX2 / Nano / Xavier computers
  • XIMEA xEC2 customized carrier for Jetson TX1 / TX2 /TX2i
  • ConnectTech carriers for Jetson modules such as TX2 or Xavier
  • Jetson Nano and AGX Xavier Developer Kits
  • Receiving and communication hardware

 

Software

To meet customer’s requirements every solution has all necessary software to start using it in an application. Uploaded software package usually includes:

  • Linux for Tegra (L4T) operating system
  • CUDA computing platform with necessary utilities
  • XIMEA programming interfaces (xiAPI) and tools
  • MRTech run-time application with arranged pipelines for all connected cameras with a wide variety of processing modules such as:
    • acquisition and synchronization
    • color images pre-processing, RAW processing, format transformations
    • SSD writing, rendering on a screen
    • H.264/H.265 encoding, RTSP streaming, etc.
  • user and program interfaces to control cameras and application

When necessary, MRTech SK provides customization of its software in accordance with an agreed specification.

 

Embedded Vision cases

We present here several embedded vision cases those built on XIMEA digital cameras, NVIDIA Jetson computers, and with MRTech IFF software. For each case, a brief description of the hardware / software configuration and performance results is provided.

Every case (or kit) includes one or several XIMEA cameras, a separate NVIDIA embedded module, an appropriate carrier for the module, and necessary accessories.

Software from MRTech arranges standard or customized image processing pipeline running on NVIDIA module with following common processing steps:

  • acquisition images from cameras
  • color pre-processing including black level, white balance, demosaic
    • for GPU processing, we usually utilize Fastvideo SDK
  • additional options such as autoexposure, gain control when needed
  • H.264/H.265 encoding, RTSP streaming via Ethernet or Wi-Fi.

RTSP streams from cameras can be received using VLC app on one of the devices such as Macbook with MacOS, desktop / notebook with Windows or Linux OS, Apple devices such as iPhone, iPad, etc. Nonetheless, for all these cases we used one or another receiving computer equipped with NVIDIA graphic card, 144Hz refresh rate display, and MRTech software for receiving streams and rendering images to measure glass-to-glass (G2G) latency. Other performance figures show the processing pipeline time not including acquisition, data transfer times and network performances.

3-cam TX2 kit

  • MX200CG-CM XIMEA camera and
  • 2x MX031CG-SY XIMEA camera
  • Nvidia Jetson TX2 with XIMEA xEC2 carrier
  • Dell XPS laptop as a receiving station

Standard processing pipeline:

  • 12-bit acquisition
  • color pre-processing
  • H.264 encoding and RTSP streaming

Operating results:

  • 2x FullHD 1080p (1920×1080) ROI @ 30 FPS
  • WQHD 1440p (2560×1440) binning @ 30 FPS

2-cam TX2 kit

  • 2x MX031CG-SY XIMEA camera
  • Nvidia Jetson TX2 with XIMEA xEC2 carrier
  • Laptop or desktop as a receiving station

Standard processing pipeline:

  • 8-bit acquisition
  • Color pre-processing
  • H.264 encoding and RTSP streaming

Operating results:

  • 2x Full frame 3.1 MP (2064×1544) @ 60 FPS
  • processing time: 15ms for both cameras
  • G2G latency: < 60 ms (with 144 Hz monitor)

High resolution with TX2

  • MX500CG-CM-X4G2 XIMEA camera
  • NVIDIA Jetson TX2 module
  • Connect Tech Elroy Carrier for TX2

Customized processing pipeline:

  • 8/10/12 acquisition
  • RAW processing
  • color pre-processing
  • H.265 encoding
  • RTSP multi streaming

Processing results:

  • 47.5 MP, 7920 x 6004 @ 5 FPS

Two USB3 cameras with TX2

  • 2x MC031CG-SY-UB XIMEA camera
  • NVIDIA Jetson TX2 module
  • Connect Tech Elroy carrier
  • desktop GPU computer as a receiving station

Standard processing pipeline:

  • 8-bit acquisition from both cameras
  • Color pre-processing
  • H.264 encoding and RTSP streaming

Operating modes:

  • Full frame images:
    • 2x 3.1 MP (2064×1544) @ 50 FPS
    • G2G latency: 80-100 ms
  • Full HD images:
    • 2x FullHD 1080p (1920×1080) ROI @ 60 FPS
    • IFF processing: 15ms for both cameras
    • G2G latency: 70 – 80 ms

3-cam setup with Xavier

  • 2x MC031CG-SY XIMEA camera
  • MX124CG-SY-FL(FV) XIMEA camera
  • Jetson AGX Xavier Developer Kit
  • GPU desktop as a receiving station

Standard full frame processing:

  • 8/10/12-bit acquisition
  • color pre-processing
  • H.264/H.265 encoding, RTSP streaming

Operating results:

  • 8-bit – up to 60/60/60 FPS
  • 12-bit – up to 60/60/45 FPS with
    • processing latency ~ 30/30/45 ms

High performance with Xavier

  • CB262CG-GP-X8G3 XIMEA camera with
    • GPixel GMAX0505 sensor
  • NVIDIA Jetson AGX Xavier dev. kit

Standard demo application:

  • 12-bit acquisition
  • color pre-processing
  • render on a screen

Example of operating results:

  • 26.2 MP 5120 x 5120 (12-bit) @ 39 FPS
  • total GPU + CPU processing: 29 ms
  • G2G latency: 90 – 150 ms