This setup demonstrates:

• IFF SDK capability to operate with a wide range of cameras – various hardware interfaces (PCIe, USB, Ethernet, MIPI), vendors (Basler, Leopard Imaging, LUCID, XIMEA) and software APIs (XIMEA API, GenTL, Video4Linux)

• Very low glass-to-glass latency (<50 ms) required for real-time interactive applications such as tele-operation, augmented reality, etc.

• Good color reproduction (and image quality in general, as proven by usage in Pawel Achtel’s 9×7 cinema camera), which doesn’t depend on camera vendor implementation, thus providing more freedom in hardware choice without sacrificing the image quality

• IFF SDK ability to run on different operating systems (Linux, Windows) and hardware architectures (ARM64, x86)

• IFF SDK interoperability and support of the following technologies:

  • HTTP – used for pipeline control from web browser or other HTTP clients

  • WebRTC – used for video streaming to web browser

  • RTSP – used for video streaming, any RTSP client is capable of receiving the stream (but our solution is specifically targeted at low latency)

  • OpenCV – used for rendering, sample application source code available on GitHub

• IFF SDK functionality available out-of-the-box (no code modifications, only configuration files adjustments) with three sample applications provided with IFF SDK package and available on GitHub:

  • farsight – streaming from Jetson

  • imagebroker – stream receiving and render on laptop

  • crowsnest – web interface (parameters control, WebRTC video streaming) served from Jetson

Hardware

• NVIDIA Jetson AGX Orin devkit 32GB, MAXN power mode with jetson_clocks running
• 2.5 Gbps Ethernet switch with PoE (which powers GigE camera)
• MSI laptop, Intel Core i9, 32 GB RAM, NVIDIA GeForce RTX 3070 Ti 8GB, 240Hz 2560×1440 display, 2.5Gbps Ethernet, Windows 11

Cameras

1. PCIe (2 lanes Gen2, 8Gbps, FireFly) camera XIMEA MX031CG-SY-X2G2-FF (Sony IMX252 sensor) with 1920×1080 ROI @ 12 bit / 90 FPS (limited by exposure, can be up to 169 FPS for 12 bits)
2. USB3.1 (4 Gbps) camera Basler daA3840-45uc (Sony IMX334 sensor) with 1920×1080 ROI @ 12 bit / 60 FPS (limited by camera/sensor)
3. GigE Vision (2.5 Gbps) camera LUCID TRT024G-CC (Gpixel GMAX4002 sensor) with 1920×1080 ROI @ 12bit / 60 FPS (limited by camera, can be up to 107 FPS for 8/10 bits)
4. MIPI (4 lanes D-PHY V1.2, 8Gbps) camera Leopard Imaging LI-IMX477-MIPI (Sony IMX477 sensor) with 2×2 binning and 1920×1080 ROI @ 10 bit / 60 FPS (limited by driver)

Processing pipeline

• acquisition of raw Bayer frames from cameras [CPU, DMA]
  • writing to disk of DNG files (by command) [CPU]
• bitdepth conversion (to 16 bits) [CUDA]
• black level subtraction [CUDA]
  • histogram calculation [CUDA]
  • auto white balance and auto exposure [CPU]
• white balance [CUDA]
• DFPD demosaic (11×11 window) [CUDA]
• DCP (DNG color profile) color correction (matrix and LUT) [CUDA]
• gamma correction [CUDA]
• conversion to YUV format [CUDA]
• H.264 (H.265 also possible) compression with 10 Mbps bitrate [NVENC]
• RTSP streaming [CPU]
• WebRTC re-streaming (provided by janus open-source software) [CPU]

Glass-to-glass latency

with 1ms exposure time

1. PCIe: ~35ms (30-40ms), 40ms for WebRTC
2. USB3: ~40ms (30-50ms)
3. GigE: ~50ms
4. MIPI: ~35ms (30-40ms)