This setup demonstrates:

• Common IFF SDK use case: image delivery to AI application based on artificial neural networks or other machine learning and intelligence technologies
• Recently added Vulkan IFF SDK component, bringing support for image processing on a wide range of GPUs
• Integration of IFF SDK and GStreamer pipelines, which allows to re-use parts of existing GStreamer-based solutions, while utilizing IFF SDK capabilities to interface with machine vision cameras and to quickly pre-process acquired images
• IFF SDK support for various hardware platforms (Intel and ARM architectures) and operating systems (Linux and Windows), including Yocto-based Linux ARM64 distributions
• Capabilities of MRTech team for project work – this prototype was prepared in a month’s time from idea to working solution, including implementation of mentioned above Vulkan and GStreamer IFF SDK features

Hardware

• Qualcomm RB3 Gen 2 Core Kit, Qualcomm Linux (Yocto-based)
• USB3.1 (4 Gbps) camera XIMEA MC031CG-SY-UB (Sony IMX252 sensor), 12 bit 960×1080 ROI (half-FullHD, 1 MPix) 60 FPS

Processing pipeline

• acquisition of raw Bayer frames from camera [CPU]
  • writing to disk in DNG format (by command) [CPU]
  • histogram calculation [CPU]
    • auto white balance and auto exposure [CPU]
• black level subtraction [GPU, Vulkan]
• white balance [GPU, Vulkan]
• HQLI demosaic (5×5 window) [GPU, Vulkan]
• DCP (DNG color profile) color correction (matrix only) [GPU, Vulkan]
• gamma correction [GPU, Vulkan]
  • render on the screen [GPU, gstreamer]
• AI (neural network) depth estimation [DSP/NPU, GPU, gstreamer]
• render on the screen [GPU, gstreamer]