Machine vision camera types overview

There are a variety of machine vision cameras available on the market from different vendors, with different hardware interfaces, implementing different data protocols and accessible via different programmatic APIs. In this paper, we will do a brief overview of most options and see how it’s possible to use each type of camera with MRTech IFF SDK.

MRTech demo case for Embedded World 2024 expo with PCIe, USB, GigE and MIPI cameras (left to right)

MIPI cameras

MIPI standards are popular in embedded applications, such as mobile phones, and MIPI cameras are also supported by NVIDIA Jetson platform. Specifically for camera connectivity MIPI CSI-2 standard is often used, with D-PHY and C-PHY standards being the most popular implementations on a physical level. Several cable and connector options are available, the most common being flexible flat cable (FFC). Cable length is usually limited to values below one meter, though standard supports distances up to four meters. The connection between the camera and host can have several lanes, allowing to scale the bandwidth. Usually, two or four lanes are used. Maximum speed per lane theoretically can be up to 18 Gbps, but in practice, it’s defined by receiver implementation. For example, Jetson AGX Orin limits C-PHY speed to about 10 Gbps, which is still an impressive figure. Several vendors manufacture MIPI cameras, to name a few: Leopard Imaging, Arducam, FRAMOS, Basler, etc.

IFF SDK supports MIPI cameras through Video4Linux2 API (Linux-only), used in particular on the NVIDIA Jetson platform. Here is an example of camera source configuration:

{
  "id": "cam",
  "type": "v4l2cam",
  "cpu_device_id": "cpu_dev",
  "v4l2_device": "/dev/video0",
  "black_level": 64,
  "exposure": 10000,
  "fps": 60.0,
  "sensor_mode": 3,
  "pixel_format": "RG10",
  "width": 1920,
  "height": 1080,
  "preallocate_buffers": 6,
  "custom_params": [
    { "override_enable": 1 },
    { "bypass_mode": 0 }
  ],
  "gain": 64
}

 

Several NVIDIA-specific settings warrant a separate explanation:

  • black_level parameter should be set to the value available in camera_overrides.isp file provided by camera/driver vendor (average of opticalBlack.manualBias* parameters).

  • Make sure that sensor_mode is set correctly according to width, height and pixel_format settings. Failure to do so may result in a broken picture (“interlaced” look) or acquisition failure. Consult camera driver description to determine the correct settings.

  • Setting preallocate_buffers is required, as opposed to usual automatic buffer number calculation implemented in IFF SDK, due to lack of support in NVIDIA drivers for dynamic buffer allocation in the case of MIPI image acquisition.

  • override_enable parameter ensures correct handling of settings like exposure and gain, otherwise, values can become de-synchronized between camera and OS.

  • bypass_mode needs to be reset in case libargus applications (utilizing NVIDIA hardware ISP) were used in the same boot session.

GMSL cameras

GMSL (Gigabit Multimedia Serial Link) is an interface standard aimed at automotive applications. It allows passing of MIPI CSI-2 signal from an image sensor (providing also power and access to sensor registers) through a coaxial cable up to 15 meters long. Version 2 of GMSL standard provides up to 6 Gbps of bandwidth. In the case of the NVIDIA Jetson platform, one can use either a special expansion or carrier board with integrated GMSL deserializer, interfacing with Jetson’s MIPI CSI-2 interface. So on the software side, GMSL cameras can be used exactly as MIPI ones, at least in the case of the NVIDIA Jetson platform. Usually same vendors, that have MIPI cameras in their portfolio, also manufacture GMSL cameras.

USB cameras

USB is the most common computer interface used for connecting external devices (flash drives, printers, keyboards, mice, etc.). Several revisions exist, varying mainly in bandwidth provided. The most popular standard among machine vision cameras is USB3.1 with 4 Gbps effective transfer speed. Cable length is usually limited by five meters unless active cables are utilized. The most widely supported protocol is USB3Vision, though UVC (used by web cameras) and custom vendor protocols are also used by some cameras on the market. Most, if not all, machine vision camera vendors have USB cameras in their portfolio, e.g. XIMEA, Basler, Lucid, etc. On the software level, several options exist:

  • UVC cameras on Linux can be accessed using Video4Linux2 interface. So in the case of IFF SDK same camera source configuration can be used as with MIPI cameras, sans NVIDIA-specific quirks mentioned.

  • Both custom and USB3Vision protocols are usually supported by vendor-provided SDK, such as pylon from Basler or xiAPI from XIMEA. IFF SDK supports image acquisition through xiAPI, here’s an example configuration for such a case (adjust serial_number for your camera):

{
  "id": "cam",
  "type": "xicamera",
  "cpu_device_id": "cpu_dev",
  "serial_number": "CECAU2302000",
  "image_format": "RAW16",
  "custom_params": [
    { "bpc": 1 }
  ],
  "exposure": 5000,
  "fps": 90.0,
  "gain": 3.0
}

bpc parameter enables bad pixel correction functionality of xiAPI. Consult xiAPI manual for a list of all available parameters for custom_params section: https://www.ximea.com/support/wiki/apis/XiAPI_Manual

  • A more universal and vendor-agnostic option, while still supporting both USB3Vision and custom protocols, is GenICam standard. Machine vision camera vendor usually provides a library called GenTL Producer with a standardized API for acquiring images from the camera and manipulating its settings. To use a GenTL-compatible camera in IFF SDK, the following configuration can be used (adjust serial_number for your camera):

{
  "id": "cam",
  "type": "genicam",
  "cpu_device_id": "cpu_dev",
  "producer": "/opt/pylon/lib/gentlproducer/gtl/ProducerU3V.cti",
  "serial_number": "40372229",
  "pixel_format": "BayerRG8",
  "custom_params": [
    { "BslColorSpace": "Off" }
  ],
  "exposure": 5000,
  "fps": 50.0,
  "gain": 3.0
}

producer setting in this example references GenTL Producer from Basler for their USB3Vision cameras. BslColorSpace setting is Basler-specific feature, disabling it ensures that raw data streamed from the camera is linear and not gamma-corrected since gamma correction is usually done at the later stage of IFF SDK pipeline.

Ethernet cameras

Ethernet network equipment is very widespread, ensuring low prices, good availability and quality. It also means that a lot of times infrastructure is already in place for camera deployment. Provided bandwidth increased over time, with 10 Gbps becoming accessible for common use, which is quite on par with other interfaces described here. Copper cables can be up to 100 meters long, and with switches or optical links, the length can be increased further. On the protocol level, two main options exist:

  1. GigEVision – aimed specifically for machine vision cameras, capable of streaming raw data;
  2. RTP, RTSP and other compressed streaming protocols – are used mainly by surveillance cameras, but can also be utilized in machine vision applications for remote delivery of processed images.

Many vendors produce Ethernet machine vision cameras, Basler and Lucid being among them, to name a few.

On the software level used API depends on protocol:

  1. GenICam (GenTL Producer) or vendor SDK can be used to access GigEVision cameras. Please refer to the description in USB cameras chapter on how to use it with IFF SDK.
  2. To receive a RTSP stream the following configuration can be used in IFF SDK (adjust url as needed):
{
  "id": "cam",
  "type": "rtsp_source",
  "cpu_device_id": "cpu_dev",
  "url": "rtsp://192.168.55.1:8554/cam"
}

 

PCIe cameras

PCIe is a relatively rare interface for machine vision cameras, only two vendors manufacture such devices: XIMEA and Balluff (former Matrix Vision). However, it has a big advantage in possible bandwidth and reduces latency and CPU load through DMA (direct memory access) technology. On a physical level, several revisions exist with different speeds per lane, ranging from 2.5 Gbps (PCIe 1.0) to 10 Gbps (PCIe 3.0). Cameras can provide connection with up to 8 lanes, which gives an astounding 80 Gbps total bandwidth. Several connector types are available, such as FFC, iPass, Firefly, MTP. With the latter three options, optical cables can be used, allowing for lengths up to 100 meters. On the software level, same as with USB and Ethernet, two main options exist – vendor SDKs and GenTL Producers. For example IFF SDK configuration see USB cameras chapter.

Comparison

  • MIPI cameras are best suited for small form-factor devices. Bandwidth and latency are at the top range, only with some PCIe cameras surpassing them. The main problem is that MIPI interface is only available on embedded devices like NVIDIA Jetson, while connecting a MIPI camera to an ordinary PC may be tricky. Another downside is limited functionality of vendor drivers, when sometimes you may find yourself in need of driver customization to implement the desired sensor mode. This may increase the project price and development time.

  • GMSL cameras are similar to MIPI cameras, but provide an advantage in the form of the longer possible length of the cable. They dominate the automotive market.

  • USB cameras are the easiest to use – USB ports are available on every computer nowadays. Bandwidth is in a mid-range, but depending on the application it may as well be enough. Software USB stack may increase the latency and CPU load somewhat compared to MIPI or PCIe.

  • Ethernet cameras are the most flexible regarding their placement relative to the computing module, thanks to very well-developed network infrastructure possibilities. Most common devices provide only 1 Gbps of bandwidth, which is on the low end, but more expensive options exist with 2.5 / 5 / 10 Gbps interfaces, making them competitive with other options. But software stack overhead in the case of 10 Gbps interface can be a problem for CPU load and latency.

  • PCIe cameras are the best choice when bandwidth and/or latency are crucial. The main limitation would be the limited range of camera models available on the market.

 

Honorary mentions

  • Camera Link – still in wide use, mainly in legacy systems, but in our view an outdated interface. It can be used with IFF SDK through GenTL Producer library from the camera vendor.

  • FireWire – extinct technology, but machine vision cameras used to be produced with this interface, e.g. by XIMEA, back in the day.

 

Conclusion

Each machine vision application has its own requirements, which results in different cameras being the optimal choice for each particular situation. MRTech IFF SDK abstracts away most of the differences between camera types and their quirks. Together with high-quality technical support, ensured by long-term MRTech team experience in machine vision and image processing area, IFF SDK allows you to focus on more interesting parts of your application.

As usual, you can contact us (support@mr-technologies.com) with any questions or to get a free MRTech IFF SDK software evaluation package. The technical manual is available online, providing a description of all available parameters of the mentioned camera source components and other IFF SDK functionality.

Getting started

Latency measurement

IFF SDK on Github