System prototype of 4K stereoscopic camera
In 2022, our company MRTech designed, assembled, tested, and delivered to one of our customers a prototype of 4K side-by-side stereoscopic camera based on NVIDIA Jetson for an entertainment application.
The main objective of the prototype was to demonstrate the ability to capture two image streams from two high-resolution MIPI cameras, process and encode the streams on NVIDIA Jetson Xavier NX, and send the high-quality streams to the network. These images can then be fed separately to each eye and be processed by the brain to reproduce a 3D image as if the scene were viewed by a human being. The target resolution and performance of the prototype were two 1920×2016 image streams at 60 FPS, which together constitute a 4K (3840×2016) stereoscopic image at 60 FPS.
The customer would also like to be able to select fisheye lenses, adjust lens spacing, control ROI, offsets and resulting image quality, calibrate the color of output images, etc.
Below is how we did it.
The prototype was assembled on a specially designed stand using the following components:
- two Arducam IMX477 HQ Camera Modules with Sony IMX477 sensors
- two Entaniya 220 degree M12 4K Fisheye lenses
- two FFC cables with 22-pin and 0.50mm pitch
- NVIDIA Jetson Xavier NX
- Auvidea JNX30D carrier for NVIDIA Jetson Xavier NX
- power supply 12V, 3A
See the picture of what this system looks like.
Prototype software configuration:
- L4T operating system
- Auvidea firmware for Jetpack 4.6 to support Auvidea JNX30D carrier board for NVIDIA Jetson Xavier NX
- IMX477 sensor driver with operating mode for 3840×2160, 10-bit image acquisition at 60 FPS
- MRTech IFF SDK and its sample farsight application
The farsight application runs on Jetson with the following image processing pipeline:
- 10-bit input images
- image crop
- black level
- auto-exposure and white balance
- demosaic with different options
- color correction with a pre-created color profile
- recording the resulting images to a file
- H.265 encoding with various bitrates
- RTSP streaming (Ethernet via USB)
Here are the results of the prototype testing:
- Input image size: 2x 1920×2160 @ 10-bit @ 60 FPS
- Processing image bitdepth: 16-bit
- Demosaic algorithm: DFPD with 11×11 window
- Encoder bitrate: 10 – 100 Mbps
- Output framerate: 60 FPS
- Xavier NX CPU load for each core: ~ 10%
- Xavier NX GPU engine load: ~ 90%
- Xavier NX and camera power consumption: 19.4 W
Check out the video to see how it works.
To learn more about the IFF SDK and to study the SDK documentation, please follow the links below.
IFF SDK for image processing
MRTech on Github
