Live panorama image synthesis in GPU

	23.4.2010 \|

Very high computational power of modern programmable GPUs (Graphics Processing Units) enables implementation of panoramic cameras based on principles, which were behind the capabilities of computers available only a few years ago. The ability to create single 360-degree panoramic view from images acquired by multiple cameras at high frame rate is now also part of Control Web image processing virtual instruments.

More and more tasks involving cameras require continuous imaging of 360 degree space around the camera head. Such true panoramic image is useful for security systems in open landscape, in huge halls, supermarkets, parking lots etc. Panoramic cameras are also very useful in ski centers to show web visitors the snow conditions, on beaches etc.

This problem is usually solved by rotation camera, which creates panoramic view during its rotation. But this method works only for static scenes and provides very long image refresh periods. Sometimes is the panoramic view created by single column camera equipped with rotating mirror. Such cameras are quite complex, expensive and offer only very low sensitivity, because only single column of panoramic image is exposed at once. To perform exposure of the whole panorama then takes quite long time or each exposure of every column must be very short. Panoramic view can be also obtained by utilizing of convex mirror or fish-eye lens and subsequent image transformations. Still, image of single hemisphere may not be enough.

Remark:

Let us note that the possibility to create panoramic image from convex mirror image of camera with fish-eye lens is also available in Control Web image processing virtual instruments. What is more, these virtual instruments can create so-called “virtual lens” to move around the whole field of view purely in software, without any moving parts in the camera.

The interesting possibility is the usage of multiple cameras with the subsequent composition of panoramic image from individual frames. Such task may seem simple in principle, but there are number of tough problems, which can be effectively solved only by using of complex image processing.

The most important problems are:

individual cameras in camera head must be mechanically precisely aligned
different cameras often capture scene with very different light conditions
white balance can also be quite different on images captured by different cameras
images are distorted by wide-angle lenses and the transform function can vary among individual lenses
the amount of both useful data and also portions of images omitted from final image is quite high

We decided to design camera head with four cameras. All cameras are of digital DataCam line with high dynamic range and very low noise. What is more, the transparency of front optical windows of the camera head varies according to the light intensity, which enables capturing of both bright as well as dark scenes. The CCD detectors are oriented vertically. Vertical orientation ensures maximal utilization of CCD area and minimizes the area not used to create final panoramic image. Another plus of this architecture is the ability of the camera head to capture scene almost overhead and well below the head itself. Images are digitized in parallel and transferred into host PC through four USB 2.0 channels.

The most interesting part of the whole system is the GPGPU processing software. High-quality results can be obtained only it the camera head is precisely calibrated. It is almost impossible to achieve the required precision purely by mechanical alignment. This is why the software itself performs image transformations to eliminate residual mechanical differences. Geometrical transformations are only the last step in creating panoramic image, very important is also the quality of images read from individual cameras, debayerization of raw data and image filtering.

Robust military version of panoramic camera head
Semi-automatic calibration of camera head on test bed

Scene containing near and far objects as well as bright sunlit scenes together with dark shadows in the room
Scene with demanding white-balance conditions

Cameras “see” also in very low light conditions (two table-top lamps on the background)
Example of operator interface if remote monitoring system

Screenshot with images of four panoramic heads standing indoors in different heights at daylight (left)
and at nigh (right)

Image processing and camera control virtual instruments significantly enhance capabilities of the Control Web version 6 industrial application development system.

^rc@mii.cz