In virtual reality imaging technology, the best known is binocular imaging. For example, IMAX in the film industry launched the "Phantom 65 IMAX 3D Camera" and 4K dual-lens 3D camera in 2011. Dual-lens imaging, or binocular imaging, simulates different images received by the two eyes to form three-dimensional stereo vision to present a visual effect close to nature. However, binocular imaging has strict restrictions on viewers' freedom due to the fixed shooting Angle, and users are not allowed to change the Angle of view. To be able to change the perspective at any time, and provide 3D visual effects at all angles, is to rely on panoramic imaging.

Panoramic imaging is an important supporting technology of virtual reality. Panoramic imaging can be understood in a broad sense and in a narrow sense. In a broad sense, it's a variety of imaging methods that can capture images in 360 degrees, front, back, left, right and in all directions. In a narrow sense, it is an optical method that maps a 360° environment to an imaging method in a camera. This is a strange thing to say to someone who has only recently come into contact with panoramic imaging.

It's not hard to see why. The electronics industry and imaging technology have been developing rapidly in the last 20 years. Just a decade ago, imaging sensors were expensive and scarce. Unlike today, the most common way to achieve panoramic imaging is simply to use a set of six or eight cameras to collect images and then calibrate them together. In an age of expensive imaging sensors, engineers are trying to solve problems with as few cameras as possible through clever design.

Insta360 Image TITAN camera USES eight sensors to splice together to achieve panoramic image acquisition.

(image references from official website: https://www.insta360.com/cn/product/insta360-titan/)

NuBot at THE National University of Defense Technology didn't change much in structure from 2004 to 2014,

All use the panoramic image camera to collect the panoramic map to do the robot location.

(Image is quoted from: Xiao J., Xiong D., Yao W., Yu Q., Lu H., Zheng Z. (2017) Building Software System and Simulation Environment for RoboCup MSL Soccer Robots Based on ROS and Gazebo. In: Koubaa a. (eds) Robot Operating System (ROS). Studies in Computational Intelligence, vol 707, Springer, Cham. https://doi.org/10.1007/978-3-319-54927-9_18)

A typical soccer robot USES panoramic imaging to do real-time localization and map construction (SLAM),

There is also a forward imaging system for accurate object detection.

(Image is quoted from: Xiao J., Xiong D., Yao W., Yu Q., Lu H., Zheng Z. (2017) Building Software System and Simulation Environment for RoboCup MSL Soccer Robots Based on ROS and Gazebo. In: Koubaa a. (eds) Robot Operating System (ROS). Studies in Computational Intelligence, vol 707, Springer, Cham. https://doi.org/10.1007/978-3-319-54927-9_18)

In addition to the low cost, single camera panoramic imaging actually has the additional advantage of consistent imaging parameters. One of the most common problems encountered when combining multiple cameras for panoramic imaging is how to unify the white balance and shutter time of multiple cameras. It often happens that in the process of panoramic imaging, even if the geometric mosaics of multiple cameras are correct, significant differences will be formed near the mosaics lines due to obvious differences in gray scale and color, when there are relatively large differences in light intensity or color before different cameras. Even if the fuzzy processing is carried out by fusion method, it will be found that images from different directions will have great changes in gray scale and color, which will affect the quality of panoramic imaging. But with a single camera, this problem doesn't occur. Because a single camera's panoramic imaging shares the same set of imaging parameters, it is relatively simple to automatically adjust the imaging parameters by relying on the automatic adjustment function of the sensor itself.

Dual camera Mosaic can also achieve low-cost panoramic imaging,

However, image Mosaic consumes computing power, and the correction of image parameters is difficult to achieve perfectly.

Mi Home Panorama-camera quotes from the official Xiaomi website: https://www.mi.com/mama-camera.

Single camera panoramic imaging, the specific implementation method, mainly by reflection or refraction imaging optical path, to achieve 360° imaging. In practice, there are two approaches, the double mirror scheme and the SLR mirror scheme, as shown below.

The figure shows panoramic imaging using double reflectors: 1. Camera; 2. 2. Upper reflector; Lower reflector; 3. 4. Blind spots (the image of the camera itself); 5. Visual Field (all light blue areas)

(Image quoted from Wikipedia)

The figure is a schematic diagram of the object in the imaging plane in the panoramic imaging scheme of the SLR mirror.

(Image is quoted from: E. Menegatti, E. Pagello and M. Wright, "Using omnidirectional vision within the spatial semantic hierarchy," Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), Washington, DC, USA, 2002, Pp. 908-914 vol. 1, doi: 10.1109 / ROBOT. 2002.1013472.)

It can be observed from the above schematic diagram that all the images obtained by panoramic imaging are highly distorted. This is easy to understand. In order to obtain a 360° field of view, mapping a THREE-DIMENSIONAL space to a two-dimensional image sensor plane will inevitably cause large image distortion. To get an emotional understanding of the specific imaging effects, refer to the simulation diagram in the following figure.

On the left is a simulated indoor scene, and on the right is a panoramic imaging simulation of the indoor scene

(Image is quoted from: E. Menegatti, E. Pagello and M. Wright, "Using omnidirectional vision within the spatial semantic hierarchy," Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), Washington, DC, USA, 2002, Pp. 908-914 vol. 1, doi: 10.1109 / ROBOT. 2002.1013472.)

From the above simulation, it can be observed that the straight line around the camera becomes a circle, which is actually a mirror-image straight line, while the circumpolar (vertical and radial) straight line will be projected into a conic curve. In the following simplified model, given the mirror center O, the upper pole B (the mirror is a paraboloid, if the elliptic or hyperboloid B is lower than the upper pole), and the coordinates of the space point M(X,Y,Z) on the imaging plane M(X,Y) conform to the following formula.

Central symmetric refracted projection unified model

(Image is quoted from: Florence Jacquey, Frederic Comby, Olivier Strauss, Fuzzy Edge Detection for Omnidirectional images, Fuzzy Sets and Systems, Volume 159, Issue 15, 2008, Pages 1991-2010, ISSN 0165-0114, https://doi.org/10.1016/j.fss.2008.02.022.).

Single camera panoramic imaging can get normal images through image correction, but due to the limited resolution of the sensor, it is generally difficult to be used for specific industrial detection and object recognition, and is more used for environmental positioning. But in some other scenarios, also widely used, such as pipeline maintenance detection, panoramic angiography.

For example, in the field of new biometric recognition, the palm image can be obtained in the narrow space of the shape of the door handle for identity recognition, which not only achieves good recognition effect, but also is closely connected with the life application.

The upper left is the demonstration system of door handle recognition; The top right is the composition of the door handle recognition system;

Here is the image from the imaging.

(since the image application web page: http://www.quxiaofeng.me/doorknob/cn/)

Panoramic imaging, due to the relatively unintuitive imaging method, and slightly difficult image correction, has not been widely recognized. With the continuous improvement of the computing power of the embedded end, the spread of 5G communication, and the wide application of large-pixel panoramic images and videos, panoramic imaging is bound to get more attention and more applications.