The origins of augmented reality, or AR, can be traced back to Morton Heilig's Sensorama Stimulator, made in the 1950s and 1960s. He's a filmmaker and inventor, and he's used his years of film production experience to contribute to the machine called the Sensorama beauty. [1] The images, sound, aromas and vibrations of the Sensorama beauty are also crucial for making motorcycle rides in The streets of Brooklyn crucial for New York City. The invention was far ahead of its time. Taking this opportunity, AR also unfolds its development history. [2]

Augmented Reality (ar) technology is a kind of virtual information are fused together with the real world of technology, widely used in the multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing a variety of technical means, such as computer-generated text, images, 3 d simulation model, music, video and other virtual information, applied to the real world, two kinds of information complement each other, so as to realize "enhancement" of the real world. [3]

Overview of Augmented Reality

Augmented Reality (AR), which I believe everyone is familiar with, has only just become popular in recent years. If you talked to someone about AR before 2010, it would be considered a fantasy, because there is no suitable hardware platform and it is too far away from the mass consumer market, so it can only be regarded as a laboratory toy. Behind smartphones, is not the case, camera, large screen (compared with the function of machine), the existence of the two key components, as well as the rapid development of hardware computing power, AR technology is the foundation of the ground, so have mushroomed emerged a large number of AR application, especially the Pokemon Go release, imagination suddenly opened and the public a vision of AR technology is more and more strong.

Augmented reality, as the name suggests, is augmented reality. Broadly, "augmented reality" already covers every aspect of life, such as covering tree trunks with LED lights to create artistic effects. For example, some sound sources are placed in the haunted house to create a horrible atmosphere; Even wearing perfume is an enhancement of reality, enhancing people's sense of sight, hearing, smell and other senses. Of course, all of these are augmented reality with real objects, which are not so scientific and technological.

Often referred to as augmented reality, it refers to the use of virtual content for visual enhancement, displayed on a screen or projection device. For example, 3D animations (such as QQ Torch), Microsoft's HoloLens, Google's Tango, Magic Leap and so on are commonly displayed in the camera preview of mobile phones. [4]

In general, the key to AR technology lies in the device's perception of the surrounding environment: the most basic is to determine the spatial position of the device itself; A little more advanced, real-time reconstruction of the environment (SLAM); At a higher level, it involves recognition, cognition, and interaction. In this context, positioning is the most fundamental. Only when the position (and orientation) of the device itself is accurately determined, can the virtual content be well integrated with reality to achieve a sufficient sense of reality, and this process needs to be carried out in real time. This is also one of VR's key technologies.

At present, there are mainly two types of AR applied to mobile devices: AR based on picture marking and AR based on IMU.

QQ pass the torch is a typical image tag based AR, through phone specific image (another user mobile phones on display the torch of the picture), 3 d animation effects in the preview screen overlay, moving or rotating the phone, 3 d animation and always keep the torch of images taken by a fixed position, so as to achieve augmented reality experience. Pokemon Go is a typical model based on IMU AR, that is, positioning is carried out by the sensor data of the device itself, such as gyroscope and accelerometer. The rotation of the device can be measured by the gyroscope, while the displacement can be measured by the accelerometer. The REALIZATION of AR based on IMU is relatively simple, but the accuracy is low, and drift is serious (the accumulated error is increasing), so the applicable scenes are limited.

This series of articles mainly discuss the realization of AR technology based on plane picture Marker, which can be divided into two categories: Marker and Markerless. The difference between the two is mainly whether the picture Marker is regular:

• Marker: Usually a picture composed of black and white squares in a certain coding way;

• Markerless: A generic image.

Compared with ordinary pictures, Marker identification and tracking in specific codes are easier, but their applications are limited. Therefore, the mainstream AR applications are all based on Markerless, that is, to identify an ordinary plane picture and then overlay 3D content on it, such as QQ torch.

Classification of augmented reality technologies

From a technical point of view, AR display technology is mainly divided into three types: one is the Video - -based AR [5-6], AR game PokemanGo Based on handheld mobile devices, for example, because of its do not need to purchase additional equipment, became the most common AR display technology, the mobile device ACTS as a screen window or amplifier, is used to display overlay virtual information world. Second, opto-based AR[7], such as Meta and HoloLens for head-worn Optical imaging, can be divided into Optical perspective and video perspective according to their working principles. This AR technology can bring better experience and free hands, which is the focus of many AR hardware manufacturers at home and abroad. The third is Projection Based AR[8], such as MIT's SixSense project, which projects the generated virtual information directly into the real scene that needs to be integrated without wearing AR helmet or glasses. However, it is not widely accepted in the consumer market because it requires additional Projection equipment.

The principles of augmented reality

As mentioned before, the key of AR lies in the positioning of the device itself, and AR based on image marking is to determine the spatial position of the camera by taking a known image and according to the position of the image in the camera screen, that is, to determine the spatial position of the device. Below the main process includes three stages, shooting images, image processing, and virtual content updates, including image processing is the core, in the image processing is completed, the camera outside the cords (Extrinsic camera coordinate system and world coordinate system transformation relationship), and then applied to the preview image overlay (such as OpenGL environment [9] or a 3 d engine), update the position of the virtual contents, completes the whole process of a frame and then repeating the process, so that when the mobile equipment, virtual content always on display in the right place.

As the core, image processing also includes some sub-processes, such as camera internal parameter calibration, preview frame feature extraction, matching, camera external parameter calibration, etc. (The internal parameter calibration can be completed offline).

1. Calibration of camera internal parameters

The imaging process of camera can be regarded as the process of transforming spatial points to points on the image. If the distortion effect of camera is ignored, the whole transformation process is linear. The objective of camera calibration is to find the parameters of the transformation (including distortion), so that the imaging process of the camera can be accurately described by mathematical calculation.

Without considering lens distortion, the transformation model of camera imaging can be expressed as follows:

Where, A is the internal reference matrix of the camera, including focal lengths fx and FY in the direction of X and Y, and image center CX and cy. The spatial point M is first transformed to the device (camera) coordinate system (DCS) through the external camera reference [R|t], and then to the image coordinate system (ICS) through the internal camera reference A, imaging at (u, v). The process of determining the internal parameters of the camera is the calibration of internal parameters, which is related to the focal length and hardware technology of the camera and can usually be completed offline. The checkerboard calibration method is often used for internal reference calibration, which is easy to operate and has high precision. The calibration can be completed only by taking pictures of the same checkerboard from different angles.

2. Image feature extraction and matching

Image Key points, also known as "points of interest", can be considered as local position markers with specific features on an image, while characteristics descriptors are quantities that represent "features" of the feature points, which can be used to distinguish and match different feature points. A good feature point should have the following properties:

1. Repeatability: Under different conditions (such as rotation, scale and illumination) for the same object or scene, the more corresponding features in the two images, the better.

2. Uniqueness: The amplitude patterns of features need to be diversified so that such features can be distinguished and matched.

3. Quantity: Generally speaking, the number of detected features must be large, but in image retrieval, too many features will have a certain impact on the real-time performance of retrieval. Ideally, the number of features detected is within a relatively large range, which can then be adjusted by a simple threshold.

4. Accuracy: The obtained features should be able to be accurately located, including accurate positioning in image space and scale space.

5. Efficiency: The shorter the time for detection and description, the better, so that it can be used for subsequent real-time applications.

Among them, repeatability is also the key to rotation invariability and scale invariability. SIFT[10] and its improved algorithm SURF[11] are commonly used feature extraction algorithms, while FAST[12] is famous for its speed and is suitable for real-time scenes. About the local features of the image and the extraction algorithm, you can refer to this blog for details.

Feature matching means that after extracting the feature points of two graphs, matching is carried out according to the descriptor of the feature points, and matching point pair set is obtained, which can be used for subsequent singleton matrix calculation. Feature matching can be regarded as a classification process, such as simple matching based on Hamming distance, or matching based on K-nearest neighbor algorithm [13] to reduce false matching by ratio test. FLANN[14] is a commonly used feature matching open source library.

3. Calibration of external parameters of camera

Camera parameter calibration is to calculate the above camera imaging formula of | t [R], in the case of the known camera internal, according to the feature Point matching points between two images on, the camera can be obtained in spatial location relationship between the two images (rotating R and t), if an image is a sign of input images, should be set to the origin of coordinates in the world, and the process is a PnP in computer vision (Perspectire - n - Point), could be considered outside of the camera calibration process. This problem can be solved based on the Direct Linear Transformation [15] combined with the least square method, or it can be solved by iterative method according to an initial value.

The application of augmented reality

1. Education

AR applications are changing the way teaching is done in a more interactive way.

JigSpace is a learning tool developed based on Apple ARKit [16]. Students can use mobile devices to explore virtual 3D models of objects on their desks and learn about the inner workings of various objects. By projecting the interactive 3D model into AR, abstract concepts and objects can be separated step by step, so that students can have the most intuitive feeling.

FigmentAR is a silly VR content development platform launched by Viro Media. Even those who are not familiar with VR creation can make their own content through simple operation. Teachers can also create a virtual space with 360-degree graphics, which students can then enter to browse.

2. Health care

Health care is also one of the main fields of AR application, and AR has been applied in more and more medical cases, and has been successfully applied in education and training, patient analysis, surgical treatment and other aspects.

Back in 2015, surgeons at the Warsaw Heart Institute used Google Glass to assist with surgery and to keep track of patients' blocked coronary arteries in real time. Students at case Western Reserve University School of Medicine used HoloLens to dissect virtual tissue from a digital cadaver.

3. Enterprise training

The application of augmented reality in the field of technical training has attracted the attention of many enterprises. Honeywell International, for example, has introduced hybrid reality tools for employee training.

According to the company, "many studies of millennials indicate that they are looking for more experiential learning." The company is guiding employees through training with HoloLens and testing them in real-world factory environments.

4. Retail shopping

The shortcomings of head-mounted devices mean that smartphones will be the preferred carrier of AR technology in the next few years.

The APP from Dent Reality, an augmented Reality developer, allows consumers to view real-time information about products in retail stores and USES computer vision technology and in-store tracking to help them find what they need. For manufacturers of all kinds of goods, the shelves of brick-and-mortar retailers will become the new battleground for AR promotions.

5. Virtual fitting glasses

When shopping for clothes in the mall, fitting is a tedious process. A growing number of stores are already using AR to try on clothes in a virtual fitting room developed by US augmented reality developer Bevond, which allows shoppers to try on clothes without changing outfits. The technology could also solve one of the most vexing problems of online shopping: the gap between photos of online models and the way they look in real clothes.

6. Location-based advertising

Location-based advertising is not new, but location-based advertising combined with AR is a new thing. Facebook recently launched AR applications that let developers build location-triggered elements.

It can be predicted that AR user marketing in the real world is coming soon. But if every restaurant and store we walk by triggers a floating AD or sales offer, for many, it might become a nightmare.

7. Interior design

Interior design is a very complicated work, which needs to consider the size, material, color, collocation and other factors of space and various furniture. And AR allows ordinary people to easily design interior decoration and home layout.

IKEA Place is an AR application developed by IKEA based on Apple ARKit. Consumers can view the effect of IKEA products placed at home through mobile devices to make more satisfied choices, and it has the function of saving or adding it to their shopping cart. Like IKEA, Sotheby's International Realty has launched Curate, an app that allows would-be buyers to place their choice of furniture virtually in commercial Spaces or homes they want to buy or rent to see the best results.

conclusion

Compared with VR, AR will reach more people because it is a seamless complement to People's Daily life. AR is to superimpose the computer-generated virtual world on the real world. Various practical applications in medicine, education and industry have proved that AR, as a tool, has a more far-reaching impact on human beings. Instead of creating a completely virtual world outside the real world, as VR does. Foreign analysts also believe that "AR" will become "part of a more quotidian application of mobile devices". At the same time, the popularity of mobile AR and low cost can also help companies from using AR technology, companies can AR steady growth, the augmented reality will be in around 2021 manufacturing/resources, TMT, government (including military), retail, construction/real estate, health care, education, transportation, financial services, utilities have been applied.

reference

[1] Ying Ying, Cui Ayue, Discussion on the Present situation and Development of Augmented Reality, Science, Education, And Literary Exchange. 2019, (13), 80-81

[2] Chen Yingbo, Zhang Wenlan, Chen Sirui, Analysis of venue Learning Effects based on AUGMENTED Reality -- A Case study of "AR Box "virtual simulation learning Environment, Modern Distance Education Research. 2020, 32(5), 104-112

[3] Hu Tianyu, Zhang Quanfu, Shen Yongjie et al. Overview of Augmented Reality Technology, Computer Knowledge and Technology. 2017, 13(34), 194-196 Imaging Process.

[4] Glushakov, M., Zhang, Y. and Han, Y. et al. Edge-based Provisioning of Holographic Content for Contextual and Personalized Augmented Reality, 2020 IEEE International Conference on Pervasive Computing and Communications Workshops, PerCom Workshops 2020

[5]Zhu, J., Pan, Z. Occlusion registration in Video-based Augmented Reality, Proceedings of The 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, VRCAI 2008

[6]Choi, S.H., Kim, M., Lee, J.Y. Situation-dependent remote AR collaborations: Image-based collaboration using a 3D perspective map and live video-based collaboration with a synchronized VR mode, Computers in Industry, 2018(101):51-66

[7]Serrano Vergel, R., Morillo Tena, P., Casas Yrurzum, S., Cruz-Neira, C. A Comparative Evaluation of a Virtual Reality Table and a HoloLens-Based Augmented Reality System for Anatomy Training. IEEE Transactions on Human-Machine Systems 50(4),9106786, pp. 337-348

[8]Cascini, G.Email Author, O'Hare, J.Email Author, Dekoninck, E. et al Exploring the use of AR technology for co-creative product and packaging design, Computers in Industry 123, 2020, 103308

[9] Baek, N. An emulation scheme for OpenGL SC 2.0 over OpenGL, Journal of Supercomputing 76(10), 2020, 7951-7960

[10] Pai, S., Shettigar, R. Gender Recognition from Face Images Using SIFT Descriptors and Trainable Features, Advances in Intelligent Systems and Computing

1133, 1173-1186

[11]Sarangi, P.P., Mishra, B.S.P., Dehuri, S., Cho, S.-B. An evaluation of ear biometric system based on enhanced Jaya algorithm and SURF descriptors, Evolutionary Intelligence 13(3), 2020, 443-461

[12] Fan, Q., Gong, Z., Zhang, S., (...), Yin, Z., Ding, H. A vision-based fast base frame calibration method for coordinated mobile manipulators, Robotics and Computer-Integrated Manufacturing 68,102078

[13] Maleki, N., Zeinali, Y., Niaki, S.T.A. A k-NN method for lung cancer prognosis with the use of a genetic algorithm for feature selection, Expert Systems with Applications

164,113981

[14] Guo, S., Bai, Q., Yue, B., Li, X., Zhou, X. Transmission Line Galloping Detection Based on SURF Feature and FLANN Matching Algorithm, Communications in Computer and Information Science 1253 CCIS, 435-443

[15] El-Ashmawy, K.L.A. Using direct linear transformation (DLT) method for aerial photogrammetry applications, Geodesy and Cartography 44(3), 71-79

[16] Cervenak, R., Masek, P., ARKit as indoor positioning system, International Congress on Ultra Modern Telecommunications and Control Systems and Workshops 2019-October,8970761

[17] Tayeh, R., Bademosi, F., Issa, R.R.A. BIM-GIS Integration in HoloLens, Lecture Notes in Civil Engineering 98, 1187-1199