Abstract: AR (Augmented Reality, AR), is a kind of real-time computing the position of the camera image and Angle and combined with the corresponding image technology, is a kind of information the real world and virtual world information "seamless" integration of new technology, this technology's goal is the set of virtual world on the screen and interact in the real world. Because of its unique combination of virtual and real, real-time interaction and other advantages to play a great help to the medical field, has become a new means of auxiliary medical treatment. This paper introduces the characteristics and key technology of AR technology, and its current application in medicine, including AR on medicine use real-time visual guidance system in the femoral head necrosis in the pulp core decompression positioning test, and augmented reality auxiliary electromagnetic ercp bile duct intubation under the navigation positioning technology model research, preliminary success; In addition, the future AR technology is predicted and prospected.

Key words: AR; Medicine; Real-time visualization

I. BREAKTHROUGH of AR technology in medicine

1. A preliminary clinical trial of visual real-time guidance system based on AR technology for localization of femoral head necrotic pulp core decompression

1.1 Case background information and current solutions

Femoral head necrosis is common in young and middle-aged patients and appropriate measures should be taken to delay or avoid joint replacement as much as possible. In the treatment of young and middle-aged patients with femoral head necrosis, pulp core decompression can effectively relieve pain and delay the progression of femoral head collapse [1]. Pulp core decompression combined with bone grafting is an improvement of pure pulp core decompression. Studies have confirmed that it has a better therapeutic effect for patients with early femoral head necrosis [2], but the intraoperative location and puncture of the necrosis focus are still blind to a certain extent, facing the problems of repeated puncture, high radiation exposure, long time and instability [3].

AR technology is a way of interactive experience between virtual images and the real environment. It superimposes information that is difficult to be directly perceived by the senses in the real world onto the real-time video screen, realizing the integration of virtual and reality [4]. Many attempts have been made to combine AR technology with surgical procedures, such as AR navigation-based vertebral root screw implantation [5], AR technology-assisted fracture treatment [6] and bone tumor resection, etc. However, no relevant literature has been reported in respect of puncture guidance of pulp core decompression at home and abroad.

The traditional method of puncture location is repeated bedside fluoroscopy during puncture, which requires the surgeon's personal experience and hand-eye coordination ability. Statistics show that even for the computer-assisted puncture related operations based on X-ray fluoroscopy, the loss rate of the intraoperative guide wire placement is 2%-15%. Some researchers have tried to introduce computer tomography (CT) into this operation, but it is only in the in vitro experiment stage at present.

1.2 Technical application and test

The application of AR technology in orthopedic surgery is still in its infancy. AR technology has a broad application prospect in orthopedic surgery practice due to its advantages of precision, convenience and real-time visualization.

The surgical guidance system developed in this study plans the virtual surgical area during the preoperative registration process. The high-precision tracking of NDI optical tracking device and virtual Kirschner needle display can be ensured in this area, and the affected hip Angle can be adjusted as needed for secondary shooting and stacking. The average registration time of the preoperative surgical area is 15min. After the registration is completed, a new operation is performed in the same area without the need for re-registration. Simply move the patient to be operated in this area. This system uses the non-invasive body surface positioning device independently developed to complete the registration of the affected area, without causing any damage other than puncture. It has higher safety and is more in line with the principle of "minimally invasive" surgery.

In visualization of real-time guidance system based on AR technology, is expected to solve the femoral head positioning ring die pulp core decompression repeatedly China open and high radiation exposure, and wear a draw with high accuracy, in the realization of femoral head necrosis in the pulp core decompression positioning real-time visualization at the same time, guarantee the accuracy of the piercing, achieve real-time intraoperative visualization method is feasible.

2. Study on the technical model of bile duct intubation positioning by endoscopic retrograde cholangiopanography with augmented reality-assisted electromagnetic navigation

2.1 background

Endoscopic retrograde pancreatic angiography (endoscopic retrograde cholangiopancreatography, ERCP) performer often face selective intrahepatic bile ducts will be super difficult problem, and trial and error intubation significantly increased the occurrence of postoperative complications. In recent years, the use of augmented reality assisted electromagnetic tracking navigation in surgery has become an important auxiliary tool in minimally invasive surgery. At present, navigation technology is mostly used in neurosurgery and orthopedics. The main reason is that there are many bony structures in the surgical area that can be referred to in these two departments, and there is no respiratory movement interference, soft tissue deformation and other influences, so it is relatively easy to register the surgical route and achieve relatively accurate navigation accuracy. However, the AR surgical navigation system in the abdominal region is susceptible to movement and respiration. This makes the navigation registration process more difficult. Our model is a static model, while the real bile duct will move relatively due to its own respiration and pulse. Therefore, AR assisted electromagnetic navigation guides ERCP bile duct intubation for clinical application. The biggest difficulty is the registration accuracy. Objective Augmented reality assisted electromagnetic navigation to guide the positioning accuracy of ERCP bile duct intubation in 3D printed biliary tract model.

2.2 Exploration and experiment

The biliary tract of a patient with extrahepatic and extrahepatic bile duct dilatation was reconstructed by ABDOMINAL CT, and a 1∶1.36 resin biliary duct model was generated by 3D printing. Markers were embedded in the biliary duct model as the "target" of navigation accuracy measurement and fixed in the abdominal body model of human body. Registration markers and markers were affixed to the body model base and skin surface respectively. The standard 3D image (STL) of the body model was obtained by CT scanning. An electromagnetic sensor was installed inside the sphincterotomy channel to track its position in the model, and an augmented reality navigation platform was constructed using image overlay. Image coordinates of body model registration markers and markers were obtained by preoperative CT scan, and their magnetic field coordinates were obtained in the experiment to complete preoperative and intraoperative registration and augmented reality accuracy analysis.

2.3 the results

ERCP bile duct intubation was performed on the body model, and sphincterotomy was accurately guided into the left and right hepatic duct under AR assisted electromagnetic navigation. The navigation accuracy error was (0.949±0.033) mm, which could meet the requirements of clinical application.

Model experiments confirmed the feasibility of AR assisted electromagnetic navigation ERCP bile duct intubation, which laid a foundation for further clinical application.

In this study, the AR assisted electromagnetic navigation system was introduced into ERCP bile duct intubation, and accurate navigation effect could be achieved in the experiment based on biliary 3D printing model. Although some technical problems still need to be solved for clinical application. However, these technical problems will be solved through research, and this navigation technology will definitely have clinical application prospects.

Second, AR development trend and future outlook

1. Augmented reality (AR) stabilizes

In recent years, global industry giants have launched AR product research and development. For example, In 2012, Google took the lead in launching an AR glasses -- Google Project Glass, followed by Microsoft, Apple, Nintendo, Huawei, Tencent and other companies also released AR products.

From the perspective of application situation of AR technology, according to the keyword search of "AR" of Wisdom bud, the global application volume of AR technology grew steadily before 2016 and rapidly after 2016, and then declined in 2020. As applications for augmented reality (AR) continue to increase, the industry is slowly maturing. According to the 2020 Maturity curve of New technologies released by Gartner (as shown in the figure above), AR technology is officially removed from the list, indicating that AR technology tends to be stable and changes from a technology to be observed to a technology that can be used.

2. The scale of AR users will continue to increase

As augmented reality (AR) technology tends to mature, its industrial application speed will also accelerate, the augmented reality (AR) industry will usher in the explosion, and its user scale will continue to increase. According to Deloitte, by 2025, nearly 75% of the world's population and almost all people using social/communication apps will be frequent AR users, who will take more than 4.5 billion videos and photos per day.

3. The industry will grow rapidly

According to data from the China Academy of Information and Communication technology, the global shipments of augmented reality (AR) terminals will reach about 630,000 units in 2020 and 41.25 million units in 2024, with a cagR of 188%. The market size of global augmented reality (AR) industry is about 28 billion yuan in 2020, and will reach 240 billion yuan by 2024, with a compound annual growth rate of 66%.

Augmented reality in the development of the future will permeate almost every field of life, such as the designer can use AR Shared perspective, office workers can be on the table, on the wall, and even on the floor to do PPT, chef cooking can watch it again before cooking steps, doctors see virtual presentation before the surgery, the game is no longer confined to the screen into the real world. We believe AR technology can change the way people live.

reference

[1] Floerkemeier T,Luts A,Nackenhorst U,et al.You only look once; unified,real-time object detection[C]//Computer Vision&Pattern Recgnition.IEEE,2016.

[2] Chen D D et al.3D printed navigation template assisted pulp core decompression and bone grafting for the treatment of non-traumatic femoral head necrosis in ARCO2 stage [J]. Chinese journal of tissue engineering research, 2020,24 (27) : 4322-4327.

【3】 chengj. Chen k. Chen W.Comparison of morker-based AR and markerless AR:Acase studyon indpor decoration systemlML2017

[4] Elmi-Terander A,Burstrom GNachabe R. Et al. Pedic le Screw Placement

[5] Making Anigreality Surgicaln avigation with intraoperative 3D imaging Afirst in-human prospectivecohort StudylJl Spine (Phila Pa 1976), 2019, 44 (7); 517-525.

[6] MaLZhaoZZhangB. EtaThreedimensional augmented realit surgical navigation with hybrid aptical and electromagnetie Distal Intramedullary nail interlock Inglil.Int1 Med Robot,2018.14(4):1909.