Remote driving of vehicles based on virtual Reality technology - Dr. Li She


Abstract: Unmanned driving and artificial intelligence are hot topics in the field of vehicle research. Considering the test cost, how to conduct road test of unmanned vehicles in virtual environment has become a new research direction. In driverless experiments, a driver is required to maintain control of the vehicle in the test vehicle, which undoubtedly increases the risk of injury. Remote driving, on the other hand, not only preserves the driver's control of the vehicle, but also reduces the risk of injury. At the same time, in the field of mobile robots, remote driving also has a good development prospect.

Key words: virtual reality; Remote driving; Unmanned vehicle

1 the introduction

With the continuous progress of sensors and machine learning, many domestic and foreign passenger vehicle manufacturers are aiming at the development direction of unmanned and intelligent vehicle driving [1]. The rapid rise of vehicle autonomous driving technology has promoted the rapid heating up of unmanned driving industry. However, if complete driverless driving is realized and put into actual operation, there are still many technical difficulties at present, and it cannot reach all-weather and unstructured road driving [2]. It will take a long time for driverless technology to develop from its initial exploration to its full development and popularization [3].

Driverless technology is mainly faced with the development limitations of many technologies such as road status identification, vehicle control and vehicle information interaction [4], and a series of social problems [5] such as ethical norms, legal responsibilities and regulatory policies have gradually emerged. Road condition recognition and environment perception based on artificial intelligence are one of the most important technologies of driverless driving. Vehicles need to accurately identify and make corresponding judgments on pedestrians, passing vehicles, traffic lights, etc. [6]. However, in recent years, Uber, Google, Tesla and other self-driving car fatal accidents [7] have raised greater doubts and challenges to the current road condition identification technology. Figure 1 shows the accident scene of a self-driving SUV of Uber.


Figure 1 Self-driving SUV crashes

For unmanned vehicles, information interaction between vehicles and modern transportation systems and between vehicles is also particularly important. In addition, OTA(over-the-air download), as The development trend of vehicle program upgrade of driverless vehicles in The future, The download speed, coverage and signal stability of The network also play a vital role in The driving safety and convenience of driverless vehicles [8]. From this point of view, the development of driverless vehicles will have a long way to go.

Remote control technology was first proposed in the early 20th century, and with the emergence of computer technology, wireless communication technology, sensor technology and other emerging technologies, it has been widely applied in various fields such as sea, land and air, affecting all aspects of human life [9]. By using remote control technology, machines or equipment can reach difficult environments that are difficult for humans to reach, effectively reducing work costs and avoiding casualties [10]. Reliable navigation, effective motion control and real-time acquisition of positioning sensor data are the keys to vehicle remote control technology [11]. The traditional remote control technology of the vehicle is to realize the control of the vehicle by analyzing the current environment of the vehicle by the operator based on the images returned by sensors and cameras installed on the vehicle, and then making decisions to adjust the attitude of the vehicle. Blind spots can be created because of the limited number of sensors and cameras installed on vehicles. In addition, the operator's sense of presence is not strong, the operation accuracy can not be guaranteed. Communication technology also limits the real-time performance of data transmission, making it impossible for the remote operator to adjust the control of vehicle motion attitude according to the change of environment. Because the remote vehicle can only rely on human behavior decision to move, but the operator cannot fully obtain the vehicle and environmental information, when the vehicle faces an emergency situation, due to the lack of certain autonomy, such as obstacle judgment, environment perception, etc., the remote control of the vehicle fails. With the vigorous development of machine learning, artificial intelligence and other emerging disciplines, the degree of intelligence and autonomy of vehicles is becoming more and more high, providing a new starting point for remote driving of vehicles. In addition, with the development of 5G network, the characteristics of super-large bandwidth, ultra-low latency and ultra-high reliability effectively guarantee the real-time transmission of data [12], and also provide reliable communication guarantee for remote driving of vehicles. On June 27, 2018, during the Mobile World Congress (MWC) in Shanghai, China Unicom, together with Ericsson and Yu Zi Technology, conducted the first 5G ultra-remote driving real car demonstration in China and achieved success [13], as shown in Figure 2.


FIG. 2 On-site demonstration of 5G ultra-remote intelligent driving

Virtual reality technology has strong Immersion, Interaction and Imagination, namely the so-called 3I characteristics [13], which has gradually penetrated into all fields of science and technology in recent years. At GTC 2018, Nvidia showed how users can control a car remotely by combining the Holodeck with VR Settings including the HTC Vive, driving racing seats and steering wheel, and a car with a remote controller. A major focus of its demonstration is the autonomous vehicle and AI autonomous driving platform DRIVE[14]. Due to the late research of driving technology in virtual reality environment, China still needs further development to gradually narrow the gap with developing countries in the new historical environment, so as to realize stronger and more immersive virtual simulation technology.

In August 2016, product Design company in silicon valley 219 Design team developed a can use HTC Vive virtual reality helmet control arm, can allow the user to accurate and intuitive real-time control arm, and his arm trajectory can be recorded after the robotic arms in response to a command can simulate user record arm movement, Thus, the manipulator is "taught" to realize the specified action [15]. In October 2017, MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) created a new virtual reality headset that allows operators to remotely control a robot using an Oculus Rift headset and a Touch handle, and even use a manual controller to directly control the robot's arm [16]. According to foreign media reports in December 2017, NASA and SONY are using PlayStation VR to develop a demo app called Mighty Morphenaut to practice how to control humanoid robots in space [17]. In January 2018, Brown University computer scientists developed software that allows users to immerse themselves in a robot's surroundings and control it from thousands of miles away. The software connects the robot's moment arm gripper, camera and sensor to VR hardware via the Internet. With the help of a hand-held controller, the operator can control the moment arm of the robot by simply moving his or her own arm, thus completing complex operation tasks [18].

China's relevant enterprises are also committed to the RESEARCH of VR remote control. At present, Baidu and SAIC group are in a relatively leading position in the research of virtual driving of unmanned vehicles in China.

Figure 3 shows MIT using VR technology to remotely control the robot.


Figure 3 MIT remote control robot with VR technology

At home and abroad, through the research of VR remote control robot, manipulator arm and a series of achievements, make VR remote control vehicles become possible. By combining the movement of the human and the remote machine, the operator can control the remote machine at the same time, the remote machine has certain autonomy, so as to realize the training and remote control of the machine.

Therefore, combined with the virtual reality technology, comprehensive development of modern communication, and improve the degree of autonomous vehicles, through the virtual environment to complete the comprehensive observation and control of unmanned vehicles, realize vehicle virtual remote driving, not only can effectively solve the unmanned vehicle is currently faced with environmental problems such as inadequate recognition, vehicle control inaccurate, but also can ensure the safety of vehicle, It can also make the operator have the experience of driving in its own environment.

2. Remote driving technology framework

With the continuous development of digital technology, virtual driving technology based on virtual reality is making continuous progress. Virtual driving technology has also been applied to the road test of unmanned vehicles and driver training from the study of car simulator. The virtual driving system is mainly composed of control system, visual simulation system, driving simulator [19], audio equipment, display and other components. Among them, the control system for physical objects, including the steering wheel, accelerator pedal, brake pedal and gearshift, drivers change the movement of vehicles in a virtual environment by control system, visual simulation system for vehicle attitude, and pass through vehicle information display drivers, the driver in time perception vehicle motion posture and traffic environment change; The driving simulator adjusts the orientation and Angle according to the changes of the vehicle's motion posture, so as to keep the driver's attitude consistent with the body posture, and enhance the experience and authenticity.

The vehicle remote control system mainly includes two parts: monitoring terminal and on-board terminal. The system structure is shown in Figure 4.


FIG. 4 Structure of vehicle remote control system

The main position and attitude information of the vehicle is collected through satellite positioning and sensors and transmitted to the operation interface of the monitoring personnel through the network. The operator analyzes and judges the information and issues relevant commands to realize the adjustment of the vehicle motion attitude [20]. Remote control system mainly transmits information through wireless network. The emergence of 5G network provides a good way to solve the problem of communication delay.

Virtual car is in the virtual environment through the vehicle dynamics model, road model, tire model and so on to achieve the vehicle driving, while real car is on the actual road surface for driving. Due to the influence of modeling accuracy, in order to ensure that the position and pose of virtual vehicle and real vehicle are as consistent as possible, based on the analysis of virtual driving system and vehicle remote control system, the structure diagram of vehicle remote driving technology based on virtual reality is established, as shown in Figure 5.


FIG. 5 Structure diagram of vehicle virtual remote driving technology

In FIG. 5, the information between real cars and virtual cars adopts 5G technology to realize workshop communication. The driver controls the motion state of the virtual vehicle by controlling the steering wheel, gas pedal, brake pedal and shift handle, etc. The virtual vehicle adjusts the control of the real vehicle through attitude prediction, so that the motion state of the real vehicle changes. If the state of the real vehicle is inconsistent with that of the virtual vehicle, the attitude correction module between the two vehicles will transfer the data to the attitude prediction module to realize the attitude adjustment of the real vehicle. Surrounding environment were collected at the same time, the real vehicle sensor, information of virtual vehicle real-time generate virtual environment, and transfer to the visual simulation system, through the HTC helmet display device information feedback to the driver, due to the change of motion and the vehicle through the driving seat body posture information feedback, To ensure that remote drivers have the feeling of actual car driving, further improve the safety of remote vehicle driving.

Operators wore a VR head show, show in virtual environment by observing the head motion state of the vehicle with the surrounding environment, in the vehicle for the corresponding operation instructions, such as the manipulation of the shift lever to change gear, the steering wheel to make the vehicle steering, etc., to make real the corresponding changes in the environment of the vehicle movement state, to realize the remote control of the real vehicle. At the same time, in order to eliminate the blind area of vision, radar and sensors are installed on the real vehicle to make the vehicle have certain autonomy. Compared with self-driving technology, vehicle virtual remote driving technology is not real men out of the vehicle control, remote issued a series of control actions, but by operators and through the virtual environment of actual vehicle monitoring control, can not only realize "unmanned" of the vehicle driving, effectively improve the remote vehicle safety performance, separate the drivers and vehicles, And can truly realize the vehicle unmanned.

3. Analysis of key technologies

1) Real-time environment generation technology

Virtual driving environment can provide a simulation environment similar to actual driving for automatic driving and virtual driving, and can also construct a typical and complex driving environment. Due to the particularity of its functions, the use of virtual reality technology to achieve real-time simulation of the driving environment and constitute a multidimensional, perceptive, measurable and realistic virtual driving environment is particularly important for the remote driving of vehicles [21]. In the virtual world, the physical model of the virtual world focuses on geometric representation and lacks realistic physical and behavioral models, which makes the virtual object and real object have a big difference. In addition, virtual reality technology is currently mainly studied in the visual aspect, and the research on hearing, touch and force feedback is still relatively backward, which may lead to some limitations in human-computer interaction, reducing the authenticity and real-time. The real driving environment is characterized by complexity and uncertainty, which makes it a great challenge to render virtual driving environment in real time.

2) Vehicle dynamics simulation technology based on artificial intelligence

Artificial intelligence is one of the branches of computer science, including many fields, including robotics, language recognition, image recognition and so on. With the deepening of artificial intelligence research, artificial intelligence is gradually applied to various fields. At present, the application of ARTIFICIAL intelligence in vehicles is mainly used for the automatic driving of vehicles to complete the recognition of visual, language and other information of vehicles [22]. However, the structure of vehicle dynamics system is complex, with more degrees of freedom and more connections between components, which increases the difficulty of accurately describing vehicle dynamics performance in virtual environment. Artificial intelligence based on deep learning is applied to the research of vehicle dynamics simulation model. The further development of vehicle dynamics simulation technology in virtual environment is of great significance.

3) Collision detection technology

The models in the virtual scene, including sky, ground, vehicles and other objects, are independent of each other. In the interactive control process, the viewpoint (virtual vehicle) can directly pass through each object. However, in real life, it is impossible for two impenetrable objects to have the same spatial region. Therefore, Collision Detection can be used to determine whether there is a Collision between different objects in the virtual scene at a certain moment [23]. However, the current collision detection technology does not meet the needs of detection between arbitrary objects, and is still limited to regular geometric convex bodies. Therefore, it is of great significance to study the collision detection technology that meets the requirements of virtual driving environment.

4) Wireless communication technology

In recent ten years, mobile communication technology has become the most widely used and rapidly developing communication technology. With the rapid development of mobile communication technology, new technologies and new services continue to emerge. In order to realize the remote control driving of the real world vehicle in the virtual environment, it is necessary to transmit the relevant data of the virtual world and the real car in real time and send instructions in time. Wireless communication technology acts as a bridge between the remote control system of the vehicle and the data communication of the monitoring center. Using wireless communication network to establish vehicle remote control system, which has the characteristics of always online, wide network coverage, low communication rate, fast data transmission and good real-time performance. The real-time and rapidity of communication technology are of great significance for adjusting the real-time attitude of actual vehicles and responding to emergency situations [24]. For remotely driven vehicles, real-time transmission of data and pictures is particularly important. Therefore, how to solve the problem of delay and hysteresis in wireless communication technology is a key technology. The development of 5G wireless communication technology [25] also provides greater possibilities for the realization of remote control driving of vehicles.

4 conclusion

At home and abroad are analyzed in this paper, virtual driving and unmanned vehicle the characteristics of the remote control, combining with the driving environment variety, vehicle dynamics model in simulation accuracy as well as the actual driving demand such as immersion, interaction, to improve traffic safety, this paper introduces the virtual reality environment remote driving of the vehicle frame, analyzes the related key technologies, The future development based on virtual remote driving is described.


[1] Chen Hong, Guo Lulu, Bian Ning. Technology Review, 2017(11):52-59.]

[2] Xia Xia, Wang Xinqing, Yu Yaowei, Meng Fanjie, Ma Zhaoye, Wang Dong, Shao Guangxin. Journal of Ordnance Equipment Engineering,2018(6):165-170.

[3] Mu Kangle. Electronic Technology and Software Engineering, 2017(21):112-114.

[4] Chen Xiaobo. [J]. Integrated Transportation, 2016(11):9-13.]

[5] Bai Hui-ren. Science and Society, 2018(1):72-87.

[6] Wang Shifeng, Dai Xiang, Xu Ning, Zhang Pengfei. Journal of Changchun University of Science and Technology (Natural Science Edition),2017(1):1-6.

[7] Zhao Guangming, Zhou Wenhui, Wang Yifan. Study on the Implications of Autonomous Vehicle Death Accident on Vehicle Safety Management in China and The United States [J]. Road Traffic Management, 2016(11):30-32.

[8] WANG Qinpu, ZHAO Jia, ZHAO Hao. Challenges and Suggestions for The Development of Driverless Vehicles [J]. Bus Technology and Research, 2016(6):2-6.

[9] Chen Yang. Research on wireless remote control technology and its system platform implementation [D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2006.

[10] HAINSWORTH D. Mine emergency survey vehicle[C].Proceedings of the 25th International Conference of Safety in Mines  Research Institutes. Pretoria, South Africa, 1993.

[11] TERRENCE F, CHARLES T. Vehicle Teleoperation Interfaces[J]. Autonomous Robots, 2001(11):9 -18.

[12] China Unicom and Ericsson, Yu Zhi Technology to complete 5G network ultra-remote intelligent driving real vehicle demonstration [J]. Telecom Technology, 2018(6):61.

[13] Su Jianming, Zhang Xuhong, Hu Qingxi. Computer Simulation, 2004(1): 18-21.

[14] dodson TB. NVIDIA Holodeck with VR remote control real car [EB/OL]. [2018-03-28]. HTML.

[15] Wang Rui. VR control or can be used for future operation of mechanical arm, bomb [EB/OL]. [2016-08-31].

[16] Li Liangqi. Massachusetts institute of technology of virtual reality technology to realize robot remote control [EB/OL]. [2017-10-17]. http:// WWW.

[17] jia hui. NASA is using Playstation VR manipulating space robot [EB/OL]. 2015-12-15.

[18] Remote control of robots by VR, which may be used to disarm bombs and the International Space Station [EB/OL]. Phoenix. [2017-12-15]. HTTP: / / http://

[19] Li Mingkun. Development and Application of Virtual Driving System Based on Unity3D [D]. Zhenjiang: Jiangsu University of Science and Technology,2017.

[20] Wang Xue. Design and Implementation of Vehicle Information Service System Client Supporting Internet of Vehicles connection [D]. Changchun: Jilin University,2014.

[21] Tao Dingfeng. Research on Virtual Battlefield Environment Generation Technology [D]. Nanjing: Nanjing University of Science and Technology, 2005.

[22] Hu Bohan. Technology Communication, 2017(5): 80-81.

[23] Wang Yingying. Design and Implementation of Remote Monitoring Center of Agricultural Vehicle Based on Virtual Reality [D]. Nanjing: Nanjing Agricultural University,2012.

[24] Mr. Yuan. Design and Development of Vehicle Remote Monitoring System based on Android Smart Phone [D]. Chongqing: Chongqing University,2014.

[25] Xiao Yanan, TENG Yinglei, Song Mei. 5G from the development of 4G communication Technology [J]. Internet World, 2014(10):66-69.