Brain-computer Interface technology for virtual reality - Dr. Zhao Jitao


Abstract: BCI-VR is a new application model of BCI that combines BCI technology with virtual reality (VR). Bci-vr combines the advantages of the two complementary, at the same time promote innovation, showing broad application prospects. This paper introduces the difficulties and prospects of bCI-VR research in recent years from the basic structure of BCI system, THE control of BCI on VR and the influence of VR on BCI.

Key words: brain computer interface virtual reality environment imagination action steady-state visual evoked potential event-related potential

1 introduction

Brain-computer interface (BCI) is a new external information communication and control technology between human brain and computer or other electronic devices, which does not depend on the conventional brain information output pathways (peripheral nerve and muscle tissue). The first apparent use of BCI is to provide a new means of auxiliary motor function and external information exchange for people with normal thinking but motor disability. Therefore, since the birth of BCI, the mainstream of its research and development is mainly to control external devices and replace some missing functions of patients, or assist text expression. Especially in recent years, BCI has made great progress in helping disabled people control prosthetics, wheelchairs, even spelling, typing and online games [1]. Recently, in the field of multimedia and entertainment, a novel APPLICATION mode of BCI has emerged: BCI technology is combined with virtual reality (VIRtu-al Reality) to form a new form of virtual reality-based brain-computer interface (BCI-VR). The simplest way to combine BCI and VR is to design a BCI system that provides users with an immersive 3D virtual reality environment and on-site sensing feedback for real-time use.

2 Basic composition of BCI-VR system

A typical BCI-VR system is usually composed of brain-computer interface (BCI) and virtual reality (VR) hardware, and contains two independent software. One is BCI software, which is used to record physiological (mostly electrophysiological) signals related to brain thinking intention, and generate control commands for external devices after real-time processing, feature extraction and thinking state classification. The second is VR software, which is used to simulate and perform a virtual world, provide users with real-time situational feedback and timely process control commands from BCI. The two pieces of software must be able to communicate with each other in order to exchange information and commands in a timely manner.

In BCI-VR system, the main task of BCI as an input device is: to provide instructions, so that users can set the content of the instructions themselves and output these instructions simply, efficiently and without causing fatigue through brain thinking activities at any time; VR interface in this system mainly undertake sensory stimulation and feedback tasks and should do: show the sensory stimulation required by BCI and induce physiological signals required by BCI as soon as possible, and provide users with meaningful feedback tasks to control BCI; Finally, we need to design a VR display program for smooth connection with BCI.

BCI technology is a new technology to establish direct information exchange and control between human brain and computer or other electronic equipment. It can make part of the central nervous system information transmission block and thus lost the basic motor function of paralyzed patients have reconstruction or reconstruction of new nerve pathways, so as to restore its motor function. There are three kinds of BCI technologies based on electrophysiological signals: BCI technology based on spontaneous eeg, BCI technology based on evoked potential and BCI technology based on implanted potential.

Virtual reality (VR) uses computers and other devices to simulate a three-dimensional virtual world, providing users with sensory simulation experience of sight, hearing and body, so that users can observe and experience things in VR anytime and anywhere. The acquisition and analysis of eeg signals in VR environment can increase the user's sense of immersion and give real-time feedback to the user, so it can improve training efficiency, shorten training period and achieve excellent results in a short term. This has great advantages over ordinary BCI systems.

3 BCI's control of VR

BCI records brain signals, extracts physiological or mental state characteristic parameters, and conducts real-time analysis and processing for classification and recognition of thinking intention to generate control commands. VR software simulates, renders and builds virtual worlds, providing user feedback and processing commands from BCI.

At present, the EEG signals used by BCI-VR system to control VR include motor imagery (MI) potential and steady-state visual evoked potential. SSVEP) and event Related potential (ERP). Imagine that the action potential will generate event related desynchronization (ERD)/event related synchronization(ERS) potential signals; More P300 components with an incubation period of about 300ms were used in the event-related potentials.

3.1 Imagine action potential control VR

ERD/ERS potentials are generated in motor and sensory cortex when the brain is engaged in the imaginary action position. ERD/ERS are signals occurring in specific frequency bands of motor and sensory cortex. With the degree of correlation between the signal and the event, the amplitude of the signal will decrease or increase synchronously. ERD/ERS is associated with the motor process and the brain replaces the real action with imaginary action, that is, ERD/ERS phenomenon also exists under preaction conditions [2]. Body imagination actions can cause ERD phenomenon, and detection and extraction of corresponding ERD feature patterns can be used to distinguish the parts of imagined actions [3]. Studies on motor nerve rehabilitation show that BCI based on imagination can effectively promote the repair of motor nerve pathway of original injury of residual limb by imagining and controlling repeated movement of residual limb.

Online BCI system can have two control modes: synchronous prompt and free control. In the synchronous prompt mode, there is an external prompt, which requires the user to perform the corresponding task within a period of time after the external prompt appears. The BCI system collects the eeg signal within the corresponding time after the prompt stimulus for processing. Because the time of signal generation after prompt is fixed, the system is easy to operate in data processing. A BCI system is not the same thing as a user paced, or even the same thing as the user decides when to generate a control signal. This kind of BCI is more difficult than the former in system construction and signal processing, but it has practical value for users.

3.2 Steady-state visual evoked potential controls VR

Steady-state visual evoked potential (SSVEP) is an EEG signal frequently used by the BCI system. It comes from the response of the visual cortex to external scintillate stimuli (usually with a frequency greater than 6Hz), and can be obtained by extracting EEG signals from the occipital region.

Lalor et al. were the first to use SSVEP-BCI to control games in 3D virtual environments. The game has a monster walking along a rope from platform to platform. When the monster walks forward, it may lose its balance, and the user must control it through the BCI system to keep it balanced. To this end, both sides of the BCI screen display a square flashing at steady-state frequency to induce SSVEP signals of different frequencies. When the user tries to control the movement of the monster to the left or right to restore the balance of the monster, BCI will timely detect the SSVEP signal generated by the user's concentrated vision on the left or right squares and issue corresponding control instructions to restore the balance of the monster. The COMBINATION of BCI system based on SSVEP and virtual reality environment is suitable and effective, but its limitation is inseparable from flicker stimulation.

3.3 P300 potential control VR

In 2000, Bayliss and Ballard combined VIRTUAL reality and BCI technology for the first time, and designed the scene of users driving in virtual reality environment. The control signal of stopping at a red light is the P300 component of the event related potential (ERP) generated by the user during situational driving. P300 refers to the positive wave component of ERP with an incubation period of about 300ms, which is often the cognitive response of the brain to rare events. When a "target stimulus" is present in the BCI sequence (the target is usually less likely to occur in sequence events), the P300 signal can be recorded in the apical region of the scalp approximately 300ms after the target appears. When the driver needs to stop at a red light, the stress response of the user is enough to induce P300 signal. In a BCI system using P300, the user must focus on the target stimulus given a number of stimuli, with one output for each stimulus.

The BCI system can be trained and extracted from eeg signals in experiments, and used to control instructions together with user-specific information.

4. Influence of virtual reality environment on BCI

Compared with the traditional BCI interface without VR scenes, the interaction mode of BCI-VR system is more direct and intuitive, and can overcome some limitations of activities in the pure virtual reality environment. In particular, bCI-VR system mainly relies on visual stimulation. Users can implement control in VR environment by simple eye fixation or direct attention to the required objects (such as watching TV to open and watch TV, watching the door to open and close the door, etc.). In addition, imaginary motion signals can provide more intuitive, more flexible and richer VR control methods. For example, in the virtual reality environment, moving forward can be controlled by imagining the movement of feet, and handheld devices can also be used to increase the immersion of subjects when moving in VR. On the other hand, feedback is an important part of THE BCI system. Through feedback, subjects can know their efficiency in completing tasks and continue to learn and improve. Adding realistic VR environment feedback plot is an important factor that BCI-VR system can improve control efficiency and promote task completion.

In addition to BCI control performance, many other aspects of VR's impact on BCI can be found. For example, it has been found in the study of imagination movement that the user's heart rate will change correspondingly when performing the imagination movement, which can also be used for signal classification of BCI-VR system.

In conclusion, the use of virtual reality environment can increase users' enthusiasm and improve the performance of BCI.

5. Technical difficulties and prospects

As mentioned in the introduction above, bCI-VR is a new technology combining BCI and VR, which can take advantage of their respective strengths and complementary advantages, and open up new broad application prospects. At the same time, there are also many challenging technical difficulties and obstacles on the road of BCI and VR technology integration, structural transformation and application innovation [4].

First, BCI in BCI-VR system must assume the role of input equipment, the main task is to provide VR with stable and effective, convenient and intuitive, flexible control instructions. Because the roaming control in the virtual reality scene created by VR technology has the above characteristics. In this sense, VR is not an easy object to control and the quality of control is extremely high. This was the first major challenge in designing a BCI-VR system. To meet this challenge, the BCI of the new system is required to have the following capabilities: Not the same variety of app commands as a common VR input device, or the ability to post these commands anywhere, anytime, at any pace you want. Third, control commands can be freely selected according to the user's state of mind (that is, users and BCI can easily interact, and should be intuitive, convenient, efficient and not easily lead to user boredom or fatigue).

The second technical difficulty comes from the design and presentation of virtual reality scenes, and there are also many challenges. The main difficulties include: First, it can provide users with meaningful VR feedback so that they can manipulate BCI in a diversified and arbitrary way. Second, the virtual reality scenes presented should be able to closely and uninterrupted integrate and integrate BCI's various stimulation needs for inducing thinking eeg. And as far as possible to ensure that the virtual reality scene is authentic, so that users can maintain the depth of virtual reality immersion without interruption or destruction. Third, although the typical virtual reality scene and standard BCI training rules are far from each other, VR application scheme design must also be highly usable and versatile.

The final technical difficulties involve the software and hardware of the new BCI-VR system. There are also several auxiliary technical difficulties related to the software and hardware of the new system that need to be considered when using BCI in VR scenarios: First, BCI's EEG amplifier must be able to work normally in a harsh environment full of various noise interferences such as VR. Second, the recording and transmission of EEG data must also be carried out in an ideal and quiet manner, requiring that there should be no communication line conflicts and various environmental interferences in VR scenes. Thirdly, the combination of BCI system and VR system must be close enough to ensure the fast information exchange speed in the field experiment. Fourthly, in some virtual reality scenes, users may need to walk around, so active electrodes should be used to detect the eeg to better avoid the interference caused by human movement.

It is believed that with the improvement of the overall technical level of BCI, its auxiliary technology level related to hardware and software will also make great progress, and there will not be insurmountable obstacles.


[1] Li PENGhai. Research on new technology of BCI control of mobile peripherals based on computer vision. Tianjin: Tianjin University, 2011.

[2] WANG L J. Research on ERD/ERS signal analysis and recognition method based on BCI multi-action mode D. Nanchang: Nanchang University, 2010.

[3] Ming Dong, WANG Kun, HE Feng, et al. D. Chinese journal of scientific instrument, 2014,35 (9) : 1921-1931.

[4] KONG Liwen, XUE Zhaojun, Chen Long, et al. Journal of Electronic Measurement and Instrument, 2015, 000(003):317-327. (in Chinese)