1, research background
One way to enhance sports imagination is action observation, that is, observing the movement of body parts related to sports imagination tasks. Previous studies have shown that mirror neurons understand and learn actions through imitation, which leads to the activation of corresponding regions. Therefore, when a person observes that another entity reflects imaginary body movements, action observation plays a role in inducing mirror neuron stimulation.
There is a significant difference between the event-related desynchronization (ERD) modes of 2D and 3D motion, and the ERD of 3D visualization group is enhanced. Richer visualization and stronger ownership of observed motion can produce better ERD.
Recently, a research paper published in IEEE Transactions on Nervous System and Rehabilitation Engineering explored whether the rich immersion of virtual reality (VR) would affect the repetitive training of sports imagination by observing the handshake. In order to study whether different display media will affect the action observation in sports imagination, researchers use two different displays: immersive VR headphones and monitors to display the same graphic handshake. In addition, the research takes the graphic scene as the stimulus, and emphasizes the influence of illusion and concretization in immersive VR on action observation in sports imagination training. In order to investigate the brain activity when using these two different media, the researchers used EEG and identified the changes of neural signals induced by sensory motor cortex. In order to measure the distinguishability of spatial brain activity patterns in different motor imagery tasks, researchers applied machine learning techniques commonly used in brain-computer interface to learn and distinguish brain activities in different types of motor imagery.
2. Research process
The researchers conducted two experiments on each participant to investigate whether the use of immersive VR headphones to provide sports observation in sports imagination training has an impact on performance:
(1) Motion Imagination Based on Immersive VR (IVR-MI): An experiment of providing graphic handshaking scenes for motion imagination training with immersive VR headphones.
(2) Display-based Motion Imagination (MD-MI): an experiment in which a non-immersive display is used to display the same scene in motion imagination training.
Compared with MD-MI results, the influence of VR on sports imagination was analyzed.
2. 1 subject
* * * Twenty healthy participants aged between 20 and 37 participated in these two experiments. Before the experiment, all participants were also required to use VR headsets for a long time to ensure that they had no problems when using VR headsets. Participants were randomly divided into two groups with equal numbers: group A received MD-MI before IVR-MI, and group B received IVR-MI before MD-MI. In order to reduce the possibility that the previous experiment will affect the results of the next experiment, the latter experiment should be conducted at least 7 days after the previous experiment. The experimental results will not be disclosed to the subjects until the end of the two experiments to avoid any feedback that may affect their performance. The data collected by each participant was visually inspected, and the data of two participants were excluded because they showed extensive noise, and finally 18 participants were left for analysis.
2.2 scheme
This graphic scene consists of two virtual hands and arrows on a black background, which is realized by the Unity game engine. Before each experiment, adjust the position of the virtual hands so that the distance between the two virtual hands is roughly equal to the shoulder width of the subjects (Figure 1a).
(1)IVR-MI setting: Participants will wear Oculus Go after wearing the EEG cap with electrodes, and do not use vertical straps to prevent the straps on overlapping electrodes from tightening.
(2)MD-MI setting: placing a monitor with a monitor arm on the table in front of the subject can provide three degrees of freedom. Each participant can freely adjust the angle of the monitor arm.
Each participant can adjust the camera angle in the Unity application to maximize the virtual hand. Participants were asked to put their hands on the table so that their own hands would be replaced by virtual hands.
2.3 data acquisition
ActiChamp and actiCAP of BrainProducts are used to retrieve EEG data from each participant's scalp. The data were sampled at a sampling rate of 500Hz, and active electrodes were placed according to the international 10-20 system. During the whole experiment, 20 electrodes (FC5, C5, CP5, FC3, C3, CP3, FC 1, C 1, CP 1, Cz, CPZ, FC2, C2, CP2, FC4, C4, CP4) were recorded. BrainVision is used to record EEG signals, and the impedance of each electrode is controlled below 5k to obtain high-quality data. The data is band-pass filtered between 8-25 Hz. After collection, EEG data are re-referenced by applying the average reference to all electrode positions used. Pretreatment data were used to analyze neural activity.
2.4 experimental design
The experiment was conducted in a dark soundproof room to minimize any environmental interference. Each motion imagination experiment consists of six stages of 10 continuous motion imagination experiments. If necessary, participants can take a break between the two stages. Each experiment consists of a random sequence, which includes a continuous right-hand grasping task, a continuous left-hand grasping task and a rest task (Figure 2a).
A single task includes an initial 4-second instruction period followed by a 6-second motion imagination period followed by a 2-second rest period (Figure 2b). In the process of instruction, the participants are given a cross clue indicating the rest task or an arrow clue indicating the left or right hand to grasp the imaginary task, so as to tell the participants what the next task is and instruct them to stare at the corresponding hands. In the whole action imagination cycle after the instruction cycle, the virtual hand corresponding to the arrow clue simulates a series of grasping actions, and instructs the subjects to observe and imagine the same action in a moving way. Finally, during the break, the virtual hand remains motionless, and participants can move or blink to prevent eye fatigue. During the guidance period and the motor imagination period, the subjects were instructed to avoid any movements, including blinking. During the whole experiment, two virtual hands were displayed and participants were asked to imagine them as their own hands.
3. Research methods
3. 1 ERD analysis
The brain regions corresponding to electrode positions C3 and C4 are related to the grasping actions of the right hand and the left hand, respectively. In order to measure the changes of brain activity in a single cycle, we first calculate the average power spectrum of EEG data recorded by three motor imagination tasks with the following equation:
In order to analyze the changes of ERD amplitude with time caused by the subjects' left and right motion imagination in each conversation, we use the following formula to calculate the ERD ratio of two motion imagination tasks relative to rest tasks:
Therefore, according to the difference of brain pattern characteristics induced at each electrode position in different motor imagery tasks, the ERD ratio of each stage is calculated.
In order to analyze the performance of motor imagination in each experiment, the researchers further calculated the average ERD ratio of each experimental participant and applied the following formula:
Considering that everyone's most active frequency band may be different, the frequency band of each participant in the two equations is determined by choosing the frequency band with a bandwidth of 2Hz, which leads to the maximum average ERD ratio of all tasks in the two experiments.
The ERD results of C3 and C4 controlling motor imagination with their right and left hands were analyzed respectively, and the performance of subjects in two different tasks was discussed. In order to investigate the influence of different display media on each subject, this paper takes the designated group (indicating the experimental order) and display media as two factors, and makes a two-way variance analysis of the calculated average ERD value. In order to further test the statistical enhancement of ERD in each conversation, Dunnett nonparametric multiple contrast test was applied, in which the ERD ratio of the first conversation was used as a control. Therefore, in the two experiments, the ERD ratios of right-handed motor imagery task and left-handed motor imagery task were compared respectively (Figure 3).
3.2 Discriminant analysis
Through the discriminant analysis of neural activity in two experiments, a classic machine learning model is constructed to further evaluate the performance. In order to compare the classification accuracy of each participant in the two experiments, 6-second EEG data of each motor imagination cycle were extracted. In order to increase the amount of data to be learned by the model, this paper further enhances the EEG data every 6 seconds, and divides the data into 2-second time windows in steps of 100 milliseconds.
The common spatial pattern (CSP) algorithm is used to extract spatial features from the preprocessed EEG data, and Fisher linear discriminant analysis (LDA) is used to create a classification model, which predicts whether the EEG data segment involves rest, left-handed or right-handed motion imagination tasks. In order to evaluate the EEG data of motor imagery, we adopted two different cross-validation methods: 1)6-fold cross-validation, in which the data from a single experiment were analyzed, and each fold corresponds to the data retrieved from a single conversation of 10 motor imagery experiments; 2) 10 folding cross-validation, in which data from a single session is used, and each folding corresponds to data retrieved from a single experiment. Cross-validation is used to test the accuracy of distinguishing three different motor imagination tasks: left-handed grasping, right-handed grasping and static state. For statistical analysis, this paper tested the results of 60% cross-validation by two-way analysis of variance to show the overall performance of each experiment. In order to further test the statistical enhancement of neural activity discrimination, Dunnett nonparametric multiple contrast test was used for the cross-validation results of 10, in which the accuracy of the first conversation was used as a control.
4. Research results
4. 1 statistical analysis hypothesis verification
Before variance analysis and parameter test of cross-validation accuracy results of ERD results of left-handed and right-handed motion imagination, the necessary assumptions were verified. Table 1 shows the results of Shapiro-wilk normal test and Levene homogeneous variance test. The results of P value showed that the variance of all cases did not violate the normality and uniformity (p & gt0.05).
4.2 Experimental analysis of performance
In order to compare the performance of participants using two different display media, we analyzed the ERD ratio and the ERD amplitude, where the ERD ratio is represented by the average ERD ratio of participants during the exercise imagination period, and the ERD amplitude represents the average ERD collected from each time period over time.
The ERD ratio and ERD amplitude of left and right hand motion imagination in two experiments are compared, as shown in Figure 4. The result of variance analysis in figure 4a shows that the ERD of IVR-MI is greater than that of MD-MI(IVR-MI and MD-MI are 49.3212.08 and 34.7514.75, respectively), and the difference is very significant (F (1 675). Compared with MD-MI, the ERD value of IVR-MI's right hand movement imagination is also larger (53.29 12.57 and 41.3215.19, respectively), and the difference is very significant (F (1, 0/9). On the other hand, there is no significant difference between the two groups (F( 1, 16)=0. 13 1, p & gt0.72; F( 1， 16)= 1.034，p & gt0.32)。
Fig. 4b shows the ERD amplitudes of participants with respect to time, which is calculated by averaging the ERD amplitudes of each participant in all sessions. The red and blue waveforms of IVR-MI and MD-MI show that there is a significant difference in ERD between the left and right hands during the exercise imagination period, and the ERD amplitude of IVR-MI is greater than that of MD-MI. As shown by the gray scale on the X axis, the time domain range of left-handed motion imagination is between 1.05.4 seconds and 6.27.0 seconds, and that of right-handed motion imagination is between 1.45.8 seconds and 6.07.2 seconds. There is a significant difference between these two amplitudes. In the guidance period (left-handed motor imagination T
4.3 Experimental Cross-validation
Fig. 5 shows the results of 6 times the accuracy of object-related cross-validation for IVR-MI and MD-MI, where the haploid represents the data obtained from each session. Analysis of variance showed that the accuracy of the two media was significantly different (f (1, 16) = 20.990, P < 0.00 1), and the accuracy of IVR-MI was higher than that of MD-MI (67.85 13.50 and 55, respectively). On the contrary, there was no significant difference between the two groups (f (1, 16) = 0.008, p & gt0.93).
4.4 Conversational changes in performance
This study further analyzes how the ERD performance of left-handed and right-handed motor imagination changes with the change of training time. As shown in Figure 6, during the imaginary left-handed movement, the ERD rates of IVR-MI and MD-MI are both linearly positively correlated (IVR-MI r=0.345, P
Select the ERD ratio of the first session as the baseline and compare it with other sessions to analyze the improvement of ERD performance compared with each session, as shown in Figure 6 and Table 2. For the left-handed motor imagination, the subjects of IVR-MI and MD-MI improved significantly after the fifth time, but the improvement of IVR-MI and MD-MI was stronger (P < 0.0 1 and P
4.5 Conversational Change of Cross-validation
Fig. 7 shows the results of distinguishing brain activity patterns using 10 times cross-validation in each session, where haploids represent data from each experiment. For IVR-MI and MD-MI, the accuracy showed a positive linear relationship (r=0.276, P
In order to analyze the improvement of cross-validation accuracy in different time periods, we conducted Dunnett nonparametric multiple contrast test on the accuracy results of the first time period. The results in Figure 7 and Table 3 show that participants during IVR-MI can show significant improvement in discrimination from the 5th treatment (5th and 6th treatments P
4.6 Fisher ratio topographic map
In order to further study the spatial characteristics obtained from different hand imagination tasks, we use ERD results to apply Fisher ratio to each electrode. As shown in fig. 8, electrode positions C3 and C4 are the main factors to distinguish left-handed and right-handed motion imaginations. Compared with the Fisher ratio of MD-MI (C3 and C4 are 0.544 and 0.377 respectively), the Fisher ratios of C3 and C4 in IVR-MI group are higher (C3 and C4 are 0.997 and 0.566 respectively).
Step 5 discuss
In this study, VR headset and monitor are used as media to observe the movements of left and right hands, and to investigate the influence of immersion and illusion on sports imagination training. By comparing the ERD ratio and cross-validation accuracy obtained from the two experiments, this paper proves that perceiving the same action through different media in training may lead to different motor imagination performances.
The results show that participants can get better performance of sports imagination when using VR headphones. In the aspect of practicing motor imagination through repeated training, it is not only proved that repeated action observation will affect the performance of motor imagination of subjects, but also found that using VR headphones may improve the performance of motor imagination with less time cost. The results of ERD ratio and cross-validation accuracy using VR headphones show great improvement. This paper proves that using VR headset can effectively improve the performance of ERD and increase the spatial resolution of brain activity than using monitor.
The researchers also studied the ERD amplitude and Fisher ratio to solve the problem that only different display media will affect the ERD ratio of the central motor cortex (C3 and C4). The results of this study show that the amplitude pattern of ERD increases slightly during the guidance period, and there is no significant difference. Then there is a statistically significant difference between the two experiments, which greatly increases during the exercise imagination period and then decreases during the rest period (Figure 4b). Although the researcher predicted that the slight increase with no significant difference during the guidance period was the result of the preparation and planning of the guidance action, the ERD amplitude of IVR-MI was significantly higher than that of MD-MI only in the early stage of exercise imagination and rest, which indicated that this statistical difference was caused by the operation of exercise imagination. In addition, Figure 8 shows that in the two experiments, the main spatial features that distinguish different motor imagery tasks come from C3 and C4 electrodes, which indicates that only the difference of display media has little influence on the factors that may affect our results, such as the spatial features from visual cortex. These results show that the action observation through VR headphones is more effective than the action imagination operation displayed through the display.
As mentioned above, this paper focuses on whether the repetitive motion imagination training of motion observation is effective through the immersion and illusion of VR system. ERD performance and cross-validation results verify the hypothesis of this paper. The results show that the ERD ratio is higher and the spatial brain activity is more differentiated in repetitive motion imagination training. The results show that rich immersion itself will affect the motion imagination (by presenting the same graphic hand movements). Therefore, for any graphic scene that can be simulated, using immersive VR headphones may be beneficial to the training of sports imagination compared with non-immersive displays.
This study has some limitations and possible improvements. Some people may worry that the graphic scenes of this study may be considered different to some extent, because the proportion of virtual hands in the two display media may not be exactly the same. In order to solve this problem, before starting each experiment, pay attention to each participant's feedback, and then adjust the size to maximize the reality. In addition, although the researchers adjusted various environmental components to expand the concreteness in the study, they did not directly quantify the concreteness level of each user in the two experiments. Because there is a considerable time gap between the two experiments, the researchers think that any possible survey or questionnaire is potentially unreliable, but they use previous work results to claim that VR enhances concreteness. Finally, the relatively small sample size is also a limitation. Although each participant has repeated the experiment many times, considering the different performance of each participant, the statistical ability of analysis may be limited. Therefore, the research results of this paper should be carefully explained. According to the research results of this paper, the future research will focus on using the indicators of this paper to compare the use of VR headphones (a fully immersive visualization tool) and stereoscopic 3D glasses (a semi-immersive virtual reality system).
6. Conclusion
Different from the comparison between the visual scene itself and the motion imagination, this study focuses on the joint effect of immersive virtual reality and concretization on the motion imagination. Compared with other existing media, VR headsets can provide a more realistic experience and enhance illusion and immersion. Inspired by this, the researchers studied whether the immersive VR headset can also be used to enhance the performance of sports imagination by comparing the movement observation of the same virtual hand between VR headset and monitor.
In this paper, two different aspects of brain patterns related to motor imagination of these two media are studied: the change of signal oscillation rhythm from brain regions related to motor imagination and the distinguishability of signal spatial characteristics, and the machine learning model commonly used in brain-computer interface is used to explore. The results of these two analyses show that the use of VR headphones may lead to greater oscillation and spatial resolution of neural signals. Therefore, in clinical treatment, rehabilitation, brain-computer interface and other fields, VR headphones combined with immersion and illusion can better present the action observation in sports imagination training. In clinical treatment, rehabilitation, brain-computer interface and other fields, the use of VR headphones can better present the action observation in sports imagination training.