Published on in Vol 23, No 12 (2021): December

Preprints (earlier versions) of this paper are available at, first published .
The Effects of Virtual Reality Treatment on Prefrontal Cortex Activity in Patients With Social Anxiety Disorder: Participatory and Interactive Virtual Reality Treatment Study

The Effects of Virtual Reality Treatment on Prefrontal Cortex Activity in Patients With Social Anxiety Disorder: Participatory and Interactive Virtual Reality Treatment Study

The Effects of Virtual Reality Treatment on Prefrontal Cortex Activity in Patients With Social Anxiety Disorder: Participatory and Interactive Virtual Reality Treatment Study

Original Paper

1Department of Psychiatry, School of Medicine, Keimyung University, Daegu, Republic of Korea

2Department of Psychiatry, Keimyung University Dongsan Medical Center, Daegu, Republic of Korea

3OBELAB Inc, Seoul, Republic of Korea

4Department of Biomedical Engineering, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea

5School of Psychology, Korea University, Seoul, Republic of Korea

6Department of Psychiatry, College of Medicine, Chungnam National University, Daejeon, Republic of Korea

7Department of Psychiatry, Chungnam National University Sejong Hospital, Sejong, Republic of Korea

Corresponding Author:

Chul-Hyun Cho, MD, PhD

Department of Psychiatry

College of Medicine

Chungnam National University

82 Munhwa-ro


Daejeon, 35015

Republic of Korea

Phone: 82 044 995 4775


Background: Attempts to use virtual reality (VR) as a treatment for various psychiatric disorders have been made recently, and many researchers have identified the effects of VR in psychiatric disorders. Studies have reported that VR therapy is effective in social anxiety disorder (SAD). However, there is no prior study on the neural correlates of VR therapy in patients with SAD.

Objective: The aim of this study is to find the neural correlates of VR therapy by evaluating the treatment effectiveness of VR in patients with SAD using portable functional near-infrared spectroscopy (fNIRS).

Methods: Patients with SAD (n=28) were provided with 6 sessions of VR treatment that was developed for exposure to social situations with a recording system of each participant’s self-introduction in VR. After each VR treatment session, the first-person view (video 1) and third-person view (video 2) clips of the participant’s self-introduction were automatically generated. The functional activities of prefrontal regions were measured by fNIRS while watching videos 1 and 2 with a cognitive task, before and after whole VR treatment sessions, and after the first session of VR treatment. We compared the data of fNIRS between patients with SAD and healthy controls (HCs; n=27).

Results: We found that reduction in activities of the right frontopolar prefrontal cortex (FPPFC) in HCs was greater than in the SAD group at baseline (t=–2.01, P=.049). Comparing the frontal cortex activation before and after VR treatment sessions in the SAD group showed significant differences in activities of the FPPFC (right: t=–2.93, P<.001; left: t=–2.25, P=.03) and the orbitofrontal cortex (OFC) (right: t=–2.10, P=.045; left: t=–2.21, P=.04) while watching video 2.

Conclusions: Activities of the FPPFC and OFC were associated with symptom reduction after VR treatment for SAD. Our study findings might provide a clue to understanding the mechanisms underlying VR treatment for SAD.

Trial Registration: Clinical Research Information Service (CRIS) KCT0003854;

J Med Internet Res 2021;23(12):e31844



Social anxiety disorder (SAD) is a common psychiatric disease, with 8.4%-15% of the population worldwide diagnosed with it [1]. According to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), SAD is characterized by fear or anxiety in situations in which people receive negative attention from others [2]. Individuals with SAD avoid social situations and have difficulties in maintaining interpersonal relationships, with serious impairment in academic, occupational, and social functions [3]. Comorbidities, such as depressive disorders, anxiety disorders, substance use disorder, obsessive compulsive disorder, and avoidant personality disorder, typically accompany SAD. Moreover, patients are likely to be single and unemployed and develop suicidal ideas, low self-esteem and a low quality of life, and chronic illnesses without treatment [4,5].

Effective pharmacological agents for SAD treatment include selective serotonin reuptake inhibitors, serotonin and norepinephrine reuptake inhibitors, benzodiazepines, and beta-adrenergic antagonists. Furthermore, nonpharmacological therapies, such as cognitive behavior therapy (CBT) or social skill training, and a combination of pharmacological and nonpharmacological therapies are considered treatment modalities [6-8]. Although these treatments are effective, patients with SAD may be reluctant to receive psychiatric medication, have difficulty in visiting the treatment room, and drop out of treatment to avoid social situations or spatial constraints. Thus, patients may not actively participate in treatments [9,10].

To overcome the limitations of conventional therapeutics, digital intervention strategies, such as virtual reality (VR), that incorporate advanced technology into conventional therapies have been developed. Since the 1990s, VR has been used in various clinical treatments in medicine. However, initially, it could not reproduce reality well due to technical limitations, but with gradual developments in digital technologies, such as graphics and sounds, the treatment effects of realistic VR have improved considerably [11-13].

Since Rothbaum et al [14] reported the effects of progressive exposure therapy using VR in acrophobia, many studies have reported the effects of VR in various psychiatric disorders, such as specific phobia, posttraumatic stress disorder, SAD, attention deficit hyperactivity disorder, autism spectrum disorder, schizophrenia, depressive disorders, anxiety disorders, eating disorders, addiction, and mild cognitive impairment [13,15-19]. VR therapy has numerous advantages because it enables self-directed treatment at home or in locations where the patient feels comfortable. Furthermore, VR motivates patients toward treatment, mitigating the limitations of conventional therapist-led therapy [15,20,21]. VR can depict various social situations that cannot be reproduced in real treatment spaces because of spatial or human constraints and allows modification of patient treatment for specific situations [22].

Although VR treatment for SAD has attracted considerable attention, it is still a novel treatment modality. Pertaub et al [23] identified that social anxiety can be induced by virtual situations, which yielded results similar to conventional exposure therapy. Robilard et al [24] reported improved detection of anxiety symptoms with VR-CBT in patients with SAD. Randomized controlled trials have reported the effects of VR exposure therapy in patients with SAD [25-27]. However, limited clinical studies have been performed for investigating the effectiveness of VR treatment, and most studies have only identified effectiveness based on psychological assessments that relied on patients’ subjective expressions [28].

In this study, we designed an advanced therapeutic tool compared with existing VR treatments. During the sessions, the participants’ own voices were recorded, and scenes were constructed with the participants’ self-introduction in first person and third person to introduce participatory and interactive features that could be used therapeutically. The use of this VR program for measuring the psychological scale changes in patients with SAD has been reported earlier in a study [29]. In this study, we examined changes and responses in brain frontal region activities using functional near-infrared spectroscopy (fNIRS). fNIRS can be used to examine the functional activity in certain areas of the brain by measuring changes in the concentrations of oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) in brain tissue to assess neurological activation. fNIRS is safe, portable, and easy to use compared with neuroimaging tools such as functional magnetic resonance imaging (fMRI) [30]. Previous neuroimaging studies have reported changes in the medial prefrontal cortex (MPFC), ventrolateral prefrontal cortex (VLPFC), amygdala, anterior cingulate cortex (ACC), and posterior cingulate cortex (PCC) in SAD. The changes in the prefrontal cortex (PFC) were reported before and after VR therapy and during treatment [31].

The purpose of this study is to identify the neural correlates of symptom improvement after VR treatment in SAD based on the efficacy of the VR program reported previously [29]. We used VR-derived video clips with first- or third-person interviews to observe changes in brain activity. The neural correlates of symptom improvement after VR treatment in patients with SAD were studied by analyzing and assessing changes in neuronal activities using fNIRS compared with healthy controls (HCs).

Study Participants

In total, 40 patients with SAD and 34 HCs matched for age, sex, and handedness were enrolled in the study. After data quality verification with fNIRS, we included 28 patients (19 [68%] women) diagnosed with primary SAD and 27 HCs (13 [48%] women) in the study. The SAD and HC groups were recruited from universities through online advertisements on bulletin boards. The inclusion criteria for the SAD group were as follows:

  • Korean-speaking men or women aged between 19 and 31 years
  • Satisfy DSM-IV criteria for SAD according to the Mini-International Neuropsychiatric Interview (MINI; a psychiatrist conducted MINI to select the SAD and HC groups in this study) [32]
  • Psychotropic medication–naive patients without psychiatric comorbidities (excluding depressive and panic disorders)
  • Patients not undergoing psychotherapy
  • Patients without a history of neurological disorders
  • Patients without a history of psychotic symptoms upon VR exposure
  • Patients who experience epilepsy and are not vulnerable to visual stimuli

The exclusion criteria were as follows:

  • History of intellectual disability or organic brain damage
  • Individuals who experience psychotic symptoms upon VR experiences
  • People vulnerable to visual stimuli
  • People who are unsuitable for participation in fNIRS research (eg, people who cannot sustain the fNIRS test because of discomfort or anxiety in wearing fNIRS equipment or experiences of side effects, such as headache)

The HC group did not have preexisting neurological or psychiatric conditions. The Korean Social Avoidance and Distress (K-SAD) scale was prepared using an online questionnaire, and patients with a score of 82 or above were enrolled in the SAD group. All participants provided informed written consent after explanation of the study procedures and were required to pass an fNIRS scanning safety eligibility test. The study was approved by the Korea University Anam Hospital institutional review board (IRB no. 2018AN0377). The study was conducted in accordance with the Declaration of Helsinki ethical principles.

VR Therapy for SAD

An overview of the participatory VR treatment for SAD is available in previous reports [29,33]. Here, a similar participatory VR program was used for the treatment of patients with SAD. The VR program had an introduction, a core, and a finishing phase, with 3 levels depending on the difficulty of the core phase. In the introductory phase, participants learned how to use VR for relaxation and meditation. During the core phase, virtual situations were set up. In these virtual setups, several participants met to perform team tasks. Participants were exposed to social situations and introduced themselves to an audience. The level of difficulty was gradually increased, and participants were required to engage in discussions, and the activities were repeated. Each video clip was recorded in first- and third-person views [34]. The mirroring techniques in treatment were modified, and a hybrid treatment was used in the study. Namely, we attempted to maximize the effectiveness of treatment by consulting or discussing with participants by reconfirming VR treatment performed by themselves from the first- and third-person views. Each participant underwent 6 sessions of the treatment program. Participants were allowed to select the difficulty level of the program according to their comfort level after the first session. The researchers supervised each session to ensure the safety of the participants. The psychological scales were evaluated before the commencement of the VR program and after its completion. When the participants were watching the video clip in first and third person, we measured neuronal activities in the desired brain regions using fNIRS before VR treatment initialization and after the first and sixth sessions (Figure 1).

Figure 1. Overall schematic structure of the study. Patients with SAD and HCs were enrolled in a participatory and interactive VR treatment study. The HC group implemented fNIRS once by watching first- and third-person-view video clips at baseline. The SAD group implemented fNIRS 3 times in total at baseline, after the first and sixth sessions of VR treatment. fNIRS: functional near-infrared spectroscopy; HC: healthy control; SAD: social anxiety disorder; VR: virtual reality.
View this figure

Clinical Symptom Assessments

The clinical symptoms of the participants were assessed using psychological scales. Subjects with SAD were scanned in 2 testing sessions, before and after VR therapy. We reported an analysis of psychological scale changes to evaluate the effectiveness of VR therapy for patients with SAD [29]. In this study, we studied the correlation between fNIRS results and psychological scales. Diagnostic assessment was measured using MINI [31]. Additionally, for social anxiety, we considered the Korean version of the Beck Anxiety Inventory (BAI) [35], and State-Trait Anxiety Inventory-X (STAI-X) [36]. Moreover, the Internalized Shame Scale (ISS) [37], the Post-Event Rumination Scale (PERS) [38], the Korean version of the Social Phobia Scale (K-SPS) [39], and the Korean version of the Social Interaction Anxiety Scale (K-SIAS) [39] were evaluated. Next, we also incorporated the Brief-Fear of Negative Evaluation (BFNE) scale [40], the K-SAD scale [41], and the Liebowitz Social Anxiety Scale (LSAS) [42].

For the HC group, fNIRS was evaluated once, whereas for the SAD group, it was measured 3 times (at baseline, after the first VR session, and after the sixth VR session). In each measurement, a 2-back task was conducted between the first- and third-person versions. We used a procedure similar to Wagner’s task [43]. Working memory in the SAD group was studied previously using the N-back task, in which the SAD group exhibited lower accuracy and longer response times than the HC group [44]. Similarly, cognitive function was assessed using the N-back task, which measured working memory. To provide human-recorded first- and third-person views, fNIRS was performed before and after the N-back operation to predict the expected therapeutic effect of VR-based psychotherapy. The 2-back task required the participants to determine whether the current stimulus was similar to the stimulus presented earlier in the trials. The task paradigm was composed of a block design that was repeated 3 times with alternate rest and task periods of 30 seconds.

fNIRS Data Analysis

The fNIRS system (NIRSIT, OBELAB Inc., Seoul, Republic of Korea), which monitors activation in the PFC, is a portable, wireless, wearable, and multichannel (48 channels) brain imaging system that covers the entire forehead. Wavelengths of 780 and 850 nm were used, and 48 regions of light source and detectors separated by 3 cm were considered. The weight of the device is 550 g, and the sampling rate is 8.138 Hz. To remove high-frequency noise due to environmental artifacts and low-frequency noise due to the blood circulation in the body, the fNIRS signal was filtered using a bandpass filter with a cutoff frequency of 0.005-0.1 Hz. Considering the signal-to-noise ratio, the signal quality of each channel was evaluated after filtering, and values less than 30 dB were eliminated as unreliable.

Using the wavelength-dependent hemoglobin absorption coefficient and differential path length coefficient, the modified Beer-Lambert law was applied to the filtered signal, and changes in oxyhemoglobin (ΔHbO2) [45] and deoxyhemoglobin (ΔHbR) concentrations were extracted [46]. To account for the time-dependent change in hemodynamics, baseline correction was performed by subtracting the average from the calculated change in the oxygen concentration. Next, based on the Montreal Institute of Neurology standardization space (ICBM152), all 48 channels were grouped into 8 regions. After grouping each region of the PFC, the rejected channels identified during the channel rejection process were filled with the average ΔHbO2 of the channels allowed within each region.

Statistical Analysis

The mean and standard deviation of demographic data, such as age and educational year, were calculated for both groups. After evaluating the variance equality through the Levene test, the difference in ΔHbO2 between the HC and SAD groups was analyzed by conducting an independent t test. One-way repeated ANOVA and post hoc analysis were conducted to understand subject variation in the SAD group from baseline. After the first and sixth VR sessions, the Box test for equivalence check and the Mauchly test of sphericity were conducted to analyze fNIRS data. Data processing was performed using the NIRSIT package in MATLAB (version 2019b; MathWorks Inc., Natick, MA, USA). Finally, we used the Pearson correlation coefficient to identify the relationship between psychological scale scores and area-specific changes in brain activation. Additional statistical analyses were performed using IBM SPSS Statistics 21 (SPSS Inc., Chicago, IL, USA), where the criterion for statistical significance was set at P<.05.

Demographic Data

In this study, we analyzed the data from 28 patients with SAD and 27 people in the HC group. The mean age (SD) for patients with SAD were 23.74 (3.55) years, and for the HC group, they were 23.18 (3.27) years. The mean (SD) educational years for the SAD and HC groups were 15.00 (1.88) and 14.63 (1.37), respectively. No difference was observed between the groups in terms of age and years of education.

The mean (SD) K-SAD scores were 106.68 (15.33) for the SAD group and 48.93 (17.10) for the HC group. Significant differences were observed in the severity of SAD symptoms (Table 1).

Table 1. Demographic and clinical data between SADa patients and HCsb.
Demographic and clinical dataPatients with SADHCs
Gender, n (%) male/n (%) female28 (9 [32%]/19 [68%])27 (14 [52%]/13 [48%])
Age in years, mean (SD)23.74 (3.55)23.18 (3.27)
Education years, mean (SD)15.00 (1.88)14.63 (1.37)
K-SADc scored, mean (SD)106.68 (15.33)48.93 (17.10)
STAI-Se scored, mean (SD)49.21 (10.17)35.48 (7.81)
STAI-Tf scored, mean (SD)53.21 (10.30)34.19 (7.19)
ISSg scored, mean (SD)51.86 (16.70)15.22 (9.74)
BFNEh scored, mean (SD)44.39 (8.91)27.74 (7.35)
PERSi scored, mean (SD)46.96 (8.34)32.81 (11.61)
LSASanxj scored, mean (SD)38.86 (12.66)13.96 (10.56)

aSAD: social anxiety disorder.

bHC: healthy control.

cK-SAD: Korean Social Avoidance and Distress.


eSTAI-S: State-Trait Anxiety Inventory-State.

fSTAI-T: State-Trait Anxiety Inventory-Trait.

gISS: Internalized Shame Scale.

hBFNE: Brief-Fear of Negative Evaluation.

iPERS: Post-Event Rumination Scale.

jLSASanx: Liebowitz Social Anxiety Scale-anxiety.

Psychological Scale Scores of SAD and HC Groups

In this study, the scores of the SAD group were considerably higher than those of the HC group for all the tested scales that assessed associated anxiety symptoms (Table 1). In the SAD group, after VR treatment, the scores on the psychological scales for the SAD group decreased in all cases except 2, namely STAI-S (subscale of STAI, P=.11) and the Liebowitz Social Anxiety Scale-anxiety (LSASanx, subscale of LSAS, P=.16). (In a previous paper, we detailed the psychological scales used in the study and their purposes of assessment. Furthermore, we previously reported the changes in psychological scales according to VR treatment [29]).

Baseline PFC Activity in SAD and HC Group Measure Using fNIRS

We conducted fNIRS measurements for study participants before VR treatment to analyze brain activity in each region of the PFC, such as the dorsolateral prefrontal cortex (DLPFC), the VLPFC, the frontopolar prefrontal cortex (FPPFC), and the orbitofrontal cortex (OFC); see Figure 2. Brain activities among the SAD and HC groups were compared. A reduction in activation was measured by viewing first-person video clips (video 1), especially in the right FPPFC (P<.05). However, a negligible difference was observed between the two groups when the 2-back task was executed, and video clips in the third person were used for the treatment program (video 2). Next, while executing the 2-back task, activations in the left-brain areas of the HC group were lower than those of the SAD group, but the difference was not significant (Table 2).

Figure 2. Activity of the PFC in the HC and SAD groups measured by fNIRS at baseline. When fNIRS was performed while both groups watched video 1, the activity of the right FPPFC was more decreased in the SAD group than in the HC group (P<.05). However, when implementing the 2-back task and watching video 2, there was no significant difference in the activity of the PFC between both groups. fNIRS: functional near-infrared spectroscopy; FPPFC: frontopolar prefrontal cortex; HC: healthy control; PFC: prefrontal cortex; SAD: social anxiety disorder.
View this figure
Table 2. Comparison of the activation of the PFCa measured by fNIRSb between SADc patients and HCsd at baseline.
PFC regiontdfP value
Video 1

DLPFCe (Rf)–1.835853.07

VLPFCg (R)–0.996553.32

FPPFCh (R)–2.014153.049i

OFCj (R)–0.420653.68

DLPFC (Lk)–1.517253.14

VLPFC (L)–0.998953.32

FPPFC (L)–1.557753.13

OFC (L)–1.679653.10
2-back task

DLPFC (R)–0.172453.86

VLPFC (R)0.343053.73

FPPFC (R)–0.570953.57

OFC (R)0.067753.95

DLPFC (L)–1.908553.06

VLPFC (L)–1.769253.08

FPPFC (L)–1.876253.07

OFC (L)–1.824453.07
Video 2

DLPFC (R)0.241053.81

VLPFC (R)0.160053.87

FPPFC (R)0.679153.50

OFC (R)–0.305053.76

DLPFC (L)–0.100353.92

VLPFC (L)–0.293153.20

FPPFC (L)0.506353.61

OFC (L)–1.824453.84

aPFC: prefrontal cortex.

bfNIRS: functional near-infrared spectroscopy.

cSAD: social anxiety disorder.

dHC: healthy control.

eDLPFC: dorsolateral prefrontal cortex.

fR: right side.

gVLPFC: ventrolateral prefrontal cortex.

hFPPFC: frontopolar prefrontal cortex.

iP values in italics are significant (P<.05).

jOFC: orbitofrontal cortex.

kL: left side.

Changes in PFC Activity in SAD After VR Treatment Using fNIRS

We compared the brain activities measured by fNIRS (1) at baseline with those obtained after the first and sixth sessions of treatment in patients with SAD and (2) those obtained after the first session with those obtained after the sixth session. We observed significant differences in the right FPPFC (P=.01) and OFC (P=.045) and the left FPPFC (P=.03) and OFC (P=.04) when the treatment plan included video 2 for patients with SAD (Table 3 and Figure 3).

Table 3. Comparison of the activation of the PFCa measured through fNIRSb before and after VRc treatment in SADd patients.
PFC regiondfBaseline vs session 1Baseline vs session 6
tP valuetP value
Video 1

DLPFCe (Rf)270.2855.78–0.3951.70

VLPFCg (R)27–0.0279.98–0.6768.50

FPPFCh (R)27–0.4071.69–1.2572.22

OFCi (R)27–1.9413.06–1.5350.14

DLPFC (Lj)27–0.7767.44–0.7083.48

VLPFC (L)27–1.8585.07–1.5250.14

FPPFC (L)27–0.8758.39–0.9972.33

OFC (L)27–0.1165.91–0.3891.70
2-back task

DLPFC (R)271.1630.250.5698.57

VLPFC (R)271.7643.091.0058.32

FPPFC (R)270.9091.371.1983.24

OFC (R)271.7939.082.4581.02k

DLPFC (L)270.2466.810.3295.74

VLPFC (L)270.9830.330.4685.64

FPPFC (L)270.3272.750.7516.46

OFC (L)271.7245.102.0201.05
Video 2

DLPFC (R)27–1.7973.08–1.7163.10

VLPFC (R)27–1.2856.21–1.3256.20

FPPFC (R)27–3.1821.004l–2.9260.007l

OFC (R)27–2.8014.009l–2.0984.045k

DLPFC (L)27–1.8807.07–1.8464.08

VLPFC (L)27–0.9249.36–1.4879.15

FPPFC (L)27–2.3205.028k–2.2514.03k

OFC (L)27–2.6050.015k–2.2136.035k

aPFC: prefrontal cortex.

bfNIRS: functional near-infrared spectroscopy.

cVR: virtual reality.

dSAD: social anxiety disorder.

eDLPFC: dorsolateral prefrontal cortex.

fR: right side.

gVLPFC: ventrolateral prefrontal cortex.

hFPPFC: frontopolar prefrontal cortex.

iOFC: orbitofrontal cortex.

jL: left side.

kP values in italics are significant (P<.05).

lP values in italics are significant (P<.01).

Figure 3. Activity of the PFC of patients with SAD at baseline, after the first VR session and after the sixth VR session. Comparing between baseline and after the first VR session, when watching video 2, there were significant differences in activity in the right FPPFC (P<0.01) and OFC (P<0.01) and in the left FPPFC (P<0.05) and OFC (P<0.05). Comparing between baseline and after the sixth VR session, when watching video 2, there were significant differences in activity in the right FPPFC (P<0.01) and OFC (P<0.05) and in the left FPPFC (P<0.05) and OFC (P<0.05). When patients with SAD were performing the 2-back task, there was a significant difference in activity in the right OFC (P<0.05) between baseline and after the sixth VR session. There was no significant difference in PFC activity when watching video 1. FPPFC: frontopolar prefrontal cortex; OFC: orbitofrontal cortex; PFC: prefrontal cortex; SAD: social anxiety disorder; VR: virtual reality.
View this figure

Correlations Between Brain Activities and Psychological Scale Results

We analyzed the correlations between psychological scales and brain activities through fNIRS before and after VR treatment in patients with SAD. The STAI-S scores showed significant correlations with the activities of the right and left FPPFC, the right and left OFC, and the left DLPFC. Furthermore, the STAI-T scores were significantly correlated with the right and left OFC, and the ISS was significantly correlated with the right and left FPPFC and the right and left OFC in video 1. We also determined that STAI-S scores were significantly correlated with the right and left VLPFC, the right FPPFC, and the left DLPFC; the ISS was significantly correlated with the right and left DLPFC, the left VLPFC, the right and left FPPFC, and the right and left OFC; and the BFNE was significantly correlated with the right VLPFC in video 2. IN addition, PERS was significantly correlated with the right OFC, the K-SAD scale was significantly correlated with the left OFC, and LSASanx was significantly correlated with the right DLPFC, VLPFC, FPPFC and the right and left OFC while executing the 2-back task (Table 4).

Table 4. Correlations between brain activities and results of psychological scales.
Psychological scaleDLPFCa (Rb)VLPFCc
Video 1

2-back task

Video 2


aDLPFC: dorsolateral prefrontal cortex.

bR: right side.

cVLPFC: ventrolateral prefrontal cortex.

dFPPFC: frontopolar prefrontal cortex.

eOFC: orbitofrontal cortex.

fL: left side.

gBAI: Beck Anxiety Inventory.

hSTAI-S: State-Trait Anxiety Inventory-State.

iP values in italics are significant (P<.01).

jP values in italics are significant (P<.05).

kSTAI-T: State-Trait Anxiety Inventory-Trait.

lISS: Internalized Shame Scale.

mPERS: Post-Event Rumination Scale.

nSPS: Social Phobia Scale.

oSIAS: Social Interaction Anxiety Scale.

pBFNE: Brief-Fear of Negative Evaluation.

qK-SAD: Korean Social Avoidance and Distress.

rLSASanx: Liebowitz Social Anxiety Scale-anxiety.

sLSASavo: Liebowitz Social Anxiety Scale-avoidance.

Principal Findings

To the best of our knowledge, this study is the first to determine the effects of VR therapy on SAD by using fNIRS measurements. From this study, we found that changes in activity in specific brain regions, such as the FPPFC and OFC, are closely related to VR treatment. In addition, we developed a VR program to enhance the efficacy of the treatment by creating first- and third-person views of exposure situations. In the first-person view, the patients directly present content to an audience, and in the third-person view, the patients visualize themselves speaking to an audience objectively.

The effects of VR treatment on patients with SAD were studied by measuring the changes in PFC brain activity using fNIRS. We were able to find that the activation of the right FPPFC was higher in the SAD group than in the HC group when watching video 1. This can be understood in a similar context as the blood flow of the MPFC decreases in situations in which anxiety is predicted in healthy individuals [47,48]. A positron emission tomography (PET) study reported that the blood flow in the right DLPFC, left inferior temporal cortex, and left amygdaloid-hippocampal region increases in patients with SAD [48]. However, using fNIRS analysis, we did not observe any difference in activation between the right DLPFC among the study groups. Previously, an fNIRS scan revealed that the activation change in the VLPFC in patients with SAD was less than that in the HC group during a verbal fluency task [49]. Here, the activation change in most regions of the left PFC was low and not significant.

In this study, we observed changes in the prefrontal area before and after VR treatment. Thus, activation in the right FPPFC and OFC and the left FPPFC and OFC significantly increased when patients with SAD were watching the video in the third-person view. By using PET after pharmacological treatments, Evans et al [50] reported that the regional brain metabolism of the ventromedial prefrontal cortex (VMPFC) in patients with SAD increases. Further, Hiser et al [51] reported that the activation of the anterior VMPFC increases upon pharmacological treatment or psychotherapy. We obtained similar results to these previous studies. However, the metabolism in the medial dorsal PFC decreased in patients with SAD after pharmacological treatment [52].

We obtained positive self-referential stimuli (positive words), neutral self-referential stimuli, and negative self-referential stimuli (negative words) after VR treatment to identify the activation changes in the brain region measured through fMRI in a previous study [33]. Therefore, patients with SAD exhibited increased activation in the right PCC/precuneus, lingual gyrus, left inferior temporal gyrus, precentral gyrus, and postcentral gyrus for positive self-referential stimuli. Patients with SAD also revealed increased activation in the left middle occipital gyrus, parahippocampus, left Rolandic operculum, and left caudate nucleus for negative self-referential stimuli. In this study, we observed the brain activity of participants watching the video clips, unlike the previous study, and determined the neural correlates by fNIRS, which were used to measure the activation of the prefrontal area, unlike fMRI, which can be used to measure the activation of the whole brain area.

Lastly, we identified correlations between psychological states in patients with SAD with brain areas activated upon VR therapy. The STAI-T scores and OFC activation, and ISS scores and prefrontal region activation (except the right VLPFC) were correlated. Studies have reported the correlation between the severity of SAD symptoms and the activation of certain brain areas. Klumpp et al [53] reported that improvements in the symptoms are correlated with the increase in the activation of brain areas, such as the medial orbitofrontal and dorsomedial frontal gyrus. Marin et al [54] reported that hypoactivation of the VMPFC is correlated with the severity of anxiety. Based on fNIRS analysis, Yokoyama et al [49] reported that changes in VLPFC activation are negatively correlated with social fear and activation of the left DLPFC is positively correlated with social anxiety [55].

Studies have reported that amygdala activation is high in patients with SAD [48,56]. Here, we could not study amygdala activation, because measuring subcortical brain regions with fNIRS is difficult. The VMPFC and the VLPFC are known to be associated with the amygdala in patients with SAD. Generally, the VMPFC regulates social functions and inhibits the amygdala to regulate the fear response [51]. Thus, increased activation of the ventromedial prefrontal regions from social anxiety treatments is beneficial. In addition, the degree of functional connectivity between the VLPFC and the amygdala is negatively correlated with the severity of anxiety [57]. This result supports our results that activation of the VLPFC and the degree of anxiety treatment are correlated.

The blood flow in the right DLPFC decreased in patients with SAD during the repetition task [48]. We observed that activation of the right OFC during the 2-back task in our study increased after treatment, which contradicts the results of some previous studies. Further, regional cerebral blood flow of the rhinal cortex, amygdala, hippocampal region, and parahippocampal region decreases in patients with SAD during task performance. The regional cerebral blood flow of the middle frontal cortex and dorsal ACC changes in patients with SAD, particularly in patients who effectively perform the task [58]. We obtained similar results in our study.

Differences were observed in brain activation in high anxiety–provoking situations rather than in low anxiety–provoking situations [59]. Patients with SAD were more anxious during the first-person view than during the third-person view, which resulted in differences in activation in the HC group. In addition, the changes in brain activation appeared earlier when the video in the third-person view was used, because the patients felt less anxious. However, the effects of the treatment appeared later when the video in the first-person view was used, probably because the patients felt more anxious. Therefore, the effectiveness of the treatment modality should be verified by conducting more treatment sessions.


This study had some limitations. First, the sample size was relatively small. However, our analysis was thorough, and we were able to investigate functional brain activity after VR therapy in patients with SAD. Second, the number of sessions for VR treatment was small, and the follow-up period was relatively short. We observed the changes in psychological scale scores and brain activities measured through fNIRS after VR treatment. However, after the sixth session of treatment, the anxiety levels observed on most psychological scales in the SAD group were higher than in the HC group. This could be because changes in brain activity due to VR treatment were not prominent as yet. Perhaps more treatment sessions are required. Thus, more VR treatment sessions should be conducted for a larger sample size to observe prominent therapeutic benefits. Third, we could not measure the brain activity in real time. Applying electroencephalography to a VR head-mounted display (HMD) has enabled measurement of brain waves [60]. However, we were not able to apply fNIRS and VR HMD because of several technical limitations. Lastly, we could not observe changes in brain areas other than the PFC (especially amygdala), owing to mechanical limitations of portable fNIRS. Although fMRI has the advantage of being able to perform comprehensive and in-depth measurement of the entire brain function, it is difficult to apply various tasks and has a high economic burden, so there are obvious limitations in actual clinical application. However, in terms of cost and usability, portal fNIRS has a high possibility of being applied to actual clinical practice in combination with VR, so it is worthy of overcoming the disadvantages of measurement compared to fMRI.


We conducted VR therapy on patients with SAD to prove the treatment effects after developing a VR treatment program that was exposed to social situations in the first- and third-person views. We identified the brain activity changes in the FPPFC and OFC in patients with SAD after VR treatment in this study, which can provide functional clues in specific brain area for VR treatment. In addition, we found that the STAI-T scores were correlated with the activity of the OFC and the ISS scores were correlated with most of the PFC except the right VLPFC. This can be a clue that the change after treatment in some psychological domains in SAD is related to a specific PFC region. Future studies are required to identify and confirm the neural correlates of VR treatment in patients with SAD for sufficient periods in large sample sizes.


This work was supported by the National Research Foundation (NRF) of Korea grant funded by the Ministry of Science and Information and Communications Technology (MSIT), Government of Korea (NRF-2020R1C1C1007463 and NRF-2021R1A5A8032895) and the Information and Communications Technology (ICT) & Future Planning for Convergent Research in the Development Program for R&D Convergence over Science and Technology Liberal Arts (NRF-2017M3C1B6070978).

Authors' Contributions

HL, JKC, DJ, JWH, and CHC wrote the first draft of the manuscript. JKC, DJ, JWH, and CHC participated in data collection. All the authors edited the manuscript versions. All the authors were also involved in the interpretation of the results and read, commented on, and approved the final version of the manuscript.

Conflicts of Interest

None declared.

  1. Heimberg RG, Hofmann SG, Liebowitz MR, Schneier FR, Smits JAJ, Stein MB, et al. Social anxiety disorder in DSM-5. Depress Anxiety 2014 Jun;31(6):472-479. [CrossRef] [Medline]
  2. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders: DSM-5. Washington, DC: American Psychiatric Association; 2013.
  3. Bögels SM, Alden L, Beidel DC, Clark LA, Pine DS, Stein MB, et al. Social anxiety disorder: questions and answers for the DSM-V. Depress Anxiety 2010 Feb;27(2):168-189. [CrossRef] [Medline]
  4. Cuijpers P, van Straten A. Improving outcomes in social anxiety disorder. Lancet Psychiatry 2014 Oct;1(5):324-326. [CrossRef]
  5. Keller MB. The lifelong course of social anxiety disorder: a clinical perspective. Acta Psychiatr Scand Suppl 2003(417):85-94. [CrossRef] [Medline]
  6. Rodebaugh TL, Holaway RM, Heimberg RG. The treatment of social anxiety disorder. Clin Psychol Rev 2004 Nov;24(7):883-908. [CrossRef] [Medline]
  7. Bandelow B. Current and novel psychopharmacological drugs for anxiety disorders. Adv Exp Med Biol 2020;1191:347-365. [CrossRef] [Medline]
  8. Apolinário-Hagen J, Drüge M, Fritsche L. Cognitive behavioral therapy, mindfulness-based cognitive therapy and acceptance commitment therapy for anxiety disorders: integrating traditional with digital treatment approaches. In: Kim YK, editor. Anxiety Disorders: Rethinking and Understanding Recent Discoveries. Singapore: Springer; 2020:291-329.
  9. Rose GM, Tadi P. Social Anxiety Disorder. Treasure Island, FL: StatPearls; 2021.
  10. Olfson M, Guardino M, Struening E, Schneier FR, Hellman F, Klein DF. Barriers to the treatment of social anxiety. Am J Psychiatry 2000 Apr;157(4):521-527. [Medline]
  11. Rizzo AS, Koenig ST, Talbot TB. Clinical virtual reality: emerging opportunities for psychiatry. Focus (Am Psychiatr Publ) 2018 Jul;16(3):266-278 [FREE Full text] [CrossRef] [Medline]
  12. Lee JM, Ku JH, Jang DP, Kim DH, Choi YH, Kim IY, et al. Virtual reality system for treatment of the fear of public speaking using image-based rendering and moving pictures. Cyberpsychol Behav 2002 Jun;5(3):191-195. [CrossRef] [Medline]
  13. Park MJ, Kim DJ, Lee U, Na EJ, Jeon HJ. A literature overview of virtual reality (VR) in treatment of psychiatric disorders: recent advances and limitations. Front Psychiatry 2019;10:505 [FREE Full text] [CrossRef] [Medline]
  14. Rothbaum BO, Hodges LF, Kooper R, Opdyke D, Williford JS, North M. Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia. Am J Psychiatry 1995 Apr;152(4):626-628. [CrossRef] [Medline]
  15. Valmaggia LR, Latif L, Kempton MJ, Rus-Calafell M. Virtual reality in the psychological treatment for mental health problems: a systematic review of recent evidence. Psychiatry Res 2016 Feb 28;236:189-195. [CrossRef] [Medline]
  16. Oing T, Prescott J. Implementations of virtual reality for anxiety-related disorders: systematic review. JMIR Serious Games 2018 Nov 07;6(4):e10965 [FREE Full text] [CrossRef] [Medline]
  17. Zeng N, Pope Z, Lee JE, Gao Z. Virtual reality exercise for anxiety and depression: a preliminary review of current research in an emerging field. J Clin Med 2018 Mar 04;7(3):42 [FREE Full text] [CrossRef] [Medline]
  18. Vass E, Fekete Z, Lencse L, Ecseri M, Kis B, Szekrényes-Varga Á, et al. Treatment of Theory of Mind deficits in schizophrenia by using Virtual Reality, the VR-ToMIS training (Virtual Reality based Theory of Mind Intervention in Schizophrenia). Psychiatr Hung 2019;34(3):287-299. [Medline]
  19. Lalonde G, Henry M, Drouin-Germain A, Nolin P, Beauchamp MH. Assessment of executive function in adolescence: a comparison of traditional and virtual reality tools. J Neurosci Methods 2013 Sep 30;219(1):76-82. [CrossRef] [Medline]
  20. Didehbani N, Allen T, Kandalaft M, Krawczyk D, Chapman S. Virtual reality social cognition training for children with high functioning autism. Comput Hum Behav 2016 Sep;62:703-711. [CrossRef]
  21. Kellmeyer P. Neurophilosophical and ethical aspects of virtual reality therapy in neurology and psychiatry. Camb Q Healthc Ethics 2018 Oct;27(4):610-627. [CrossRef] [Medline]
  22. Roy S, Klinger E, Légeron P, Lauer F, Chemin I, Nugues P. Definition of a VR-based protocol to treat social phobia. Cyberpsychol Behav 2003 Aug;6(4):411-420. [CrossRef] [Medline]
  23. Pertaub DP, Slater M, Barker C. An experiment on fear of public speaking in virtual reality. Stud Health Technol Inform 2001;81:372-378. [Medline]
  24. Robillard G, Bouchard S, Dumoulin S, Guitard T, Klinger E. Using virtual humans to alleviate social anxiety: preliminary report from a comparative outcome study. Stud Health Technol Inform 2010;154:57-60. [Medline]
  25. Anderson PL, Price M, Edwards SM, Obasaju MA, Schmertz SK, Zimand E, et al. Virtual reality exposure therapy for social anxiety disorder: a randomized controlled trial. J Consult Clin Psychol 2013 Oct;81(5):751-760. [CrossRef] [Medline]
  26. Kampmann IL, Emmelkamp PMG, Hartanto D, Brinkman W, Zijlstra BJH, Morina N. Exposure to virtual social interactions in the treatment of social anxiety disorder: a randomized controlled trial. Behav Res Ther 2016 Feb;77:147-156. [CrossRef] [Medline]
  27. Bouchard S, Dumoulin S, Robillard G, Guitard T, Klinger, Forget H, et al. Virtual reality compared with exposure in the treatment of social anxiety disorder: a three-arm randomised controlled trial. Br J Psychiatry 2017 Apr;210(4):276-283. [CrossRef] [Medline]
  28. Wechsler TF, Kümpers F, Mühlberger A. Inferiority or even superiority of virtual reality exposure therapy in phobias? A systematic review and quantitative meta-analysis on randomized controlled trials specifically comparing the efficacy of virtual reality exposure to gold standard exposure in agoraphobia, specific phobia, and social phobia. Front Psychol 2019;10:1758 [FREE Full text] [CrossRef] [Medline]
  29. Pinti P, Tachtsidis I, Hamilton A, Hirsch J, Aichelburg C, Gilbert S, et al. The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Ann N Y Acad Sci 2020 Mar;1464(1):5-29 [FREE Full text] [CrossRef] [Medline]
  30. Yoon H, Seo EH, Kim J, Choo IH. Neural correlates of self-referential processing and their clinical implications in social anxiety disorder. Clin Psychopharmacol Neurosci 2019 Feb 28;17(1):12-24 [FREE Full text] [CrossRef] [Medline]
  31. Yoo SW, Kim YS, Noh JS, Oh KS, Kim CH, NamKoong K, et al. Validity of Korean version of the MINI-international neuropsychiatric interview. Anxiety Mood 2006;2(1):50-55.
  32. Kim H, Lee S, Jung D, Hur J, Lee H, Lee S, et al. Effectiveness of a participatory and interactive virtual reality intervention in patients with social anxiety disorder: longitudinal questionnaire study. J Med Internet Res 2020 Oct 06;22(10):e23024 [FREE Full text] [CrossRef] [Medline]
  33. Hur J, Shin H, Jung D, Lee H, Lee S, Kim GJ, et al. Virtual reality-based psychotherapy in social anxiety disorder: fMRI study using a self-referential task. JMIR Ment Health 2021 Apr 14;8(4):e25731 [FREE Full text] [CrossRef] [Medline]
  34. Cho CH. Social Anxiety Disorder VR treatment. 2019.   URL: [accessed 2021-12-08]
  35. Yook SP, Kim ZS. A clinical study on the Korean version of Beck Anxiety Inventory: comparative study of patient and non-patient. Korean J Clin Psychol 1997;16(1):185-197.
  36. Kim JT, Shin DK. A study on the standardization of STAI for Korea. N Engl J Med 1978;21(11):69-75.
  37. Lee IS, Choi HR. Assessment of shame and its relationship with maternal attachment, hypersensitive narcissism and loneliness. Korean J Couns Psychother 2005;17(3):651-670.
  38. Im S, Kwon SM, Choi H. Korean J Clin Psychol 2007 Feb;26(1):39-56. [CrossRef]
  39. Kim HS. Memory Bias in Subtypes of Social Phobia (masters thesis). Seoul, South Korea: Seoul National University; 2001.
  40. Hong Y, Moon J, Jo H. A validation study of the Korean version of the Fear of Negative Evaluation Scale-2. Korean J Psychol 2011;30:117-134.
  41. Lee JY, Choi CH. A study of the reliability and the validity of the Korean versions of social phobia scales (K-SAD, K-FNE). Korean J Clin Psychol 1997;16(2):251-264.
  42. Yu E, Ahn CY, Hwan PK. Factor structure and diagnostic efficiency of a Korean version of the Liebowitz Social Anxiety Scale. Korean J Clin Psychol 2007 Feb;26(1):251-270. [CrossRef]
  43. Wagner G, Koch K, Schachtzabel C, Peikert G, Schultz CC, Reichenbach JR, et al. Self-referential processing influences functional activation during cognitive control: an fMRI study. Soc Cogn Affect Neurosci 2013 Oct;8(7):828-837 [FREE Full text] [CrossRef] [Medline]
  44. Judah MR, Grant DM, Lechner WV, Mills AC. Working memory load moderates late attentional bias in social anxiety. Cogn Emot 2013;27(3):502-511. [CrossRef] [Medline]
  45. Delpy DT, Cope M, van der Zee P, Arridge S, Wray S, Wyatt J. Estimation of optical pathlength through tissue from direct time of flight measurement. Phys Med Biol 1988 Dec;33(12):1433-1442. [CrossRef] [Medline]
  46. Sun J, Liu F, Wang H, Yang A, Gao C, Li Z, et al. Connectivity properties in the prefrontal cortex during working memory: a near-infrared spectroscopy study. J Biomed Opt 2019 Mar;24(5):1-7 [FREE Full text] [CrossRef] [Medline]
  47. Weed LL. Medical records that guide and teach. N Engl J Med 1968 Mar 21;278(12):652-7 concl. [CrossRef] [Medline]
  48. Mathew SJ, Ho S. Etiology and neurobiology of social anxiety disorder. J Clin Psychiatry 2006;67 Suppl 12:9-13. [Medline]
  49. Yokoyama C, Kaiya H, Kumano H, Kinou M, Umekage T, Yasuda S, et al. Dysfunction of ventrolateral prefrontal cortex underlying social anxiety disorder: a multi-channel NIRS study. Neuroimage Clin 2015;8:455-461 [FREE Full text] [CrossRef] [Medline]
  50. Evans KC, Simon NM, Dougherty DD, Hoge EA, Worthington JJ, Chow C, et al. A PET study of tiagabine treatment implicates ventral medial prefrontal cortex in generalized social anxiety disorder. Neuropsychopharmacology 2009 Jan;34(2):390-398. [CrossRef] [Medline]
  51. Hiser J, Koenigs M. The multifaceted role of the ventromedial prefrontal cortex in emotion, decision making, social cognition, and psychopathology. Biol Psychiatry 2018 Apr 15;83(8):638-647 [FREE Full text] [CrossRef] [Medline]
  52. Doruyter A, Dupont P, Taljaard L, Stein DJ, Lochner C, Warwick JM. Resting regional brain metabolism in social anxiety disorder and the effect of moclobemide therapy. Metab Brain Dis 2018 Apr;33(2):569-581. [CrossRef] [Medline]
  53. Klumpp H, Fitzgerald DA, Phan KL. Neural predictors and mechanisms of cognitive behavioral therapy on threat processing in social anxiety disorder. Prog Neuropsychopharmacol Biol Psychiatry 2013 Aug 01;45:83-91 [FREE Full text] [CrossRef] [Medline]
  54. Marin M, Zsido RG, Song H, Lasko NB, Killgore WDS, Rauch SL, et al. Skin conductance responses and neural activations during fear conditioning and extinction recall across anxiety disorders. JAMA Psychiatry 2017 Jun 01;74(6):622-631 [FREE Full text] [CrossRef] [Medline]
  55. Nishiyama Y, Okamoto Y, Kunisato Y, Okada G, Yoshimura S, Kanai Y, et al. fMRI study of social anxiety during social ostracism with and without emotional support. PLoS One 2015;10(5):e0127426 [FREE Full text] [CrossRef] [Medline]
  56. Stein MB, Stein DJ. Social anxiety disorder. Lancet 2008 Mar;371(9618):1115-1125. [CrossRef]
  57. Fitzgerald JM, Klumpp H, Langenecker S, Phan KL. Transdiagnostic neural correlates of volitional emotion regulation in anxiety and depression. Depress Anxiety 2019 May;36(5):453-464 [FREE Full text] [CrossRef] [Medline]
  58. Doruyter AG, Dupont P, Stein DJ, Lochner C, Warwick JM. Nuclear neuroimaging in social anxiety disorder: a review. J Nucl Med 2018 Dec 05;59(12):1794-1800 [FREE Full text] [CrossRef] [Medline]
  59. Glassman LH, Kuster AT, Shaw JA, Forman EM, Izzetoglu M, Matteucci A, et al. The relationship between dorsolateral prefrontal activation and speech performance-based social anxiety using functional near infrared spectroscopy. Brain Imaging Behav 2017 Jun;11(3):797-807. [CrossRef] [Medline]
  60. Wang W, Ho RLM, Gatto B, Der Veen SMV, Underation MK, Thomas JS, et al. A novel method to understand neural oscillations during full-body reaching: a combined EEG and 3D virtual reality study. IEEE Trans Neural Syst Rehabil Eng 2020 Dec;28(12):3074-3082. [CrossRef]

ACC: anterior cingulate cortex
BAI: Beck Anxiety Inventory
BFNE: Brief-Fear of Negative Evaluation
DLPFC: dorsolateral prefrontal cortex
DSM-5: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition
fMRI: functional magnetic resonance imaging
fNIRS: functional near-infrared spectroscopy
FPPFC: frontopolar prefrontal cortex
HbO2: oxygenated hemoglobin
HbR: deoxygenated hemoglobin
HC: healthy control
HMD: head-mounted display
ISS: Internalized Shame Scale
K-SAD: Korean Social Avoidance and Distress
K-SIAS: Korean version of the Social Interaction Anxiety Scale
K-SPS: Korean version of the Social Phobia Scale
LSAS: Liebowitz Social Anxiety Scale
MINI: Mini-International Neuropsychiatric Interview
MPFC: medial prefrontal cortex
OFC: orbitofrontal cortex
PCC: posterior cingulate cortex
PERS: Post-Event Rumination Scale
PET: positron emission tomography
PFC: prefrontal cortex
SAD: social anxiety disorder
STAI: State-Trait Anxiety Inventory
VLPFC: ventrolateral prefrontal cortex
VR: virtual reality

Edited by G Eysenbach; submitted 07.07.21; peer-reviewed by JH Baek; comments to author 28.07.21; revised version received 25.08.21; accepted 21.11.21; published 17.12.21


©Hojun Lee, JongKwan Choi, Dooyoung Jung, Ji-Won Hur, Chul-Hyun Cho. Originally published in the Journal of Medical Internet Research (, 17.12.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.