A student using a virtual reality headset.
Srdjan Lesaja, a VCU biomedical engineering Ph.D. student, models a virtual reality headset. (Photo courtesy VCU Engineering)

Next-level virtual reality: Environments that respond to the user’s mental state

A VCU researcher is investigating how VR headsets could react to the user in a way that helps them better enjoy the experience.

Share this story

Dean Krusienski, Ph.D., a professor in the Department of Biomedical Engineering in the Virginia Commonwealth University College of Engineering, is investigating how to make virtual reality environments responsive, in real time, to the user’s state of mind.

His project to develop techniques that incorporate feedback from brain signals into VR interactions has received a two-year grant from the National Science Foundation and its Cyber-Human Systems and Integrative Strategies for Understanding Neural and Cognitive Systems programs.

“We are trying to estimate a user’s cognitive state while they are performing in an interactive virtual reality environment,” he said. “The ultimate goal is to manipulate the VR experience based on our estimation of the cognitive state.”

For example, someone using VR to relax could experience their environment adjusting to help them reach a deeper state of relaxation. 

Another application would be in training, such as for air traffic controllers. If the VR system could determine the user’s level of stress or workload in a skilled task training situation, it could adjust the difficulty level to make the most efficient use of the training time, Krusienski said. 

“If the task is too hard for them, we could slow it down or make the objects they’re interacting with larger or smaller,” he said. “If it’s too easy, we can add challenges to the environment.” 

He is focusing on collecting feedback from brain signals through electroencephalogram monitoring as a way to assess the user’s mental workload or emotional state. As a tool for diagnosing medical conditions, an EEG uses electrodes attached to the scalp to detect electrical activity in the brain. Krusienski is also exploring how to optimally configure EEG sensors so they can be incorporated into VR headsets. 

We are trying to estimate a user’s cognitive state while they are performing in an interactive virtual reality environment. The ultimate goal is to manipulate the VR experience based on our estimation of the cognitive state.

He is collaborating with Yusuke Yamani, Ph.D., assistant professor in the Department of Psychology at Old Dominion University. 

Krusienski, who is also the graduate program director for the Department of Biomedical Engineering, was trained in electrical engineering and has been studying brain-signal analysis since he was a postdoctoral fellow.

Being able to sense a user’s state of mind could provide benefits in a host of situations, such as helping people improve their performance in driving or in health care. 

In gaming, he said, if a VR system could receive immediate feedback on the player’s level of excitement, stress or even fear, it could “react in a way that better engages them and helps them enjoy the experience.”

The project is part of his lab’s goal to achieve a greater understanding of the brain.

“We try to keep this research as fundamental as possible to increase the likelihood that they’ll be more broadly applicable,” he said.