User Research for VR – six real-life challenges
Over the last few months, our team has been focusing on a number of Virtual Reality (VR) projects. We've had to adapt our usual approaches to meet the peculiarities of this context. While user research in VR shares many common aspects with any other research projects, there are a number of significant differences. In this article, I discuss six of them.
1. Give users time to acclimatise to the experience
Nic gets artistic with Google's Tilt Brush on the HTC Vive
For many of the non-VR usability sessions that we’ve delivered for clients, our participants have had a good grasp of how to use a keyboard and mouse (or relevant accessibility) tools, if not necessarily the pinch and swipe of mobile devices.
However, VR is different. It’s new enough – at least from a commercial perspective – that few people have much experience with it and there are multiple hardware vendors and control options. Until VR becomes commonplace and one interaction style wins over the rest, we’ll be testing the combination of hardware and software together. This has challenges for both participant and the user research team.
To mitigate for this, we need to allow time for users to get familiar with the hardware and controls before we can start testing any application. We also consider the order in which the participant is exposed to different experiences – as they become more familiar with the controls, their experience is likely to improve.
2. Think about health and safety
Health and safety is not usually a major concern for non-VR user research. Standard office policies apply, especially keeping the complimentary beverages away from the keyboard. But the VR user research project introduces a whole new set of concerns:
Once immersed in the virtual experience, it’s easy to lose awareness of the physical environment. Avoiding collisions in the virtual world can lead the user to collide with objects in the real world. As hosts of the research session, we’re responsible for the wellbeing of our participants – we must ensure that we provide a large enough physical space free from hazards and keep a constant watch on our subject to prevent collisions, trips, entanglement and numerous other potential problems.
After a busy day of user research, that headset can start to get a bit funky. No-one wants to be sharing their predecessor’s sweat. So it's important to allow time between sessions for a quick clean up, dry off or change of padding where appropriate.
For some people, using VR can be very uncomfortable. In one project, we found that about 60% of users experienced some level of nausea or dizziness. Obviously we want to test applications so that the release will not cause discomfort to users, but we also need to be mindful of our test participants. How many participants have to suffer and to what degree, before we as researchers pull the plug on a project that we're being paid to deliver results on?
While being conscious of the ethical considerations, there are a number of factors that we can influence to help reduce the occurrence or severity of cybersickness:
- Ensure that the headset is properly adjusted for each user – the better quality Google cardboard headsets allow you to make adjustments for each eye
- Reduce exposure time – does every part of the session have to take place inside the virtual environment? Consider having a break to ask some of your questions
- Give participants the option of sitting where possible to reduce the occurrence
- Turn off any unnecessary background applications that could potentially reduce system performance. There is a direct link between lower frame rates and cybersickness
- Ask participants not to come in if they are feeling unwell or tired
- Don't prime the participant by suggesting that they may feel sick – although you should express that they can take a break at any time
- The other way that we can reduce the effect of cybersickness is through our influence. There's an excellent article from Block Interval about changes that the development team can make
3. Is my facilitation ruining the experience?
Getting valuable feedback from a participant is always a fine balance between observation as they freely explore the application – and asking questions to extract their inner thoughts and feelings.
The VR environment further complicates this. The beauty of a good VR environment is that it takes you to another place outside of your normal reality. Having a disembodied voice asking you questions can only detract from the immersion of the experience but, in order to gain an understanding of the user's existence, this enquiry is vital.
To mitigate somewhat and limit disrupting that reality as much as possible, it’s important to be really clear in advance about what you're trying to discover. In future, it will be interesting to experiment with different approaches to facilitation. For example, having a facilitator also within the same VR space, or having interactive prompts within the VR experience, may possibly reduce the interference with immersion.
4. Does observation affect what's being observed?
In science, the observer effect is well documented and something that user researchers need to be aware of, not just in terms of affecting the user's behaviour, but also in terms of running additional processes to monitor what the user is doing.
In one recent study using an Android VR application, we needed to run a third-party tool to enable the team to see what the participant was seeing. The combination of the VR application, the monitoring application, running Bluetooth for the controller, WiFi and default phone apps caused a significant battery drain and dramatic increase in temperature for the phone. There was a concern that this may also be affecting the performance of the application under test and therefore the experience. In this case, we concluded that there hadn’t been any significant deterioration, but that's not to say that there wasn't any.
Hard to read a participant’s expression when they’re facing the other direction
5. …And there's more moderation problems
In a ‘regular’ user research session, we can pick up on a lot of non-verbal cues such as facial expression and body language. These are not as readily accessible in a VR context – the headset blocks a lot of the face – and the participant might actually be looking in the opposite direction.
We can also often infer or predict the user's thoughts and feelings based on their interactions with the environment and their body language. But this is much harder to rely on. We might be able to have a glimpse of what they are seeing on a flat monitor and hear some of the sounds but, as observers, we are not getting the same experience as the user as we are missing out on the 3D visuals and stereophonic sound.
Facilitator’s view of the mobile application. Hopefully the participant is not experiencing double vision
6. Maybe we need some new tools
Most user researchers are probably familiar with the System Usability Scale (SUS) and have used it for assessing usability of websites and perhaps other applications. But what about VR? We've experimented with an adaptation of the SUS for VR but it's too early for us to use it with full confidence.
The challenge is half the appeal!
It's not all negative though. Recruitment is often the most painful part of a user research project. In our last VR project, we recruited online and all 25 places were booked up within 48 hours!
Although there are difficulties with running a VR project, the need for user research is arguably more important than ever when considering the virtual reality environment and, for me, these extra challenges only add to the appeal.
Credits: Applications under development from Corvecto and Pepper Creative.