User Testing VR Games: analysis and recommendations
VR is common topic in the tech, entertainment, and gaming worlds right now. It has captured people’s imaginations, appears to be heading towards relatively low cost solutions, and has an unmistakable wow factor that leaves users open-mouthed in amazement.
This sets my User Research sense tingling. I was fortunate enough to do a lot of testing on Kinect games during the motion gaming heyday and was one of the first researchers to get their hands on Kinect V2 during the development of the Xbox One. New platforms and new input devices are both exciting and present unique challenges. VR is both a new platform (in that it offers a qualitatively different experience than viewing games and entertainment on a 2D display) and a new input device (in that it allows users some degree of motion control via head tracking.
Developing for new platforms and new inputs are challenging, and doing research on them can be challenging as well. Some challenges I think that are specific to VR UR are:
- The current “wow” factor associated with VR can make getting valid user opinions more difficult
- It is harder to for researchers to view both the user and the user’s POV simultaneously, making traditional usability more difficult
- VR is still an emerging platform, and target users are still somewhat unknown, making participant recruitment more challenging
- Controls, standards, and interaction patterns for VR are still very much in flux, which means researchers and designers don’t have standard patterns to fall back on when making designs and recommendations
- There is little (public) documentation on methods specific to VR UR
I’m going to discuss each of these challenges in turn, along with a recommendation on the best way to tackle that challenge. First, though, some caveats. Unlike some of my colleagues at MS, Facebook, Sony, and Samsung, I haven’t actually worked formally on the Oculus Rift. I’m a hobbyist, but I have years of experience doing UR in multiple domains, across multiple platforms, and using multiple (sometimes novel) input methods. Also, I expect to be working on this platform eventually, no matter who I work for, so it’s smart to start thinking about these things now.
Problem: Wow Factor Interferes with Assessment
Many people have described the “wow” factor that VR provides. X said X. Y said Y. Z said Z. The consensus was that these technologies have a profound emotional effect on users. As a result of the Halo Effect (no relation to Master Chief), anything associated with something that elicits a strong positive or negative reaction will have some of that reaction sort of rub off on it. An example of how this can cause problems can be seen in Playtest, which relies heavily on attitudinal surveys to make decisions about many aspects of a game. A playtest of a VR game using VR novices would likely result in higher scores for the game than it would receive in the absence of the wow factor.
Don’t get me wrong. The wow factor is an incredibly important part of the product experience, and can sell units like crazy. The only problem is that without strong experiences that can stand on their own once the wow factor has faded, games and even platforms will end up performing less well than expected. Recall the high sales numbers of the Wii and the Kinect during the rise of motion controls. As the novelty of these experiences wore off and usability issues that were masked by the wow factor came to the forefront, games using motion controls became less popular. Eventually the platforms that were the inheritors of the motion control era (Wii U and Xbox One) underperformed in part because the expectation that users still wanted to use motion controls for their game no longer held true.
While there were a number of other problems with the games that came out for motion controls, such as the tendency to translate in-development games to motion controls rather than developing experiences that could never be realized without motion controls, this is something VR developers and URs need to keep in mind as they assess VR games so the same thing doesn’t happen to VR.
Recommendation: Consider allowing users time to acclimate to VR before exposing them to the experience of interest.
Specifically, this can be done by:
- Recruiting users from current VR owners, though you may run into enthusiast bias
- Giving users time to experience VR before exposing them to the product you’re specifically testing. This can either be done in the lobby before the test starts or in a separate session done on a different day than the test of the product you’re interested in.
Problem: Hard to see the Users POV and the User simultaneously
There is not currently a great way to observe what another person is doing in VR directly. For Oculus Rift, the most common non-VR visualization is showing what is projected by the screen, uncorrected by the lenses inside the Rift (an example of this can be seen in the image above). This can be difficult to view and very hard for a researcher to parse. Additionally, one advantage that traditional usability has is that it is easy for the Researcher to view both the user is doing and what the user can see simultaneously. Even if a researcher were to mirror a user’s experience on another Rift, they would at the same time sacrifice their view of the user’s posture, body motions, controller usage, etc.
Recommendation: Consider working in teams of two, one UR observing the user and user behavior outside of VR and the other observing the user perspective within VR.
This is a pretty expensive solution, and is much more difficult for smaller teams to resource. Alternately:
Recommendation: Consider developing tools that allow a single user researcher to observe both user perspective and user actions simultaneously
- An example of this would be mirroring the view of the user on another VR headset with inset windows showing user actions, like what is typically displayed on observation room monitors. There would be challenges in developing this, but it would allow the Researcher to not lose user perspective while still observing behavior. On the downside, observing someone else’s perspective without matching movement may result in dizziness and nausea. Bring Dramamine.
- Another solution would be a tool that would transform the dual, distorted video that is currently the observer view into a 2-D monitor view. This would not be perfect (losing, for example, the 3-D perspective of VR), but would be an improvement over current solutions.
Target Users Currently Unknown
One of the first thing a User Researcher should know when planning research is who is the target audience of the product. This information is critical to—among other things—knowing who should be recruited for studies, special considerations in who to allow and who to exclude.
It is often unclear who the target market is for new platforms or technologies. Many people predicted that the Wii would fail because hardcore gamers were more interested in power than playing golf with a not-very-accurate wiimote. Instead, Nintendo sold many millions of units to a non-traditional market. VR is in a similar position. Xbox sold many 360s to non-core gamers, but overestimated the appeal motion gaming would have to its core audience, resulting in multiple failed titles and some of the issues experienced during the Xbox One launch.
Nobody knows who will be VR’s market. Currently it’s only taken hold among enthusiasts that are willing to throw down hundreds of dollars on equipment that will likely be deprecated in a relatively short amount of time. Unsurprisingly, many of these enthusiasts are hardcore gamers, coming from PC gaming’s culture of bleeding edge technology.
Many, including companies like AltspaceVR, Facebook, and Sony are counting on VR being the new social, used by the same people who use social media on smartphones today. Chris Cox, of Facebook recently said of social use of VR: "You'll do it. Beyonce will do it." Others, such as Microsoft, claim that VR will find its niche in business as companies become increasingly decentralized and working from remote locations becomes more common. Others claim it will continue to be a niche market limited to the hardcore tech enthusiast.
Recommendation: Conduct thorough foundational research and testing to determine VR’s appeal to different demographic groups.
I am certain this kind of research is happening in labs across the world, hidden by a veil of secrecy. For the rest of us, a smart consulting firm would invest in doing this research now and release a report to interested parties for a healthy fee. If I were to design these studies I’d probably:
- Start by conducting interviews with current VR users to understand the needs VR fills for them, their motivations for engaging with VR, and compare those needs and motivations with other groups.
- Loan Gear VR to participants of various demographics and ask them to track their usage patterns over several weeks as part of a diary study.
Interaction, UI, and Controls Standards are in Flux
As new technology arises, there are often competing standards for that technology. Qwerty and Dvorak. VHS and Betamax. Flash and HTML5. Over time, users end up selecting the winner and other companies in the same industry move towards using the similar or same standards.
VR is still very much a technology in its infancy. While head-tracking seems stable and intuitive, the question of how users will move through and interact with objects in 3D space is still very much open. There are many solutions to this question out there, including Sony's Playstation Move controllers, Leap Motion, body position sensors such as Kinect, and many many more.
A separate question is User Interfaces for VR. Designers have already taken on that task, but there are no standards or patterns yet that designers and practitioners can rely on for creating or assessing VR UX.
Recommendation: Consider research and design approaches that rely on fundamentals of design and usability
There are many sources that are technology agnostic that can help us think about design and research for VR. Dieter Ram's 10 principles of good design are an excellent place to start, as are Nielson's principles of usability. Beyond that, many of the heuristics and assumptions User Researchers have been making to help products be more usable may be suspect, as most (if not all them) are specific to 2D environments. Take for example hit indicators in a first person shooter. The pattern that has emerged over time is the red marker indicating damage that appears on a point in a circle approximating the angle from which the shot was taken seen from above. Users have become accustomed to this convention, but it will become confusing again if this pattern is simply lifted straight from 2D interfaces into 3D VR interfaces. Thinking of these problems from a new perspective will involve a new skillset and toolbox for designers and researchers that is only beginning to be built.
Problem: Little Public Discussion of VR UR
Because VR is still a technology just beginning to emerge from infancy, Most of the usability and user research work being conducted with it as a focus is hidden behind NDAs and trade secrets. This will begin to lessen as URs begin to present their work at professional conferences such as GDC, HCI, and GUR SIG, but for now we have little to go on. This results in teams across the world reinventing the wheel, wasting valuable time that could be spent making VR products better.
Recommendation: Wait. And Publish
This one will resolve itself over time. In the meantime, I will be updating this blog post in the coming weeks with an annotated bibliography of the current publicly known research on VR User Research, Market Research, and Usability. I think it's important that VR UR practitioners start talking to each other sooner rather than later. Hopefully this analysis will and future posts will help stimulate the conversation between VR UR practitioners.
Hopefully these observations and recommendations will be helpful to those doing research in this field. My colleague Matt Rowens recommended that I include a link to the Oculus Rift Best practices. It contains a wealth of information that can help those new to VR think productively about VR design and usability.