Meta Quest Accuracy
Client:
Study Design:
My Role:
Date
Location
Team:
Meta Reality Labs
Quantitative benchmark
Lead UX Researcher
October-December 2022
New York, Chicago, & Seattle
2 UX Researchers, 1 Designer, 1 Research Scientist and a team of 4 Research Assistants
Executive Summary
A Quantitative benchmark aimed at improving hand tracking quality on the Meta Quest device. The research identified troubled gesture areas and camera sensitivity issues, ultimately driving accuracy score improvements that allowed hand tracking quality to reach over 90% accuracy.
BACKGROUND
User Problem & Key Challenges
User Problem: Hand tracking was underutilized, qualitative feedback gathered earlier that year pointed to low accuracy rate compared to controllers (Hand Tracking > 90%, Controls <98%)
Challenge: This multi-state initiative required extensive cross-functional coordination beyond design and engineering, involving legal, accounting, and an external research partner-Ipsos.
Research Questions & Hypothesis
1. What is the current accuracy score of our hand tracking system (using recall, F1 and precision scores)
2. Which gesture sets are bringing accuracy scores down?
Hypothesis: If we identify the troubled gestures, then we can facilitate hand tracking use and adoption because we will be able to modify hand tracking sensitivity to reduce false positives and increase accuracy
METHODS
Participants
Target Sample:
Users had to be new to VR with no prior experience. They ranged from 18-65yo
Chicago (Pilot) n=13
New York n=72
We needed a big sample to ensure accuracy scores were precise
Recruitment:
Conducting the study in multiple states opened up our sample pool since the greater Seattle area was extensively explored
Study Design
Wizard of Oz
Users were shown an arrow indicating the direction their gesture should go. Although they believed they conducted the gesture, the moderater input the real gesture
Additional Data
Biometrics were gathered to explore factors that influenced accuracy
Qualitative feedback
Users were asked for overall thoughts about the gestures they were asked to conduct.
Analytics & Presentation
Analytics
Extensive data cleaning was done to identify cases of false positives and negatives. Using this data, we calculated F1, precision and recall scores. Statistical testing was performed to see if any gesture was significantly different than the others.
Presentation
A slide deck was shown to the team and leadership together
RESULTS
Findings
1. Most gestures needed sensitivity modifications but one particular gesture was low in accuracy and based on user feedback, was harder to perform due to limited range.
2. Skin tone and resting hand position were identified as potential factors that influenced accuracy
Impact
The team refined the troubled gesture moving its location for better range and increasing camera sensitivity of the gesture.
Modification of camera sensitivity resulted in increased accuracy >90% bringing hand tracking closer to controller levels
Further research was conducted to understand influencing factors such as skin tone and resting hand position.
REFLECTIONS
Lessons
Data cleaning and analysis took over 3 weeks for this project. Now, I've built an AI agent to handle quantitative data processing since this form of data can be easily verified and fact-checked with the right strategy. I stay hands-on with qualitative data until I've captured the key findings, then use AI to shape them into a clear, accurate presentation. The result: processing time cut from days to hours!
