A/B Testing: Giving Back to Our Veterans
CSULB collaboration with Veteran Affairs Spinal Cord Injury Center
Giving Back to Our Veterans
Timeline
Dec 2019 - Jan 2021
My Role
UX Researcher + Designer
Methods
Literature reviews, observations, interviews, heuristic evaluation, prototyping, A/B testing
Tools
Qualtrics, Zoom, Figma, Excel
Overview
The VA provides care to more than 27,000 Veterans with spinal cord injuries and disorders (SCI/D) each year, making the department the largest healthcare system in the world providing lifelong spinal cord care. The autonoME Hospital Environmental Control Unit (ECU) is a device currently deployed throughout the VA’s 25 Spinal Cord Injuries and Disorders Centers to assist patients who are fully or partially paralyzed. The goal of this project is to identify usability issues with the existing ECU, conduct A/B testing, and redesign the interface.
Our work was published at HCI International 2020 and presented at HFES 2020.
The Challenge
The Long Beach VA Hospital approached out team with a problem: despite the introduction of a new environmental control unit (ECU), patients at the Spinal Cord Injury and Disorder Center weren't utilizing it. Collaborating closely with hospital staff and patients, we embarked on a journey to revamp the device's interface through a meticulous human-centered design process.
Process
Phase 1
Literature Review & Competitive Analysis
In the first month of this project, we focused our efforts on an extensive literature review. Our goal was to understand the functional limitations of spinal cord injuries and disorders (SCI/D) and its effects, how SCI/D individuals were currently using the ECU device, and any needs or pain points they experienced during use.
We also conducted a competitive analysis of similar assistive technologies across the four modalities: touch, eye gaze, sip and puff/tongue control, and voice recognition. I organized our repository of research insights into a shared Google spreadsheet.
Most frequently used features:
Watching TV or movies (80.7%)
Adjusting the bed (52.6%)
Using social media communication (14%)
Making/answering telephone calls (7%)
Reasons for not using the ECU:
Need for assistance and maintenance (89%)
Troubleshooting/repair issues with the system (71.2%)
Issues with startup protocol - patient admission, education, ECU configuration (28.8%)
Main issues during use:
75% of Veterans indicated that they had problems using the ECU due to:
Did not receive training on how to use the device (50%)
Difficult to use (41%)
Problems working the features (30%)
Technical issues (26%)
Ethnographic Observations & Interviews
Observations and interviews were conducted during our first introductory visits at the VA hospital. We worked closely with nurses, who demonstrated how to set up and use the device across the different input methods (voice, eye control, sip and puff, touch). Interviews with nurses were extremely open-ended and more of a conversation.
We gathered insights on how the device was being used, the environment it was being used in, and common errors associated with use. Our earlier research validated these insights.
Heuristic Evaluation
While working on the IRB documentation for approval to start formal interviews with SCI/D veterans, we decided to move forward with a heuristic evaluation for the touch modality of the ECU interface. A group of 7 evaluators (including myself) evaluated the touch modality of the ECU interface against Nielsen’s 10 Heuristics and Shneiderman’s 8 Golden Rules, identified usability issues, and proposed a list of design recommendations. We published this work in a conference paper presented at the Human Computer Interaction (HCI) International Conference in June 2020.
Prioritization of issues
The heuristic analysis revealed 27 usability issues, ranging from cosmetic to catastrophic. Based on the heuristic analysis, we identified the 3 most violated heuristics as: “user control and freedom,” “error prevention,” and “offer informative feedback.”
Catastrophic usability problems (15%)
Major usability problems (30%)
Minor usability problems (55%)
Addressing pain points
For each usability violation, we proposed a specific design recommendation to improve upon these issues.
Key recommendations:
Include a permanently visible, clearly defined backward navigation “Go Back” button at the top of every screen that returns the user to the previous screen
Change green-red color coding to blue-orange on Main Menu and stay consistently blue across all sub-menus
Darken the same color when pressing the button (i.e., green flashes a darker green) instead of green buttons flashing red when pressed
Phase 2
A/B Testing
We implemented the key recommendations into a Beta version of the prototype. I led the design of the fully digital prototype that we used for remote A/B testing.
Limitions & adapting our research plan
In March 2020, we were no longer permitted to visit the VA hospital due to COVID-19 safety concerns. Due to IRB regulations, we were also unable to conduct remote testing with SCI/D veterans. As an alternative approach, we decided to test with a neurotypical population (i.e., persons without disabilities) and use this data as a benchmark for comparison once we were able to resume testing with SCI/D veterans.
Hypothesis
We hypothesized that if A/B testing results indicated poor performance by a neurotypical population on the beta version of the device, it would likely suggest that a disabled population would also experience difficulties.
Subject recruitment
Participants were recruited through the CSULB Psychology, Art/Graphic Design, and Biology departments. Example screener questions included:
Do you have access to a computer, laptop, or tablet device?
Are you color blind?
Which touch screen devices do you use?
Measure UX & UI outcomes
We used validated questionnaires to measure the following --
Stress (Dundee Stress State Questionnaire, 2001)
Workload (NASA TLX, 1988)
Frustration (PSSUQ and SUS)
Fatigue (Borg scale, 1998)
Test variants
Alpha: Current version of the ECU currently implemented at the VA Hospital.
Beta: Version designed by research team that implemented all the recommendations from the heuristic analysis.
Metrics
Task completion rate
Speed
Accuracy
Tasks
Enter VA hospital phone number
Compose an email
Watch a Youtube video
Watch a training video
Randomization scheme
Alpha
Task order: 1, 2, 3, 4
Questionnaire order: NASA TLX, PSSUQ, Borg, Post-DSSQ, SUS
Beta
Task order: 2, 4, 1, 3
Questionnaire order: Post-DSSQ, PSSUQ, NASA TLX, SUS, Borg
Results
Achieved a 23% increase in task accuracy and improved overall user experience
Publications
Hancock, G.M., Anvari, S.S., Nare, M.T., Mok, N.B., Ayvazyan, A., McCoy, K.M., Bai, X., Mather, G.P., McBride, A.S., & Morales, N. (2020). Environmental control units for inpatient care at Veterans Affairs spinal cord injury centers: Heuristic evaluation and design recommendations. In: S. Yamamoto & H. Mori (Eds.), Human Interface and the Management of Information: Proceedings of the 22nd International Conference of Human-Computer Interaction. (pp. 23 – 38). Cham, Switzerland: Springer.
Anvari, S.S., Hancock, G.M., Mok, N.B.*, Ayvazyan, A.*, Machado, C. L.*, Chompff, R.M.E.*, McCoy, K.M.*, Nare, M.T.*, Shiraiwa, Y.*, Mizushima, Y.*, & Morales, N.* (Submitted). Interface design for users with spinal cord injuries & disorders: An interdisciplinary research program with the US Department of Veterans Affairs. Paper submitted for presentation at the 21st Triennial Congress of the International Ergonomics Association; Vancouver, Canada. June.