Winged - a leap motion drawing app for users with hand and arm disabilities

Solo project

Roles performed:

  • User experience researcher
  • User interface designer

Tools used

  • Figma
  • Miro
  • Paper prototyping

Executive summary

Winged aims to provide a minimalistic interface for users with hand and arm impairement to draw with minimal pain and discomfort

Problem statement

Disabilities have long had negative influence on adults emotionally and physically

There are a lot of disabled artists out there who want to be able to draw. However, most of them use other parts of their bodies (such as their mouths and legs), which will greatly cause pain and discomfort.

Based on the concept of art therapy, art is a means for users to feel the freedom to express themselves and create something new. It is especially helpful when users are bound by illnesses and feel less in control of their health.

“Art provides unlimited possibilities for personal, academic, and professional success.” - Christopher and Diana Reeve Foundation. Winged promotes inclusivity in the arts and help artists achieve their goals, either for leisure or serious career pursuit

Disabled Artists in the News
Source: Mouth and Foot Painting Artists

Audience

Our audience are those with arm or hand disabilities, unlimited in age or gender

Our user testing participants, however, are my friends and acquaintances, who have little to no hand and arm disabilities. I'm working on reaching out to more users in the target audience, so if you happen to know someone or have any recommendation in doing so, please do not hesitate to contact me at trang.le@berkeley.edu. I'd really appreciate it!

Challenges and How Might We (HMWs)

There has been research on a software that utilizes the leap motion controller to enable users to draw using hand gestures in midair (researched on by Dr. Chris Creed at University of Birmingham), but the interface in the app was too complicated for users to navigate.

Winged's main challenge is to create an interface that is easy for navigate for different physiological needs.

  1. HMW help the users navigate the interface with minimal discomfort and muscle pain?
  2. HMW create the user interface (UI) as minimalistic as possible?
  3. HMW make the options and the icons easier to click?
  4. HMW move the canvas to adjust to the users’ hand placement?
  5. HMW create independence for the users when they use our app?
  6. HMW help users boost their creativity, and more importantly, their self-esteem?

In this case study, I will propose a solution for HMWs 1-4, and leave 5) and 6) for further research and future iterations

Similar to Dr. Creed's research, we utilize the leap motion controller as a tool for users to draw.

The leap motion controller
Source: https://www.ultraleap.com/tracking/

Approach

Our approach to addressing the HMWs are hand detection, recommending a suitable wheelchair setup, voice command, and a minimalistic design

  • Minimalist interface
  • Hand gestured detected by the leap motion controller
  • Optimized wheelchair setup of the leap motion controller for each user
  • Detect users’ hand parts to customize to physiological structures
  • Voice command to minimize hand gestures and thus pain

Design Process

User Journey

The main pain point we detect from the user journey is when users get used to the app. Much physical discomfort in the beginning would turn the users away from using the app, so we want the onboarding phase goes as smooth and accommodating as possible

User flow

I created a user flow and identify where we can solve the pain point for users. The highlight of the userflow where the solution takes place is the constant feedback loop in which users can go back and forth and adjust the settings suitable to their needs.

The highlight of the userflow created in Miro

Wireflow

To visualize the interface I want to build, I first created a wireflow to imagine how users would adjust settings. But these questions arise: 1) Can we recommend a setting for users during onboarding? and 2) How do we do so for users with various hand disabilities?

Arising issues

One arising issue I encountered while designing the interface is that different hand physiologies will need different accommodations and setups. When the users' hands or arms are not exactly the same as the hands with the full number of joints and parts, we would have to consider ways to include them in our app as much as possible.

One example of user's hand physiology compared to a hand with full joints
The leap motion hand hierarchy.

Another arising issue is how to perform user testing and design iteration. As our app is interactive in the 3D space, we cannot simply test it as with a clickable prototype. It is hard for users to imagine how they would interact with the app on the 2D space, and spending too much time coding an interface that is way too elaborate and subject to many drastic changes would cost a lot of time.

Instead of going from wireframe to digital prototyping and then coding, for this app, putting each fidelity level into code will be necessary to provide the most genuine user testing experience.

Solutions

1. Recommend the best device setup for users based on their hand structures

My proposed solution is that we should have the onboarding screens to detect users' hand joints and parts that are ready for drawing.

By having the users completing some simple exercises (clicking on a button, moving their hands to a designated location), we might be able to detect which parts of their hands users are comfortable with using and moving.

2. Give users freedom to setup the interface at their convenience

While doing user testing, I realized that having the buttons placed on users' fingertips are not at all optimal. At first, I only took in consideration users who have paralyzed arms but can still move their fingers. However, this approach, even with this small subset of users, is confusing in and of itself. It was not clear to the participants I tested with that the buttons are on the tip of their fingers. Moreover, this design has a huge flaw, in which it does not provide inclusivity for various disabilities.

Many user testing participants commented that it would be easier for them to just use whichever hand or finger they want to click on buttons and navigate the interface. We give users freedom to customize the interface and interact with the app at their convenience, and still have our recommendation to support them.

3. Voice commands for users to set up devices and choose drawing options

When users start out using the app, we would not know about users' disabilities and how their hands or arms or structured. To limit this initial bulkiness for users, we could guide them through using voice commands during setup to let us know about their disabilities and choosing a suitable device.

Not just that, voice command also aides users to make drawing options without having to physically click on buttons. They can tell us which color they want to pick or which brush they would use, not dissimilar to Siri's usage.

The interface is good. Are we good to go? Not really.

4. Suitable wheelchair setup to minimize users' movements

To adapt to different users' ergonomic conditions, we need to think about a setup that is adjustable for different disabilities.

I asked one user (who has no disabilities) that "how would you go about using a drawing app if your right arm is injured by rockclimbing?" (she loves rockclimbing and is righthanded). "I would use my left arm to move the paper around and my right arm to draw, so my injured arm does not have to move around as much," she answered.

This answer inspired the fourth solution, which is setting up the leap motion controller on a wheelchair at a suitable location.

While playing around with the leap motion controller, I realized that as I moved my controller around with one hand and keep the detected hand still, the detected hand would be moving around on the screen, reflecting the "moving the paper around" idea that our participant gave.

Thus, having a wheelchair with the controller attached and moveable at the click of a button would give users a lot of freedom to make use of the canvas despite limited arm range.

Users with arthrogryposis, for example, would have limited arm range movement. "Moving the canvas around instead of the hands" is a potential solution for this group of users, and I believe it can benefit users with other conditions and illnesses.

Luckily, Winged also took into account that users might need to be oriented with this setup when they started using the app.

Six steps to detect users' hand structures


Our recommended setup after detecting users' hand structures

Here is Winged's approach together with approaches that have been done by various engineers in this effort!


Winged's approach (top), other approaches from leap motion gallery (bottom)

Hardware constraints

Shout out to the brilliant hardware engineers who have worked on hover gesture technologies. I would like to be mindful of my design's feasibility with hardware capability, and have noted some constraints:


Field of View and depth of view illustration for leap motion controller. Source: ResearchGate
  • There is a constraint to the field of view (FoV) of the leap motion device. Currently, the FoV of the device is typically 150×120°.
  • The constraint for the depth (vertical interaction) is between 10cm to 60cm preferred, up to 80cm maximum

Summary - Takeaways

One of the main challenges of the app is to adapt to different disabilities to increase inclusivity towards our users.

Some of the key solutions include:

  • Experimenting with different different device placements on wheelchair to better suit users' handicap
  • Detect users' hand joints and bone structures to recommend a setting for them
  • Using voice command to minimize hand interactions and smoothen the onboarding process
  • Let users customize button locations in whichever way that is convenient for them

Further work

I have identified the following tasks to be done to move this project forward:

  • As some users suggested, learning from existing assistive technologies and gesture settings of different drawing apps will inform us of the settings/interactions that users have been acquainted with
  • Conduct more research into different disabilities and physiological structures of those with hand impairement

© Copyright 2021 Trang Le. Made with Gatsby.js💜