This was a conceptual design of an app to allow users to use their phone to watch media apps such as YouTube or Twitch without the need to touch their phone. This played with the concept of using machine learning (ML) to track the hands and the eyes of the user so they can perform hand gestures. This was an HCI (Human Computer Interaction) project for my honours I had to do with a group.

The prototype can be accessed here:

Note that this may require PC since it makes use of the keyboard to simulate the hands-free feature, so it is ideal to have two people to use it (one as the Tester and the other acting as the system).

Problem Space

It is a well-known habit that a lot of people recently do other miscellaneous activities while watching videos or interacting with their devices to watch videos. From academic literature, we had how people have a massive habit of using their screen devices while consuming food, to which screen time and the front camera show evidence that it is possible to determine times when one is eating or not while using the device.

User Personas

Functional requirements

The main purpose of the app was to provide our users with the ability to use simple hand gestures to control a video stream on their device when their hands may be preoccupied with another task or simply just dirty from eating food. This needed to be achievable by displaying a video stream to the user from any platform of their choice (e.g. YouTube, Twitch, Netflix, etc.).

The device's built-in front-facing camera is used to detect hand gestures that control basic functions of the video streaming app such as pausing, rewinding, volume up/down, and fast forwarding.

In addition, the app would have to interpret the user's hand gestures to what has been coded into the product, in other words, the hand gestures have to correspond to the correct action.

Design Pages

Through various ideation using SCAMPER method, we had come up with some prototypes that eventually led us to having an app that acts as a widget which is drawn over the media app to be used. We determined that this would allow users to make use of the media app without a large learning curve, along with keeping the visibility of the app while being able to keep track of visibility of system status and to know when to use gestures.

A splash page was also designed to simulate creating an account to store personal information from tracked data and preferences, as well as navigation from the interface to other media apps.

Testing

The prototype was designed in such a way that keyboard bindings were made to simulate the hands-free interaction such that the tester would use the prototype while the system actor has a hidden keyboard to perform actions while observing the user. The keyboard is hidden to make sure the experience is closer to the actual system and the prototype will be evaluated based on its design and experience only.

Evaluation

Through in-depth analysis from sources in academic literature that involved hand gesture technology and mobile app design, we had taken the app through heuristic evaluation, making use of these aspects as our heuristic evaluation set: Learnability, Cognitive load, Efficiency, Error prevention and Social Acceptability. The evaluators were put through a series of tasks were they would role play as each user persona going through a specific flow of the prototype.