Defining and Analyzing a Gesture Set
for Interactive TV Remote on Touchscreen Phones
Yuntao Wang
1
, Chun Yu
1
, Yuhang Zhang
1 2
, Jin Huang
1
, Yuanchun Shi
1 3
1
Key Laboratory of Pervasive Computing, Ministry of Education,
Tsinghua National Laboratory for Information Science and Technology
1
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
2
Information Science Department, Cornell University, Ithaca 14850, NY, USA
3
Department of Computer Technology, Qinghai University, Xining, 810016, China
E-mail: {w.yuntaosea, yc2pcg}@gmail.com, yz769@cornell.edu, huangjin8808@gmail.com, shiyc@tsinghua.edu.cn
Abstract—In this paper, we recruited 20 participants
preforming user-defined gestures on a touchscreen phone for
22 TV remote commands. Totally 440 gestures were recorded,
analyzed and paired with think-aloud data for these 22
referents. After analyzing these gestures according to extended
taxonomy of surface gestures and agreement measure, we
presented a user-defined gesture set for interactive TV remote
on touchscreen phones. Despite the insight of mental models
and analysis of gesture set, our findings indicate that people
prefer using single-handed thumb and also prefer eyes-free
gestures that need no attention switch under TV viewing
scenario. Multi-display is useful in text entry and menu access
tasks. Our results will contribute to better gesture design in the
field of interaction between TVs and touchable mobile phones.
Keywords- User-elicitation methodology, remote control,
user-defined gesture, eyes-free interaction, touchscreen.
I. INTRODUCTION
TV, the media center of smart space at home, had
become a terminal for interactive applications such as online
shopping and media display. With the development of
interactive TVs, traditional push-button based remote
controls were not flexible enough to meet the complicated
demands of these applications. New remote control
technologies based on touchscreen [3, 4, 5] have been
designed and evaluated by many studies. However, these
researches focused on the efficiency of input (e.g. pointing).
As smarts phones gradually turned into the most
pervasive computing devices interacting with smart
environments [2], many applications use smart phones as the
control device for interactive TV. Apple Remote [1] presents
a 7 soft buttons design supporting tap and flick on iPhones
and supports screen sharing between TV and the smart phone.
Samsung Remote [9] and UnityRemote [10] are designed as
soft-key-based TV remotes supporting relative position
control on the touchscreen. But there is nearly no study about
the user-centered gesture design on touchscreen phones for
TV control.
In this study, we adopted user-elicitation methodologies
[6, 7, 8, 11] to define and analyze a user-defined gesture set
for interactive TVs based on touchscreens. Firstly we
summarized three basic interaction tasks on interactive TV:
navigation, text entry and shortcuts obtaining 22 referents.
Then following the user-elicitation methodology, 440 user-
defined gestures on touchscreen phones from 20 participants
were recorded and classified according to the extended
taxonomy of 5 dimensions: form, nature, binding, flow and
attention. After analyzed using agreement measure [11], 34
gestures were mapped to 22 referents with minimum
consensus-threshold of 4.
Finally, despite of the insight of mental model, we
presented some findings of the gesture vocabulary and eyes-
free gesture design in discussion section. Our results
contribute to a user-defined gesture set and insight into
users’ mental models and also eyes-free gesture design for
remote control on touchscreen phones. This work will help
designers with their future design in the field of TV and
phone interaction.
II. D
EVELOPING A USER-DEFINED GESTURE SET
A. Understanding the Tasks on Interactive TV
After exploring the functions of interactive TV including
playing movie and music, searching the Internet and so on,
we obtained the common commands as Table 1 shows.
There were three basic tasks including navigation, text entry
and shortcut, and in total we obtained 22 referents. Other
commands such as copy and paste were not considered
because of the low usage frequency. But they can be
navigated after “Show menu” command performs. We
evaluated the coverage of these referents with the UI
elements and commands of Samsung smart TV, Apple TV,
Google TV and XBMC. It proved that this referent set
include all the referents for TV control.
After obtaining the basic tasks and referents, we designed
a lab-based user study following the user-elicitation
methodology mentioned by Wobbrock et al. [11].
B. Participants
20 paid participants (age ranges from 20 to 40, M=26.7;
6 females) with different educational background
volunteered for user-elicitation design (12 students from
different disciplines: computer science (6), art (3), finance (2)
and biology (1); and 8 staffs from different industries: IT (3),
education (2), industrial design (2) and music (1)). All of
them had the experience of touchable mobile devices.