Deep Learning for Enhanced Scratch Input , ,
and for use in harsh environments where conventional computer interaction methods are infeasible (e.g., front-line
workers in biohazardous and other extreme environments).
1.2.1 Home Automation. The proposed system could readily be used in a variety of home automation use cases.
High-accuracy user input on a relatively small set of gestures is readily applied to control tasks in home automation
[Villanueva and Drögehorn 2018]. Advantages of the proposed scratch input system include the rapid deployment
of extremely large, low-cost interface surfaces and environments. With several low-cost microphones or networked
mobile devices, any space could be converted into an interactive, controllable, and responsive environment.
Since any scratching or tapping gesture can be used for the system that consistently diers from the rest acoustically,
gestures could also be made intuitive and language-independent. For example, the
vertical scratch
and
circular
scratch
gestures, resembling a binary ‘1’ (on) and ‘0’ (o) respectively, may be used to turn on and o a lamp. Similarly,
the
W-scratch
, resembling iconographic depictions of water [Siegeltuch 2017], may be used for faucets, ushing, or
other related functions.
When compared with existing commonly sold voice assistants for home automation, the proposed system also stands
out in terms of privacy. All computation for gesture classication and noise discrimination could easily be implemented
onboard with a low-cost microprocessor or system on a chip. Meanwhile, voice assistants today are nearly universally
“cloud based”, entailing a host of security concerns [Lei et al
.
2017]. To execute commands, voice recordings are recorded
and sent to a remote system (“cloud”) for processing before execution. In production, the proposed deep learning-based
scratch user interface system could easily separate the gesture classication system from the networked components,
drastically increasing user privacy and security.
1.2.2 Accessibility. As addressed in [Harrison and Hudson 2008], the majority of computer systems in common use
today are relatively small for ease of transportation (e.g., mobile phones, laptops). Even in the home, where it may be
possible to deploy a larger scale user interface system, it is often inconvenient and prohibitively expensive. This problem
is especially important for those people aected by disabilities relating to pain, exibility, and dexterity. According to
a 2012 Canadian Survey on Disability, over 25% of those reported to have some type of disability were identied as
having dexterity disability that limited their daily activity [Christine Bizier 2012].
A natural user interface (NUI) system that could easily expand to the size of a table could help to ameliorate these
limitations on daily activity as they relate to the use of ubiquitous computing systems. As will be discussed later, the
proposed system could also be congured to continuously learn and improve on a user’s data, enabling adaptation
to each individual’s accessibility needs. Since the proposed system was validated and tested with a wide variety of
consumer smartphone and tablet microphones, it would be rapidly deployable and very likely low-cost for the end user
given the widespread use of smartphones.
1.2.3 Hazardous Environments. Given the proposed system’s ease of use for system control tasks like home automation
in with little to no reliance on dexterity, it is logical to propose its use in extreme environments. In areas of extreme cold,
for instance, thick protective gear must be worn that severely inhibits manual dexterity. While conventional computer
system input devices may be dicult to use, scratches and taps remain easy to replicate. A set of gestures optimized
for use with protective equipment could be easily designed and implemented with the following system, aiding in the
maintenance of safety and eciency as safety equipment would need not be removed to use the system.
3