J Control Theory Appl 2009 7 (1) 14–22
DOI 10.1007/s11768-009-7218-z
Eye movement prediction by oculomotor plant
Kalman filter with brainstem control
Oleg V. KOMOGORTSEV
1
, Javed I. KHAN
2
(1.Department of Computer Science, Texas State University-San Marcos, San Marcos, Texas, USA;
2.Department of Computer Science, Kent State University, Kent, Ohio, USA)
Abstract: Our work addresses one of the core issues related to Human Computer Interaction (HCI) systems that use
eye gaze as an input. This issue is the sensor, transmission and other delays that exist in any eye tracker-based system,
reducing its performance. A delay effect can be compensated by an accurate prediction of the eye movement trajectories.
This paper introduces a mathematical model of the human eye that uses anatomical properties of the Human Visual System
to predict eye movement trajectories. The eye mathematical model is transformed into a Kalman filter form to provide
continuous eye position signal prediction during all eye movement types. The model presented in this paper uses brainstem
control properties employed during transitions between fast (saccade) and slow (fixations, pursuit) eye movements. Results
presented in this paper indicate that the proposed eye model in a Kalman filter form improves the accuracy of eye move-
ment prediction and is capable of a real-time performance. In addition to the HCI systems with the direct eye gaze input,
the proposed eye model can be immediately applied for a bit-rate/computational reduction in real-time gaze-contingent
systems.
Keywords: Eye movement prediction; Bio engineering; Human computer interaction
1 Introduction
There has been a substantial amount of research in the
HCI community that investigated the use of the eye gaze
information as a primary or auxiliary input to computer sys-
tems [1, 2]. This research indicates that such input is es-
pecially beneficial for target selection due to the following
reasons: a) people look at a target prior to selecting it with
an input device. Therefore, when an eye gaze is used for
selection, the selection delay introduced by an auxiliary in-
put device such as a mouse is eliminated [1]. b) Because
of the fact that the eye globe rotates in a fluid inside of an
eye socket, the eye has the capability of moving much faster
than the limbs that are burdened by bone weight. Addition-
ally, the fibers inside of the extraocular muscles are fatigue
resistant within certain limits of the eye movement ampli-
tude [3]. It is noteworthy that limb muscles do not possess
such properties, hence repetitive limb movements cause fa-
tigue and excess of such movements might cause repetitive
stress injury. c) The eye can provide an additional channel
of input in situations where the use of limbs is restricted or
unavailable, such as surgeries where both hands are occu-
pied (laparoscopy), user interfaces for the handicapped, etc.
Despite all the benefits there are some challenges that
limit eye gaze input technology, i.e., cost, accuracy, sensor
and transmission delays [2].
Our work specifically concentrates on compensation of
sensor and transmission delays. Our hypothesis is that de-
lay compensation will allow the creation of more responsive
HCI systems and provide higher level of compression in
gaze-contingent systems. The delay compensation approach
we selected is based on prediction of future eye movements.
If the future eye gaze location is predicted accurately, tar-
get selection can be pre-made, therefore increasing the re-
sponsiveness of the HCI system with direct eye gaze input.
In a gaze-contingent compression system the accurate eye
movement prediction will allow to minimize the size of the
high quality coded Region of Interest (ROI), therefore re-
ducing overall bandwidth or computational requirements [4,
5].
The challenge of the accurate eye movement prediction
lies in the fact that the Human Visual System (HVS) ex-
hibits a variety of eye movement fixation, saccades, smooth
pursuit, optokinetic reflex, vestibulo-ocular reflex, and ver-
gence [6]. In this paper, we concentrate on the first three due
to the observation that these eye movements are exhibited
when a person works with a computer. With great simplifi-
cation, their roles are described as follows: 1) fixation – eye
movement that keeps an eye gaze stable in regard to a sta-
tionary target providing visual pictures with highest acuity,
2) saccade – very rapid eye rotation moving the eye from
one fixation point to another, and 3) pursuit stabilizes the
retina in regard to a moving object of interest.
In our previous work [7] we created the Two State
Kalman Filter (TSKF) eye movement prediction model. The
TSKF model assumes that an eye has two states, position
and velocity. Our tests indicate that the TSKF allows accu-
rate eye movement prediction during fixations and pursuits
but has poor performance during saccades. To improve the
accuracy of the eye movement prediction during saccades,
we have developed an Oculomotor Plant Mechanical Model
(OPMM) [8, 9]. This model mimics eye anatomy by con-
sidering the physical properties of the extraocular muscles
and the eye globe. The OPMM has six states that represent
eye position, velocity, muscle location and muscle forces.
Our model is based on Bahil’s model [10] with two major
additions: the ability to start a saccade from any point in a
Received 18 September 2007; revised 29 May 2008.