Experimental assessment of a 3-D plenoptic endoscopic
imaging system
Hanh N. D. Le
1,
*, Ryan Decker
2
, Axel Krieger
2
, and Jin U. Kang
1
1
Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore,
MD 21218, USA
2
Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Health System,
Washington, DC 20010, USA
*Corresponding author: hle18@jhu.edu
Received October 10, 2016; accepted January 24, 2017; posted online March 1, 2017
An endoscopic imaging system using a plenoptic technique to reconstruct 3-D information is demonstrated and
analyzed in this Letter. The proposed setup integrates a clinical surgical endoscope with a plenoptic camera to
achieve a depth accuracy error of about 1 mm and a precision error of about 2 mm, within a 25 mm × 25 mm
field of view, operating at 11 frames per second.
OCIS codes: 170.2150, 170.4580, 110.6880, 120.3890.
doi: 10.3788/COL201715.051701.
Endoscopic imaging provides visualization in minimally in-
vasive surgery and helps reduce the trauma associated with
open procedures
[1]
. Recent advances in 3-D peripheral
reconstruction for endoscopy allow for better tissue assess-
ment and objective risk evaluation by surgeons, and help
enhance autonomous control in robotic surgery
[2–4]
. These
advances have enabled real-time surface reconstruction
during surgery, and include stereoscopy, time-of-flight
(ToF), structured illumination, and plenoptic imaging.
To achieve realistic 3-D surface reconstructions of a surgi-
cal site in a minimally invasive surgical setting, a 3-D endo-
scopic system should provide high precision and accuracy
within the desired field of view (FOV), and a frame rate
adequate for observation and navigation by surgeons.
Stereoscopy uses a passive wide-field illumination and
acquires two images of an object from two viewing angles
to reconstruct depth information via disparity searching.
The depth resolution ranges from 0.05 to 0.6 mm
[5,6]
, which
can be achieved with enhanced searching algorithms
[7–9]
.
However, this requires sufficient spatial offset between
the two views to achieve a high depth resolution.
ToF techniques measure the differences in phase and in-
tensity of time or frequency modulated laser pulses. Depth
information can hence be reconstructed with low compu-
tational cost based on the active light modulation infor-
mation. However, depth resolution based on ToF is
relatively poor from 0.89 to 4 mm
[10,11]
. In addition, this
technique often suffers from systemic errors in camera
temperature tolerance and varying exposure time. Other
impact factors include biological optical properties from
studied samples such as absorption and scattering coeffi-
cients that can change with respect to the incoming light
source and ray angle
[12]
.
The structured illumination technique is classical in 3-D
reconstruction, with its principle based either on disparity
searching, similar to the stereoscopy technique
[13]
,oron
the reconstructed phase information
[14,15]
from multiple
artificial light patterns on the sample. Structured illumi-
nation can achieve a very high depth resolution, up to
0.05 mm
[16]
, and has been employed in several medical ap-
plications
[17–20]
. However, it requires an active pattern pro-
jector for light modulation that cycles between different
pattern illuminations, leading to complications in camera
calibration and necessitating a high-power light source
[21]
.
In addition to the three techniques described above, ple-
noptic imaging is a fairly new 3-D reconstruction tech-
nique in the biomedical field. The technique involves a
microlens array (MLA) integrated onto an imaging sensor,
such that each point of the object can be viewed and im-
aged at different angles via adjacent microlenses. The
depth information can be deduced similarly to the stereos-
copy approach. However, in stereoscopy, the two imagers
should maintain a set angle of separation to obtain two
distinct views of the object while ensuring the desired
depth accuracy
[22–24]
. A stereoscope utilizes triangulation
in which the two imagers maintain a set angle relative
the object for correspondence searching. As an extension
of the stereoscope, a plenoptic imager utilizes only one sen-
sor with multiple micro lenses to create a higher number of
viewing positions, and thus improve the correspondence
searching performance. Moreover, a plenoptic camera of-
fers the reduction of systematic calibration due to the
known separation between each microlens (i.e., known dis-
tance between microsensors). In addition, plenoptic imag-
ing also creates the expansion of parallax computation in
both horizontal and vertical directions compared to one
dimension in a stereoscope. Currently, plenoptic imaging
is widely accepted in industry for multifocus imaging
[25,26]
;
however, plenoptic imaging in medicine is limited with a
current depth precision of 1 mm
[27,28]
in wide-field imaging.
To adapt this technique to an endoscopic setting for min-
imally invasive surgery, we propose a plenoptic endoscopy
design that consists of a clinical surgical endoscope, a ple-
noptic camera, and a relay optical system. The proposed
COL 15(5), 051701(2017) CHINESE OPTICS LETTERS May 10, 2017
1671-7694/2017/051701(5) 051701-1 © 2017 Chinese Optics Letters