Fast structured illumination microscopy via
deep learning
CHANG LING,
1
CHONGLEI ZHANG,
1,2
MINGQUN WANG,
1
FANFEI MENG,
1
LUPING DU,
1,3
AND
XIAOCONG YUAN
1,4
1
Nanophotonics Research Center, Shenzhen Key Laboratory of Micro-Scale Optical Information Technology &
Institute of Microscale Optoelectronics, Shenzhen University, Shenzhen 518060, China
2
e-mail: clzhang@szu.edu.cn
3
e-mail: lpdu@szu.edu.cn
4
e-mail: xcyuan@szu.edu.cn
Received 11 May 2020; revised 15 June 2020; accepted 15 June 2020; posted 15 June 2020 (Doc. ID 396122); published 21 July 2020
This study shows that convolutional neural networks (CNNs) can be used to improve the performance of struc-
tured illumination microscopy to enable it to reconstruct a super-resolution image using three instead of nine raw
frames, which is the standard number of frames required to this end. Owing to the isotropy of the fluorescence
group, the correlation between the high-frequency information in each direction of the spectrum is obtained
by training the CNNs. A high-precision super-resolution image can thus be reconstructed using accurate data
from three image frames in one direction. This allows for gentler super-resolution imaging at higher speeds and
weakens phototoxicity in the imaging process.
© 2020 Chinese Laser Press
https://doi.org/10.1364/PRJ.396122
1. INTRODUCTION
Fluorescence microscopy is an important tool in the life scien-
ces for observing cells, tissues, and organisms. However, the
Abbe diffraction limit [1] implies that the spatial resolution of
the fluorescence microscope can attain only half the wavelength
of incident light. Recently developed techniques in microscopy,
such as stochastic optical reconstruction microscopy (STORM)
[2,3], photoactivated localization microscopy (PALM) [4,5],
structured illumination microscopy (SIM) [6,7], stimulated
emission depletion (STED) [8,9], and other super-resolution
microscopy [10–12] can help overcome this limit to enable
the imaging of biological processes in cells at higher resolution.
Owing to its low phototoxicity and high frame rate acquis-
ition, SIM stands out among these techniques to achieve optical
super-resolution in bio-imaging [13]. In general, SIM enhances
resolution by encoding high spatial frequencies of the sample in
structured patterns (typically sinusoidal to affect the formation
of the Moiré pattern). By measuring the frequency of the Moiré
pattern in the observed image and the known frequency of the
pattern of illumination, the unknown frequency content of the
specimen can be computed. In linear SIM, it is theoretically up
to twice the frequency limit, which is imposed by the optical
transfer function (OTF) of the optical system. In nonlinear
SIM [14], by the use of the nonlinear effect of fluorescence,
it could reach more times the frequency limit.
To compute unknown frequencies from raw data, SIM
requires three images with shifting illumination patterns to
separate mixed spatial frequencies along a given orientation.
To enhance isotropic resolution, this process is performed three
times with illumination patterns obtained at different angles
and requires a total of nine raw images per super-resolved (SR)
SIM image, which means that the sample needs to be repeat-
edly exposed. Thus, reducing number of raw images in SIM
reconstruction has been researched in recent years. SR image
reconstruction using three [15–17] and four [18] raw frames
of structured illumination (SI) has been implemented to in-
crease the speed of acquisition of the images and reduce photo-
toxic effects. But these methods require assumptions about the
process of formation of the image, and the final results are lim-
ited by the imaging environment and type of noise. For exam-
ple, in the deconvolution method [19], this requires a precise
understanding of the optics and well-characterized noise-related
statistics. This has led to the design of such popular algorithms
as the joint Richardson–Lucy deconvolution [18,20], which
requires knowledge of the point-spread function of the micro-
scope and assumes Poisson noise statistics to estimate missing
information in SIM. However, such algorithms are limited by
the accuracy of their assumptions and thus cannot capture the
full statistical complexity of microscopic images.
Machine learning [21] has been used more commonly in
recent years with advances in computational performance.
The core concept of machine learning is to find a rule to realize
a correlation between the input and the output. This process is
carried out using a large amount of tagged data. Deep learning
1350
Vol. 8, No. 8 / August 2020 / Photonics Research
Research Article
2327-9125/20/081350-10 Journal © 2020 Chinese Laser Press