Learning to See in the Dark
Chen Chen
UIUC
Qifeng Chen
Intel Labs
Jia Xu
Intel Labs
Vladlen Koltun
Intel Labs
(a) Camera output with ISO 8,000 (b) Camera output with ISO 409,600 (c) Our result from the raw data of (a)
Figure 1. Extreme low-light imaging with a convolutional network. Dark indoor environment. The illuminance at the camera is < 0.1
lux. The Sony α7S II sensor is exposed for 1/30 second. (a) Image produced by the camera with ISO 8,000. (b) Image produced by the
camera with ISO 409,600. The image suffers from noise and color bias. (c) Image produced by our convolutional network applied to the
raw sensor data from (a).
Abstract
Imaging in low light is challenging due to low pho-
ton count and low SNR. Short-exposure images suffer from
noise, while long exposure can induce blur and is often
impractical. A variety of denoising, deblurring, and en-
hancement techniques have been proposed, but their effec-
tiveness is limited in extreme conditions, such as video-rate
imaging at night. To support the development of learning-
based pipelines for low-light image processing, we intro-
duce a dataset of raw short-exposure low-light images, with
corresponding long-exposure reference images. Using the
presented dataset, we develop a pipeline for processing
low-light images, based on end-to-end training of a fully-
convolutional network. The network operates directly on
raw sensor data and replaces much of the traditional im-
age processing pipeline, which tends to perform poorly on
such data. We report promising results on the new dataset,
analyze factors that affect performance, and highlight op-
portunities for future work.
1. Introduction
Noise is present in any imaging system, but it makes
imaging particularly challenging in low light. High ISO can
be used to increase brightness, but it also amplifies noise.
Postprocessing, such as scaling or histogram stretching, can
be applied, but this does not resolve the low signal-to-noise
ratio (SNR) due to low photon counts. There are physi-
cal means to increase SNR in low light, including opening
the aperture, extending exposure time, and using flash. But
each of these has its own characteristic drawbacks. For ex-
ample, increasing exposure time can introduce blur due to
camera shake or object motion.
The challenge of fast imaging in low light is well-
known in the computational photography community, but
remains open. Researchers have proposed techniques for
denoising, deblurring, and enhancement of low-light im-
ages [34, 16, 42]. These techniques generally assume that
images are captured in somewhat dim environments with
moderate levels of noise. In contrast, we are interested in
extreme low-light imaging with severely limited illumina-
tion (e.g., moonlight) and short exposure (ideally at video
rate). In this regime, the traditional camera processing
pipeline breaks down and the image has to be reconstructed
from the raw sensor data.
Figure 1 illustrates our setting. The environment is ex-
tremely dark: less than 0.1 lux of illumination at the cam-
era. The exposure time is set to 1/30 second. The aperture
is f/5.6. At ISO 8,000, which is generally considered high,
the camera produces an image that is essentially black, de-
spite the high light sensitivity of the full-frame Sony sen-
sor. At ISO 409,600, which is far beyond the reach of most
cameras, the content of the scene is discernible, but the im-
age is dim, noisy, and the colors are distorted. As we will
show, even state-of-the-art denoising techniques [32] fail to
remove such noise and do not address the color bias. An
alternative approach is to use a burst of images [24, 14], but
1