嵌入式设备图像处理详解：从CFA到显示

4星 · 超过85%的资源需积分: 50 114 浏览量更新于2024-07-16 2 收藏 49.88MB PDF 举报

《嵌入式设备图像处理》是一本深入探讨相机技术和图像/视频编码技术的专业指南，由Sebastiano Battiato、Arcangelo Ranieri Brunà、Giuseppe Messina和Giovanni Puglisi共同编著。该书针对嵌入式系统设计者和工程师，提供了从原始数据（如RAW格式）到最终图像显示的详细技术流程。它涵盖了图像传感器、信号处理、数据压缩、图像编码和解码、以及在资源受限的嵌入式设备上实现高效图像处理策略的关键知识点。书中首先介绍了相机技术的基础，包括传感器的工作原理，如CMOS或CCD阵列如何捕捉光线并转化为数字信号。这部分内容着重于硬件与软件交互，强调了对噪声抑制、曝光控制和色彩校正等基本概念的理解。接着，章节转向图像数据的预处理阶段，如去噪、白平衡调整、色彩空间转换，这些都是为了优化原始数据以便后续处理。这一部分还可能涉及图像增强和特征提取，以提高图像质量或为特定应用提取有用信息。随着数据处理的深入，编码技术成为关键环节。书中会讨论无损和有损压缩算法，如JPEG、H.264等，以及它们在嵌入式设备上的性能和效率。编码过程的目标是减小数据量，便于存储和传输，同时尽可能保持图像的视觉质量。图像编码后，解码器将这些压缩数据还原为可供显示的图像。作者会详细阐述如何在有限的内存和计算能力下实现高效的解码算法，并可能包括硬件加速技术的应用。最后，书中的内容还会涉及图像显示技术，包括驱动接口、显示器类型和色彩管理系统，确保图像能在各种嵌入式平台上准确呈现。此外，还可能讨论实时性和功耗优化，因为这些因素在嵌入式设备上尤其重要。整体而言，《嵌入式设备图像处理》是一本实用的参考书籍，适合那些需要在资源受限的环境中进行图像处理工作的人士，如无人机开发者、智能家电制造商、以及物联网设备的设计师。通过阅读本书，读者可以深入了解如何在实际应用中克服挑战，实现高效、低功耗的图像处理解决方案。

(a) (b)

Figure 1.3 : An enlarged detail of Fig.(1.2)(a) acquired by the monochromatic sensor

and the corresponding RGB image (b) obtained at the end of the pipeline.

it may be affected by noise (i.e., any kind of not actual information wrongly converted

as useful information). Usually noise level is limited and not dangerous until the actual

signal is adequate and signiﬁcantly greater, i.e., high level of Signal to Noise Ratio (SNR).

To guarantee this fundamental principle each photosite (pixel) must be conﬁgured so

that it acquires the correct level of light and thus the correct level of signal: varying the

light intensity of the scene there must be a way to change the capability of the sensor to

correctly and properly store in its cell the right level of light. This control is performed by

the integration time. It represents the time during which the photo-element is acquiring

and converting light into electrical charge. The lower the light intensity of the scene

the higher the integration time. By changing this integration time a given scene digital

acquisition can be under-exposed (too dark, too short integration time), over-exposed (too

lit, too long integration time) or correctly exposed.

Two cases must be avoided or considered extreme cases: no accumulation, which cor-

responds to black, and over-accumulation (also known as saturation) which corresponds

to extreme light or white. For actual black or white it is correct that the pixel assumes

these values but they can also come out from a bad exposure (black from under-exposition

and white from over-exposition). Also, there is no way to control integration time sep-

arately for each pixel of the sensor and this means that all the pixels of the sensor are

exposed with the same integration time, although frame by frame it may change to adapt

to variations of light which may occur in real scenes. Usually the integration time value

is chosen so that the mean brightness of a picture is around the mid-range of the possible

values (e.g., for a 8 bit per pixel image there are 256 different light levels and a correct

exposed image has a mean brightness of 128).

Finally, it is not always possible to select an appropriate integration time for each

scene. Too long or short integration time are not feasible because other problems may

occurs and affect the SNR (for details see Chapters 2, 3 and 6). Also, integration time

may be lower-limited by the framerate and/or by a safe value which aims to reduce motion

4 Image Processing for Embedded Devices Battiato et al.

blur effects. Motion blur is caused by long integration time and moving objects in the

acquisition scene or hand-shaking. It is typical in low-light acquisitions and for this reason

often a ﬂash is used in such a situation.

Each time a lower threshold limits the integration time the only way to properly read

the minimum accumulated information of the cell is to use a multiplicative gain to amplify

the information to a usable value.

In summary, a good exposure control module is compound of:

• an appropriate module which estimates the light intensity of the scene and prop-

erly settles the correspondent integration time, avoiding under-exposition or over-

exposition;

• an appropriate gain control which furnishes support and compensates the limits of

the integration time; when selecting a proper balance between integration time and

gains priority goes to the former;

• a method to identify actual black and white regions, assumed that the rest of the

identiﬁcation and proper compensation is demanded to following modules in the

image generation pipeline (see Chapter 4 for additional details);

• a loop with other modules which apply additional gains to the signals (like AWB,

see Chapter 5);

• and additional optional module to control and avoid motion blur; usually in litera-

ture this methodology goes with the name of AutoISO.

1.1.2 White Balance

One of the most challenging processes that affects perceived image quality in a digital

camera is the correct color reproduction. Human visual system is able to remove color

casts: an object appears to our eyes with the same color under different illuminant con-

ditions. On the contrary the sensor simply acquires raw data and is not able to cope

with real scene illumination variability. For instance a white paper in outdoor or indoor

environment can be recorded by the sensor with bluish or reddish colors.

In order to cope with these problems a lot of techniques have been developed. High

end cameras typically provide a variety of presets related to the most common light

sources (tungsten, ﬂuorescent, daylight, ﬂash, etc.). Moreover white balance parame-

ters can be set, for future photos, taking a picture of a known gray reference under the

same illumination source (custom white balance).

All the techniques above described need a close interaction with the user in order

to properly work. On the contrary auto white balance techniques try to guess the cor-

rect illumination properties and remove color casts without user interaction. These tech-

niques, based on strong assumptions on scene reﬂectance distribution, have been also

implemented in low cost devices (e.g., smart phone) and will be in depth described later

in Chapter 5.

Fundamentals and HW/SW Partitioning Image Processing for Embedded Devices



1.1.3 Noise Reduction

The perceived image quality is deeply inﬂuenced by image noise (named by analogy with

unwanted sound). These unwanted ﬂuctuations, if not properly managed, heavily degrade

image quality. Different noise sources, with different characteristics, are superimposed

to the image signal: photon shot noise, dark current noise, readout noise, reset noise,

quantization noise, etc.

Although many efforts have been done by manufacturers to reduce the presence of

noise in imaging devices it is still present and can be considered unavoidable in critical

situations. For instance, low light conditions together with low integration time, produce

very low SNR (signal to noise ratio), very few photon were captured, making really dif-

ﬁcult obtaining pleasing photos. This physical limit does not depend only on the sensor

characteristics but it is strictly related to the nature of light. Moreover the increasing of

the number of pixels and the limited size of the embedded devices, implying the decreas-

ing of the pixel size, produces further problems. Small pixels, acquires less photons with

respect to larger pixels. Less useful signal implies then noisier picture.

In order to cope with these problems smart ﬁlters must be designed. These ﬁlters

must be able to estimate image noise characteristics (e.g., mean and standard deviation

if a Gaussian model is used), and then remove unwanted noise without affecting image

details.

Finally, it should be noted that noise reduction can be performed during the vari-

ous stages of the pipeline. Some approaches works on RGB images, others directly on

Bayer data. The latter typically provides some advantages (demosaicing step typically

introduces nonlinearities that make difﬁcult noise reduction). Further details about noise

reduction algorithms will be provided in Chapter 6.

1.1.4 Demosaicing

Digital cameras, in order to reduce costs and complexity, acquire images by means of a

monochromatic sensor covered by a CFA (color ﬁlter array). A lot of CFA have been

developed but the most common is the Bayer pattern. This simple CFA, taking into ac-

count human visual system characteristics (human eyes are more sensitive to green with

respect to the other primary colors), contains twice as many green as red or blue sen-

sors. Some spatially undersampled color channels (three in the Bayer pattern) are then

provided by the sensor and the full color information is reconstructed by color interpo-

lation algorithms (demosaicing). Demosaicing is a very critical task. A lot of annoying

artifacts that heavily degrade picture quality can be generated in this step: zipper effect,

false color, moir

e effect, etc. Simple intra-channel interpolation algorithms (e.g., bilinear,

bicubic) cannot be then applied and more advanced solutions (inter-channel), both spatial

and frequency domain based, have been developed. In embedded devices the complexity

of these algorithms must be pretty low. Demosaicing approaches are not always able to

completely eliminate false colors and zipper effects, thus imaging pipelines often include

a post-processing module, with the aim of removing residual artifacts. Further details

about demosaicing algorithms will be provided in Chapter 7.

6 Image Processing for Embedded Devices Battiato et al.



1.1.5 Color Matrixing

The Color Matrix sub-system, also known as Color Calibration, aims to convert the color

response of the acquisition device to a standard color space. Usually the standard RGB

(sRGB) color space is used, according to the ITU-R BT.709 directive. This transformation

is needed since the spectral sensitivity function of the sensors does not match with the

desired color space. The correction is performed usually according to the formula:

RGB

out

= A ·RGB

(1.1)

where A is a 3-by-3 matrix, RGB

and RGB

out

the image before and after color matrix-

ing. The matrix coefﬁcients are not obtained using the effective response. Usually they

are retrieved using optimization methods with real acquisitions. Moreover the constraint

of the white point preservation is usually used. It corresponds to the following constraint

(as better detailed in Chapter 5):

∑

j=1

(i, j)

= 1, ∀i ∈{1,2,3} (1.2)

1.1.6 Image Formatting

The data acquired by the sensor have to be processed by the coprocessor or the host mi-

croprocessor, so both the systems must share the same communication protocol and data

format. Moreover, at the end of the image generation pipeline the image must be coded

in a standard format in order to be read by any external device. Usually the sensor pro-

vides the acquired image in the Bayer format. In the past the Bayer data were stored and

transmitted using proprietary format and protocol. Such solution has the drawback that

every customer had to design the same proprietary interface to manage the sensor data.

In the latest years the main companies making, buying or specifying camera modules

proposed a new standard called Standard Mobile Imaging Architecture (SMIA). It allows

interconnecting sensors and hosts of different vendors.

Concerning the output of the coprocessor, several standard formats are available. For

the still images the most frequently used are the Joint Picture Expert Group (JPEG) with

a lossy compression, the Targa Interchange Format (TIF) with a lossless compression. In

the top level cameras the output of the sensors can also be stored directly. In this case

usually a proprietary ﬁle format is used (e.g., the Nikon Electronic Image Format (NEF),

the Canon RAW File Format (CRW), etc.). For videos the most used are Motion JPEG,

MPEG-4, H.263 and H.264 standards.

In Chapter 11 the main data formats will be presented. Moreover some techniques

concerning the compression factor control and the error concealment will be introduced.

The compression factor control aims to obtain the ﬁle size as close as possible to a target

value whereas the error concealment aims to handle errors in the bit-stream trying to

recover the missing information.

Fundamentals and HW/SW Partitioning Image Processing for Embedded Devices



1.2 HW/SW Partitioning

Cameras embedded in mobile phones are now becoming a commodity supporting appli-

cations like capturing and transmission of still images as well as video clips (Multimedia

Messaging Services). With the increase of network bandwidth (e.g., 3G UMTS) real time

mobile video links will become feasible, enabling new applications like mobile video

telephony and video chat. It has to be noted, that the ease of use of these applications is

of high importance as this is expected to be a crucial requirement for market acceptance

of such new services. Thereby not only quality issues like frame and image stabilization

are to be focused but also the user comfort. The automatic detection and tracking of the

user’s head is such an example, which helps to keep one’s face in view of the camera

during a mobile video telephone conference. But the processing units in imaging devices

should be low-cost, low-power and, at the same time, suitable of supporting the above

mentioned mobile communication applications. In order to satisfy cost and performance

requirements, imaging device systems are generally implemented with a combination of

different components, from custom designed accelerators to standard processors. These

components can vary in their area, speed, methodology to program, and the system func-

tionality is partitioned amongst the components to best utilize this tradeoff. However,

for performance critical designs, it is not sufﬁcient to only implement the critical sec-

tions as custom-designed high-performance hardware, but it is also necessary to pipeline

the system at several levels of granularity. The custom designed accelerators can be im-

plemented by using Reconﬁgurable hardware devices, such as Field Programmable Gate

Arrays (FPGAs). The HW/SW partitioning (i.e., the deﬁnition of an architecture where

the algorithms are smartly split as hardware accelerators and software modules) is not as

straightforward as designing either software or hardware, since the application is intrin-

sically a hardware/software co-design. For instance, while an application implemented

on an FPGA can be one to two orders of magnitude faster than the application imple-

mented in software, processing in hardware incurs additional costs that are not required

for software. Some of these costs are hardware initialization costs, extra processing steps

for easy processing of the border cases, and communication of the image to and from the

reconﬁgurable device. The runtime of image processing applications varies with image

size, so processing small images on an FPGA might not be efﬁcient due to the additional

overhead. The imaging accelerators are often designed to create data-paths that are capa-

ble to process several image pixels concurrently. For the deﬁnition of these data path can

be used some well-known design approaches like:

• SIMD parallelism. Typically the data-path processes N pixels in parallel or, for

some binary operation, 8xN pixels. This type of processing is well known from

multi-media extensions used in general-purpose CPUs.

• Deeper arithmetic pipelines. These enable the encoding and execution of complex

arithmetic operations with a single microinstruction.

8 Image Processing for Embedded Devices Battiato et al.



剩余387页未读，继续阅读

自由人_SZ

粉丝: 6
资源: 7

嵌入式设备图像处理详解：从CFA到显示

Design for Embedded Image Processing on FPGAs.pdf

DESIGN FOR EMBEDDED IMAGE PROCESSING ON FPGAS

Image Processing

OpenCL Programming Guide - A. Munshi, et al

C Language Pixel Data Input and Analysis with STM32IPL: Image Processing Library for STM32 ...

【Advanced】Image Depth Estimation in MATLAB: Using Deep Learning for Image Depth Estimation

【Advanced Level】Image Face Recognition in MATLAB: Using FaceNet for Image Face Recognition

Optimization of Refresh Rate and Image Rendering Algorithms for ST7789 Displays

Frame Rate Control and Display Optimization Tips for ST7789 Displays

Practical Implementation of Driver and Hardware Acceleration Techniques for ST7789 Display

最新资源