凸优化基础与应用

需积分: 26 45 浏览量更新于2024-07-17 1 收藏 5.75MB PDF 举报

"该资源是一本关于凸优化的教材，由斯坦福大学的Stephen Boyd和加州大学洛杉矶分校的Lieven Vandenberghe合著。书中详细介绍了凸集、凸函数的基础理论，以及凸优化的对偶理论，并探讨了在约束条件下的优化问题和无约束优化问题的解决方法。" 在数学和工程领域，凸优化是一种非常重要的优化工具，它在机器学习、信号处理、经济学和控制理论等多个领域都有广泛应用。凸优化的主要特点在于，它的目标函数和约束条件都是凸的，这使得问题具有良好的性质，例如全局最优解的存在性和唯一性，以及可以利用一系列高效算法求解。首先，**凸集**是凸优化的核心概念之一。一个集合是凸的，如果对于集合中的任意两点，连接它们的所有线段都完全包含在集合内。在几何上，这意味着集合不会形成任何内部的“凹陷”。例如，直线、圆和多边形（只要不包括内部的洞）都是凸集的例子。其次，**凸函数**是指在其定义域内，任何两点的线性组合的函数值都不低于这两点各自函数值的线性组合。直观来说，如果在函数图像上任取两点并画一条直线，那么这条直线总是位于函数图像的下方。这种性质使得凸函数在优化问题中非常有用，因为它们避免了局部最小值的存在，确保了全局最小值的可寻性。接下来，**凸优化的对偶理论**是解决复杂问题的关键。通过将原问题转化为对偶问题，我们可以得到一个更易于处理的优化模型，有时甚至可以直接求解出最优解。对偶问题是由原问题的拉格朗日函数构造的，其中包含了原问题的约束和目标函数的线性组合。此外，**带约束与无约束的优化问题**是凸优化的两个基本类型。无约束优化问题相对简单，通常可以通过梯度下降法或拟牛顿法等直接求解。然而，实际问题往往涉及到各种限制，如线性约束、非线性约束等，这些就需要用到如拉格朗日乘子法、惩罚函数法或者内点法等方法来处理。最后，书中还可能涉及**凸优化算法**的实现和应用，如梯度投影算法、SMO（Sequential Minimal Optimization）算法在支持向量机中的应用，或者是二次规划问题的求解方法，如共轭梯度法和拟牛顿法等。 "凸优化"这本书深入浅出地介绍了凸优化的理论与实践，对于想要理解和应用凸优化的人来说，是一份宝贵的参考资料。

2 1 Introduction

for all x, y ∈ R

and all α, β ∈ R with α + β = 1, α ≥ 0, β ≥ 0. Comparing (1.3)

and (1.2), we see that convexity is more general than linearity: inequality re pl aces

the more res tr i cti ve equality, and the inequality must hold only for certain values

of α and β. Since any linear program is therefore a convex optimization problem,

we can consider convex optimization to be a generalization of linear programming.

1.1.1 Applications

The optimization problem (1.1) is an abstraction of the problem of making the best

possible choice of a vector in R

from a set of candidate choices. The variable x

represents the choice made; the constraints f

(x) ≤ b

represent ﬁrm requir ements

or speciﬁcations that limit the possible choices, and the objective value f

(x) rep-

resents the cost of choosing x. (We can also think of −f

(x) as representing the

value, or utility, of choosing x.) A solution of the optimization problem (1.1) corre-

sponds to a choice that has minimum cost (or maximum utility), among all choices

that meet the ﬁrm requirements.

In portfolio optimization, for example, we seek the best way to invest some

capital in a set of n assets. The variable x

represents the investment in the ith

asset, so the vector x ∈ R

describes the overall portfolio allocati on across the set of

assets. The constraints might represent a limit on the budget (i.e., a limit on the

total amount to be invested), th e requirement that investments are nonnegative

(assuming short positions are not allowed), and a minimum acceptable value of

expected return for the whole portfolio. The objective or cost function might be

a measure of the overall risk or variance of the portfolio retu r n. In this case,

the optimization problem (1.1) corresponds to choosing a portfolio allocation that

minimizes ri sk , among all possible allocations that meet the ﬁrm requirements.

Another example is device sizing in electronic design, which is the task of choos-

ing the width and length of each device in an electronic circuit. Here the variables

represent the widths and lengths of the devices. The constraints represent a va-

riety of engineering requirements, such as limits on the device sizes imposed by

the manufacturing process, timing requirements that ensure that the circuit can

operate reliably at a speciﬁed speed, and a limit on the total area of the circuit. A

common objective in a device sizing problem is the total power consumed by the

circuit. The optimization problem (1.1) is to ﬁnd the device sizes that satisfy the

design requirements (on manufacturability, timing, and area) and are most power

eﬃcient.

In data ﬁtting, the task is to ﬁnd a model, from a family of potential models,

that best ﬁts some observed data and prior information. Here the variables are the

parameters in the model, and the cons tr aints can represent prior information or

required l imit s on the parameters (such as nonnegativity). The objective function

might be a measure of misﬁt or prediction error between the observed data and

the values predicted by t he model, or a statistical measure of the u nli keliness or

implausibility of the parameter values. The optimization probl em (1.1) is to ﬁnd

the model parameter values that are consistent with the prior information, and give

the smallest misﬁt or prediction error with the observed data (or, in a statistical

1.1 Mathematical optimization 3

framework, are most likely).

An amazing variety of practical problems involving decision making (or system

design, analysis, and operation) can be cast in the form of a mathematical opti-

mization problem, or some variation such as a multicriterion optimization problem.

Indeed, mathematical optimization has become an important tool in many areas.

It is widel y used in engineering, in electronic design automation, automatic con-

trol systems, and optimal design problems arising in civil, chemical, mechanical,

and aerospace engineering. Optimization is used for problems arising in network

design and operation, ﬁnance, supply chain management, scheduling, and many

other areas. The list of applications is still steadily expanding.

For most of these applications, mathematical optimization is used as an aid to

a human deci si on maker, system designer, or system operator, who supervises the

process, checks the results, and modiﬁ es the problem (or the solution approach)

when necessary. This human decision maker also carries out any actions suggested

by the optimization problem, e.g., buying or selling asset s to achieve the opt imal

portfolio.

A r el atively recent phenomenon opens the possibility of many other applications

for mathematical optimization. With the proliferation of computers embedded in

products, we have seen a rapid growth in embedded optimization. In these em-

bedded applications, optimization is used to automatically make real-time choices,

and even carry out the associated actions, with no (or little) human intervention or

oversight. In some application areas, this blending of tradition al automatic control

systems and embedded optimization is well under way; in others, it is just start-

ing. Embedded real-time optimization raises some new challenges: in particular,

it requires solution methods that are extremely reliable, and solve problems in a

predictable amount of time (and memory).

1.1.2 Solving optimization problems

A solution method for a class of optimization problems is an algor ith m that com-

putes a solution of the problem (to some given accuracy), given a particular problem

from the class, i.e., an instance of the problem. Since the late 1940s, a large eﬀort

has gone into developing algorithms for solving various classes of optimization prob-

lems, analyzing their properties, and developing good software implementations.

The eﬀectiveness of these algorithms, i.e., our ability to solve the optimization prob-

lem (1.1), varies considerably, and depends on factors such as the particular forms

of the objective and constraint functions, how many variables and constraints there

are, and special structure, such as sparsity. (A problem is sparse if each constr aint

function depends on only a small number of the variables).

Even when the objective and constraint functions are smooth (for example,

polynomials) the general optimization problem (1.1) is surprisingly diﬃcult to solve.

Approaches to the general problem therefore involve some kind of compromise, such

as very long computation time, or the possibility of not ﬁnding the solution. Some

of these methods are discussed in §1.4.

There are, however, some important exceptions to the general rule that most

optimization problems are diﬃcul t to solve. For a few problem classes we have

4 1 Introduction

eﬀective algorithms that can reliably solve even large problems, with hundreds or

thousands of variables and constraints. Two imp or tant and well known examples,

described in §1.2 below (and in detail in chapter 4), are least-squares problems and

linear programs. It i s less well known that convex optimization is another exception

to the rule: Like least-squares or linear programming, there are very eﬀe ct ive

algorithms that can reliably and eﬃciently solve even large convex problems.

1.2 Least-squares and linear programming

In this section we describe two very widely known and used special subclasses of

convex optimization: least-squares and linear programming. (A c omple te technical

treatment of these problems will be given in chapter 4.)

1.2.1 Least-squares problems

A least-squares problem is an optimi zati on problem with no constraints (i.e., m =

0) and an objective which is a sum of s qu ar es of terms of the form a

x − b

minimize f

(x) = kAx − bk

i=1

x − b

)

(1.4)

Here A ∈ R

k×n

(with k ≥ n), a

are the rows of A, and the vector x ∈ R

is the

optimization variable.

Solving l eas t-s quares problems

The solution of a least-squares problem (1.4) can be reduce d to solving a set of

linear equations ,

A)x = A

so we have the analytical solution x = (A

−1

b. For least-squares problems

we have good algorithms (and software implementations) for solving the problem to

high accuracy, with very high reliability. The least-squares problem can be solved

in a time approximately proportional to n

k, with a known constant. A current

desktop computer can solve a least-squares problem with hundreds of variables, and

thousands of terms, in a few seconds; more powerful computers, of course, can solve

larger problems, or the same size problems, faster. (Moreover, these solution times

will decrease exponentially in the future, according to Moore’s law.) Algorithms

and software for solving least-squares problems are reliable enough for embedded

optimization.

In many cases we can solve even larger least-squares problems, by exploiting

some special structure in the coeﬃcient matrix A. Suppose, for example, that the

matrix A is sparse, which means that it has far fewer than kn nonzero entries. By

exploiting sparsity, we can usually solve the least-squares problem much faster than

order n

k. A current desktop computer can solve a sparse least-squares problem

1.2 Least-squares and li near programming 5

with tens of thousands of variables, and hundreds of thousands of terms, in around

a minute (although this depends on the particular sparsity pattern).

For extr eme ly large problems (say, with millions of variables), or for problems

with exacting real-time c omput in g requirements, solving a least-square s problem

can be a challenge. But in the vast majority of cases, we can say th at existing

methods are very eﬀective, and extremely reliable. Ind ee d, we can say that solving

least-squares problems (that are not on the boundary of what is currently achiev-

able) is a (mature) technology, that can be reliably used by many people who do

not know, and do not need to know, the details.

Using least-squares

The least-squ are s problem is the basis for regression analysis, optimal control, and

many parameter estimation and data ﬁtting methods. It has a number of statistical

interpretations, e.g., as maximum likelihood estimation of a vector x, given linear

measurements corr u pt ed by Gaussian measurement errors.

Recognizing an optimization problem as a least-s q uar es problem is straightfor-

ward; we only need to verify that the objective is a quadratic function (and then

test whether th e associated quadratic form is positive semideﬁnite). While the

basic l east -s q uar es problem has a simple ﬁxed form, several standard techniques

are used to increase its ﬂexibility in applications.

In weighted least-squares, the weighted least-squares cost

i=1

x − b

)

where w

, . . . , w

are positive, is minimized. (This problem is readily cast and

solved as a standard least-squares problem.) Here the weights w

are chosen to

reﬂect diﬀering levels of concern about the sizes of the t er ms a

x − b

, or simply

to inﬂuence the solution. In a statistical setting, weighted least-squares arises

in estimation of a vector x, given linear measurements corru pt ed by errors with

unequal variances.

Another technique in least-squares is regularization, in which extra terms are

added to the cost function. In the simpl es t case, a positive multiple of the sum of

squares of the variables is added to the cost function:

i=1

x − b

)

+ ρ

i=1

where ρ > 0. (This pr obl em too can be formulated as a standard least-squares

problem.) The extra terms penalize large values of x, and result in a sensible

solution in cases when minimizing the ﬁrst sum only does not. The parameter ρ is

chosen by the user to give the right trade-oﬀ between making the original objective

function

i=1

x−b

)

small, while keeping

i=1

not too big. Regularization

comes up in statistical estimation when the vector x to be e st imate d is given a prior

distribution.

Weighted least-squares and regul ari zati on are covered in chapter 6; their sta-

tistical interpretations are given in chapter 7.

6 1 Introduction

1.2.2 Linear programming

Another important class of optimization problems is linear programming, in which

the objective and all const r aint functions are linear:

minimize c

subject to a

x ≤ b

, i = 1, . . . , m.

(1.5)

Here the vectors c, a

, . . . , a

∈ R

and scalars b

, . . . , b

∈ R are problem pa-

rameters that specify the object ive and constraint functions.

Solving l ine ar programs

There is no simple analytical formula for the solution of a linear program (as there

is for a least-squares problem), but there are a variety of very eﬀective methods for

solving them, including Dantzig’s simplex method, and the more recent interior-

point metho d s descr ibed later in this book. While we cann ot give the exact number

of arithmetic operations required to solve a linear program (as we can for least-

squares), we c an establish rigorous bounds on the number of operations required

to solve a linear program, to a given accuracy, using an interior-point method. The

complexity in practice is order n

m (assuming m ≥ n) but with a constant that is

less well characterized than for least-sq uar es . These algorithms are quite r e li able,

although perhaps not quite as re liab le as methods for least-squares. We can easily

solve problems with hundreds of variables and thousands of constraints on a small

desktop computer, in a matter of seconds. If the problem is sparse, or has some

other exploitable structure, we can often solve problems with tens or hundreds of

thousands of variables and constraints.

As with least-squares problems, it is still a challenge to solve extremely large

linear pr ograms, or to solve linear programs with ex actin g real-time computing re-

quirements. But, like least-squares, we can say that solv in g (most) linear programs

is a mature technology. Linear programming solvers can be (and are) embedded in

many tools and applications.

Using linear programming

Some applications lead directly to linear programs in the form (1.5), or one of

several other standard forms. In many other cases the original optimization prob-

lem does not have a standard linear program form, but can be tr ans for med to an

equivalent linear program (and then, of course, solved) using techniques covered in

detail in chapter 4.

As a simple example, consider the Chebyshev approximation problem:

minimize max

i=1,...,k

x − b

(1.6)

Here x ∈ R

is the variable, and a

, . . . , a

∈ R

, b

, . . . , b

∈ R are parameters

that specify t he problem instance. Note the resemblance to the least-squares prob-

lem (1.4). For both problems, the objective is a measure of the size of the terms

x − b

. In least-squares, we use the sum of squares of the terms as objective,

whereas in Chebyshev approximation, we u s e the maximum of the absolute values.

剩余729页未读，继续阅读

杨跟的博客

粉丝: 74
资源: 5

凸优化基础与应用

cvxpy:用于凸优化问题的Python嵌入式建模语言

凸优化Convex Optimization

凸优化 Convex Optimization by Stephen Boyd

机器学习凸优化 Convex Optimization for Machine Learning

斯坦福教授讲解：凸优化 Convex Optimization

入门必备：凸优化 Convex Optimization 教材解析

斯坦福大学经典教材：凸优化Convex Optimization

凸优化Convex Optimization Overview Zico Kolter (updated by Honglak Lee)2008

如何应用凸优化技术处理实际的决策分析问题？请结合《斯坦福大学经典教材：凸优化Convex Optimization》提供具体的案例。

如何通过凸优化技术解决实际中的决策分析问题？请结合《斯坦福大学经典教材：凸优化Convex Optimization》给出具体应用案例。

最新资源