基于最小二乘法的正则化约束低秩学习：提升分类与回归效率

20 浏览量更新于2024-08-26 收藏 1.75MB PDF 举报

"《基于最小二乘法的正则化约束低秩学习》是一篇发表在2017年12月《IEEE Transactions on Cybernetics》的研究论文。该文章聚焦于近年来备受关注的低秩学习方法，尤其在诸如子空间分割和图像分类等实际任务中的有效性。然而，传统的低秩方法往往在监督学习任务，如分类和回归中，无法捕捉到低维特征子空间，这限制了其性能。本文的主要目标是发展一种能够同时处理区分性低秩表示（LRR）和鲁棒投影子空间的监督学习策略。为了实现这一目标，作者将问题转化为一个基于最小二乘法的约束秩最小化框架，通过引入正则化技术来优化模型。这种正则化方法使得数据标签结构能够与由低秩学习得到的、基于鲁棒子空间投影的原始数据的低维表示相匹配。换句话说，数据的内在结构被映射到一个在低维空间中具有更好解释性的表示上，从而增强模型的预测能力和对异常数据的鲁棒性。具体来说，研究者提出了一种算法，它在保持低秩学习的特性的同时，通过最小二乘准则寻找最佳的低秩表示，这有助于减少过拟合的风险，并确保模型在面对复杂多变的数据时仍能保持良好的泛化能力。论文还可能探讨了不同的正则化参数选择、损失函数优化以及可能的算法实现细节，以保证学习过程的有效性和效率。总结来说，这篇论文提供了一种创新的方法，将低秩学习与监督学习相结合，通过最小二乘法的正则化约束，提升模型在有标注数据上的表现，特别是在需要同时处理数据降维和分类任务的场景中。这种方法有望在多个领域，如计算机视觉、信号处理或机器学习中，发挥重要作用。"

4252 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 47, NO. 12, DECEMBER 2017

proved to be very effective in practical applications, such as

principal component analysis (PCA) and linear discriminant

analysis (LDA) [25]. Recently, Jiang et al. [26] developed

a subspace method for facial eigenfeature regularization and

extraction (ERE), and the eigenspace of the within-class scatter

matrix is decomposed into three subspaces involving a reliable

subspace spanned mainly by variation, an unstable one due to

noise as well as limited training data, and a null subspace. This

could alleviate the problem of instability, overﬁtting, and poor

generalization. As we know, PCA is an unsupervised method

and does not require label information. To take into account the

label information, an asymmetric PCA (APCA) [27] approach

was proposed, which utilizes class covariance matrices and

enables removing the unreliable dimensions of principal com-

ponents. While APCA is designed for handling the two-class

problem, supervised PCA (SPCA) [28] deals with the multiple-

class problem. Unlike APCA, SPCA imposes different weights

on the covariance matrices so as to consider class-speciﬁc

information of the data set. To summarize, the above meth-

ods are expected to generate considerable subspaces, but they

cannot explicitly yield low-rank subspaces and separate the

error matrix as our approach can, e.g., they cannot recover

the clean component of the corrupted image. In some sense,

this would hinder potentially more widespread applications of

these methods.

It is worth noting that there exist two works related to

ours. One is the supervised regularization-based robust sub-

space (SRRS) method [29], which smoothly integrates sub-

space learning and data recovery in a uniﬁed framework to

jointly learn discriminative subspace and LRR from the data.

It differs from our method in several aspects.

1) It adopts the Fisher criterion to capture the discriminant

structure while ours utilizes both the Laplacian regular-

izer and the least squares regularizer under the guidance

of the supervised information.

2) It includes a generalized eigen-decomposition problem

to obtain the projecting subspace while ours gives a

closed-form solution accordingly and avoids solving the

expensive Sylvester equation.

3) Our method can be used for regression tasks directly

while SRRS cannot. The other is robust regression

(RR) [30], which leverages the rank regularizer and the

sparse error term, but it regards the underlying data

structure as a single low-rank subspace that might cause

the inaccurate recovery.

By contrast, we assume the data populates on a mixture

of multiple subspaces to guarantee the correct recovery.

Moreover, the subspace obtained from RR does not have

the desired informative properties, e.g., the locality-preserving

ability, while our method can easily achieve this based on the

adaptive regularizer. Details of our method are elaborated in

the following sections.

III. P

ROBLEM SETTING

In this paper, we deﬁne the constrained low-rank learn-

ing problem as follows. Given a collection of data points

, x

,...,x

} and their labels {y

, y

,...,y

} distributed in

k classes, we assume they are samples approximately drawn

from a mixture of several subspaces [2]. The principal goal is

to seek the discriminant lowest rank representation Z as well as

the robust projecting subspace P. More speciﬁcally, we denote

the training data by X ∈ R

d×n

with each data point stacked in

a column, and the data matrix can be decomposed into a clean

component

X = AZ and an error component E ∈ R

d×n

, where

A ∈ R

d×m

is treated as the dictionary linearly spanning the

data space while Z ∈ R

m×n

reveals the underlying subspace

structure of the data.

More importantly, we argue that the recovered data can be

mapped onto a low-dimensional data space by the robust pro-

jecting subspace P ∈ R

d×k

(the reduced dimension k is set

to the number of classes), i.e., V = P

AZ ∈ R

k×n

. On one

hand, the low-dimensional data representation V is expected

to be closely correlated to the label indicator matrix Y ∈ R

k×n

while it acts as the estimated output given the input data. The

matrix Y takes discrete values for classiﬁcation and continu-

ous values for regression, respectively, e.g., the entries in each

column of Y are set to 1 if the sample belongs to the corre-

sponding class. On the other hand, it is easy to endow the

low-dimensional representation P

X, derived from the orig-

inal data space, with several appealing properties like the

locality-preserving ability, by the constraint matrix L ∈ R

n×n

Usually, this matrix should be semi-positive deﬁnite to make

the imposed regularizer convex.

By tradition, the lowest rank representation is employed to

construct an afﬁnity matrix for subspace segmentation in unsu-

pervised learning. Here, we mainly use it for recovering the

clean data by AZ, where Z plays a dominant role. Under such

circumstance, both the recovered training data and testing data

could show the robustness to noise or corruptions, and it also

allows to discriminate the samples from different categories.

IV. O

UR METHOD

This section concentrates on elaborating the proposed

method, including the formulation, and the optimization

framework as well as the algorithmic procedures.

A. Formulation

As mentioned earlier, our goal is to jointly seek the dis-

criminant lowest-rank representation Z ∈ R

m×n

and the

robust projecting subspace P ∈ R

d×k

in a supervised manner.

Essentially, we have to minimize rank(Z), which is yet difﬁ-

cult to solve due to its discrete nature. As a common practice

in low-rank methods [2], [5], we use the nuclear norm as its

convex surrogate. In this paper, the dictionary A is set to X.

Hence, our objective function can be formulated as

min

Z,E,P

Z

∗

+ λE

2,1

+ αTr



XLX



+ βV − Y

s.t. X = XZ +E, V = P

XZ, 1

Z = 1

(1)

where the nuclear norm ·

∗

is the sum of singular values of

a matrix, the group sparse norm ·

2,1

computes the sum of

absolute values of l

-norm on each column vector of a matrix,

e.g.,



E



for E, ·

denotes the Frobenius norm of a

matrix, 1 is a column vector with all ones. The parameter

α>0 balances the contribution of the constraint to the objec-

tive, β>0 controls the ﬁtting of the least squares term, and

剩余12页未读，继续阅读

Yoo?

粉丝: 4
资源: 932

基于最小二乘法的正则化约束低秩学习：提升分类与回归效率

核低秩表示代码MATLAB实现

来自麻省理工的正则化最小二乘法讲义

求解正则化非负低秩逼近问题的交替最小二乘算法.docx

最小二乘法求解稀疏约束，压缩感知，正则优化等问题

正则化正交最小二乘法（ROLS)

数据拟合与最小二乘法研究：列满秩、加权、正则化、等式约束问题探讨

最小二乘法＋正则化项

并且需要以正则化或非负最小二乘法为约束，代码怎么做呢。

各种最小二乘法汇总（算例及MATLAB程序）,最小二乘法的matlab程序,matlab

各种最小二乘法汇总（算例及MATLAB程序）,最小二乘法的matlab程序,matlab源码.zip

最新资源