评估与提升预测和分类模型性能

需积分: 3 87 浏览量更新于2024-07-18 收藏 6.66MB PDF 举报

"《评估与改进预测和分类》是一本由Timothy Masters编著的书籍，专注于探讨如何准确且适当地评估预测和分类模型的性能，并介绍如何通过使用高级但易于实现的技术来提升现有模型的功能。书中提供的方法适用于从简单的线性回归模型到最先进的神经网络等各种层次的研究者，帮助他们在实际应用的角度而不是模型性质的抽象层面来衡量模型的性能。通过创新的数据重采样、多次训练以及可能创建新的辅助模型，可以显著提高预测或分类的准确性。" 本书深入讨论了预测和分类理论及C++实现的算法。作者Timothy Masters提供了一种方法，使研究者能够根据实际应用场景来评估模型表现，而不仅仅是关注模型本身的特性。书中的内容涵盖了多种技术，包括： 1. **模型性能评估**：书中详细阐述了如何使用各种指标（如准确率、精确率、召回率、F1分数、ROC曲线等）来评估模型在预测和分类任务中的表现，这些指标能够帮助读者理解模型在实际问题中的优劣。 2. **数据重采样**：通过对训练数据进行不同的重采样技术，如交叉验证、自助法（Bootstrap）、留一法等，可以更好地理解模型在不同数据子集上的表现，从而识别过拟合或欠拟合问题。 3. **多轮训练**：通过多次训练模型，可以分析模型性能的变化，找到最优的超参数组合，以及了解模型是否已经收敛。 4. **创建辅助模型**：书中可能涉及到集成学习方法，如随机森林、梯度提升机或深度学习中的模型融合，通过结合多个模型的预测结果，提高整体预测或分类的稳定性。 5. **C++实现**：书中提供的算法将使用C++编程语言，这是一门高效的系统级编程语言，适合开发高性能的预测和分类系统。 6. **知识产权声明**：本书的版权归属于Timothy Masters，所有权利均受到法律保护，包括但不限于翻译、再版、复用插图、朗诵、广播、微缩复制、信息存储和检索、电子适应、计算机软件，以及当前已知或未来可能出现的任何类似或不同的方法。 7. **商标和图像使用**：尽管书中可能包含商标名称、标识和图像，但并未对每次出现的商标使用符号，而是直接使用这些名称，意味着读者需要自行了解其商标地位。《评估与改进预测和分类》是针对那些希望提升其预测和分类模型性能的实践者的宝贵资源，无论他们是在学术界还是工业界，都能从这本书中获得实质性的指导和实用的技巧。

xix

• Often, an essential part of development is estimating model

parameters for examination. e utility of these estimates is greatly

increased if you can also compute the bias and variance of the

estimates, or even produce condence intervals for them.

• Most people will train a model, verify its reasonable operation,

and place it into service. However, there are iterative algorithms in

which certain cases in the training set are given more or less weight

as a sequence of trained models is generated. ese algorithms can

sometimes behave in an almost miraculous fashion, transforming

a weak classier into one with enormous power, simply by focusing

attention on specic training cases.

• e nal step in many model development applications is to pool

all available data and train the model that will be deployed, leaving

no data on which to test the nal model. Monte Carlo permutation

tests can oer valuable performance indications without requiring

any data to be sacriced. ese same tests can be used early in the

development to discover models that are susceptible to overtting.

• A common procedure is to train several competing models on

the same training set and then choose the best performer for

deployment. However, it is usually advantageous to use all of the

competing models and intelligently combine their predictions or

class decisions to produce a consensus opinion.

• It is often the case that dierent models provide dierent degrees of

performance depending on conditions that do not serve as model

inputs. For example, a model that predicts the pharmacological eect

of a drug cocktail may perform best when the expression of one gene

dominates, and another model may be superior when another gene

expression is dominant. Automated gating algorithms can discover

and take advantage of such situations.

• It is not widely known that the entropy of a predictor variable can

have a profound impact on the ability of many models to make

eective use of the variable. Responsible researchers will compute

the entropy of every predictor and take remedial action if any

predictor has low entropy.

PrefaCe

• Many applications involve wildly throwing spaghetti on the wall to

see what sticks. A researcher may have literally hundreds or even

thousands of candidates for predictor variables, while the nal

model will employ only a handful. Computations involving mutual

information and uncertainty reduction between the candidates

and the predicted variable provide methods for rapidly screening

candidates, eliminating those that are least likely to be useful while

simultaneously discovering those that are most promising.

• One exciting subject in information theory is causation, the literal

effect of one event causing another, as opposed to their just being

related. Transfer entropy (Schreiber’s information transfer) quantifies

causative events.

These and many more topics are addressed in this text. Intuitive justification is a

focal point in every presentation. The goal is that you understand the problem and

the solution on a solid gut level. Then, one or more references are given as needed if

you want to see rigorous mathematical details. In most cases, only the equations that

are critical to the technique are provided here. Long and tedious proofs are invariably

left to the references. Finally, the technique is illustrated with heavily commented C++

source code. All code is supplied on my web site (TimothyMasters.info) as complete working

programs that can be compiled with the Microsoft Visual Studio compiler. Those

sections of code that are central to a technique are printed in the book for the purpose of

explication.

The ultimate purpose of this text is threefold. The first goal is to open the eyes of

serious developers to some of the hidden pitfalls that lurk in the model development

process. The second is to provide broad exposure for some of the most powerful model

enhancement algorithms that have emerged from academia in the past two decades,

while not bogging you down in cryptic mathematical theory. Finally, this text should

provide you with a toolbox of ready-to-use C++ code that can be easily incorporated into

your existing programs. Enjoy.

PrefaCe

dangerous if not done carefully. Ultimately, the choice of a numeric or classification

model depends on the nature of the data and on a sometimes arbitrary decision by

the experimenter. This chapter discusses methods for assessing models that make

numeric predictions.

Notation

Numeric prediction models use one or more independent variables to predict one or

more dependent variables. By common convention, the independent variables employed

by a model are designated by some form of the letter x, while the dependent variables

are designated by the letter y. Unless otherwise stated, scalar variables are represented in

lowercase italic type with an optional subscript, and vectors are represented in lowercase

bold type. Thus, we may refer to the vector x={x

, x

, …}. When we want to associate

x with a particular case in a collection, we may use a subscript as in x

. Matrices are

represented by uppercase bold type, such as X.

The parameters of a prediction model are Greek letters, with theta (θ) being

commonly employed. If the model has more than one parameter, separate Greek letters

may be used, or a single bold type vector (θ) may represent all of the parameters.

A model is often named after the variable it predicts. For example, suppose we

have a one-parameter model that uses a vector of independent variables to predict a

scalar variable. This model (or its prediction, according to context) would be referred

to as y(x; θ ).

A collection of observed cases called the training set is used to optimize the

prediction model in some way, and then another independent collection of cases called

the test set or validation set is used to test the trained model. (Some schemes employ

three independent collections, using both a test set and a validation set. One is used

during model development, and the other is used for final verification. We’ll discuss

this later.) The error associated with case i in one of these collections is the difference

between the predicted and the observed values of the dependent variable. This is

expressed in Equation (1.1).

ey y

iii

()

-x ; q

(1.1)

Chapter 1 assessment ofnumeriC prediCtions

Overview of Performance Measures

There are at least three reasons why we need an effective way to measure the

performance of a model. The most obvious reason is that we simply need to know if

it is doing its job satisfactorily. If a model’s measured performance on its training set

is unsatisfactory, there is no point in testing it on an independent dataset; the model

needs revision. If the model performs well on the training set but performs poorly on

the independent set, we need to find out why and remedy the situation. Finally, if the

performance of a model is good when it is first placed in service but begins to deteriorate

as time passes, we need to discover this situation before damage is done.

The second reason for needing an effective performance criterion is that we must

have an objective criterion by which the suitability of trial parameter values may

be judged during training. The usual training procedure is to have an optimization

algorithm test many values of the model’s parameter (scalar or vector). For each trial

parameter, the model’s performance on the training set is assessed. The optimization

algorithm will select the parameter that provides optimal performance in the training

set. It should be apparent that the choice of performance criterion can have a

profound impact on the value of the optimal parameter that ultimately results from the

optimization.

The final reason for needing an effective performance criterion is subtle but

extremely important in many situations. When we compute the parameter value that

optimizes a performance criterion or when we use a performance criterion to choose the

best of several competing models, it is a little known fact that the choice of criterion can

have a profound impact on how well the model generalizes to cases outside the training

set. Obviously, a model that performs excellently on the training set but fails on new

cases is worthless. To encourage generalization ability, we need more than simply good

average performance on the training cases. We need consistency across the cases.

In other words, suppose we have two alternative performance criteria. We optimize

the model using the first criterion and find that the average performance in the training

set is quite good, but there are a few training cases for which the trained model performs

poorly. Alternatively, we optimize the model using the second criterion and find that the

average performance is slightly inferior to that of the first criterion, but all of the training

cases fare about equally well on the model. Despite the inferior average performance,

the model trained with the second criterion will almost certainly perform better than the

first when it is placed into service.

Chapter 1 assessment ofnumeriC prediCtions

剩余529页未读，继续阅读

zyeyes

粉丝: 1
资源: 16

评估与提升预测和分类模型性能

Assessing and Improving Prediction and Classification Theory and Algorithms epub

[Practical Exercise] Data Storage and Analysis: Storing Scraped Data into MySQL and Performing Data ...

Assessing Model Generalization Capability: The Right Approach to Cross-Validation

Establishing and Training Machine Learning Models in Jupyter Notebook

MATLAB Product Design Optimization: Case Studies and Applications Analysis

MATLAB Genetic Algorithm Optimization of Neural Network Weights: Applied Research and Practical ...

Model Interpretation and Explainability Techniques in YOLOv8: Analyzing the Black Box of Deep Neural...

一款高效的Vue 3低代码表单，可视化设计，一键生成源码

Python第八周作业

滑模控制simulink仿真.zip

最新资源