【Advanced】Implementation of Logistic Regression in Matlab

# [Advanced] Logistic Regression Implementation in Matlab

# 2.1 Logistic Function and Probabilistic Interpretation

The logistic function, also known as the sigmoid function, is an S-shaped curve described by the mathematical expression:

f(x) = 1 / (1 + e^(-x))

It maps real numbers to probabilities ranging between 0 and 1. In the context of logistic regression models, the logistic function is used to map input features onto probabilities for a binary classification problem.

Specifically, given an input feature vector x, the logistic function calculates the probability p of the event occurring:

p = f(w^T x + b)

where w is the weight vector and b is the bias term. The probability p represents the likelihood of the event occurring, while 1 - p represents the likelihood of the event not occurring.

# 2. Mathematical Principles of Logistic Regression Model

### 2.1 Logistic Function and Probabilistic Interpretation

The core of the logistic regression model is the logistic function, which maps input values to probabilities between 0 and 1. The mathematical expression for the logistic function is:

f(x) = 1 / (1 + e^(-x))

where x is the input value.

The graph of the logistic function is shown below:

[Image of Logistic Function Graph]

The logistic function has the following properties:

- As x approaches positive infinity, f(x) approaches 1.
- As x approaches negative infinity, f(x) approaches 0.
- The derivative of f(x) is f(x) * (1 - f(x)).

In the logistic regression model, the logistic function is used to link input features with output probabilities. Given an input feature vector x, the logistic regression model predicts the probability of the output y as:

P(y = 1 | x) = f(w^T x + b)

where w is the weight vector and b is the bias term.

### 2.2 Log-Likelihood Function and Maximum Likelihood Estimation

The parameters w and b of the logistic regression model are estimated using the maximum likelihood estimation (MLE) method. The goal of MLE is to find a set of parameters that maximize the likelihood function of a given dataset.

For the logistic regression model, the likelihood function is:

L(w, b) = ∏[P(y_i | x_i)]^y_i * [1 - P(y_i | x_i)]^(1 - y_i)

where y_i is the true label of the ith sample and x_i is the feature vector of the ith sample.

Taking the logarithm of the likelihood function yields the log-likelihood function:

l(w, b) = ∑[y_i * log(P(y_i | x_i)) + (1 - y_i) * log(1 - P(y_i | x_i))]

The objective of MLE is to find a set of parameters w and b that maximize the log-likelihood function. This can be achieved using optimization algorithms such as gradient descent or Newton's method.

### 2.3 Methods for Solving Model Parameters

There are several methods for solving the parameters of a logistic regression model, among which the most commonly used method is gradient descent. Gradient descent iteratively updates the weights and bias terms to gradually incr