【Foundation】Detailed Explanation of MATLAB Toolbox: Deep Learning Toolbox

# 2.1 Fundamentals of Neural Networks

### 2.1.1 Structure and Operation of Neurons

A neuron is the basic unit of a neural network, mirroring the functionality of biological neurons. Each neuron takes multiple inputs and generates one output. The structure of a neuron includes:

- **Input Weights:** Weights connecting inputs to the neuron, determining the influence of each input on the output.
- **Bias:** A constant added to the neuron's inputs, used to adjust the neuron's activation threshold.
- **Activation Function:** A nonlinear function that maps the weighted sum of the neuron to the output.

The typical operation of a neuron is as follows:

1. **Weighted Sum Calculation:** Multiply each input by its weight and sum them.
2. **Bias Addition:** Add the bias to the weighted sum.
3. **Activation Function Application:***

***monly used activation functions include sigmoid, ReLU, and tanh, each with different nonlinear characteristics.

# 2. Fundamentals of Deep Learning Theory

### 2.1 Fundamentals of Neural Networks

#### 2.1.1 Structure and Operation of Neurons

A neuron is the basic unit of a neural network, mimicking the structure and function of biological neurons. A neuron consists of the following parts:

- **Input Layer:** Receives signals from other neurons or input data.
- **Weights:** Used to adjust the impact of input signals.
- **Activation Function:** Converts the weighted sum into output.
- **Output:** Passed to other neurons or as the network's final output.

The operation of a neuron is as follows:

1. Multiply input signals by their respective weights.
2. Sum the weighted inputs.
3. Input the summed value into the activation function.
4. The activation function produces an output, which can be binary (0 or 1) or continuous values (e.g., -1 to 1).

Commonly used activation functions include:

- **Sigmoid:** Maps the input to a range between 0 and 1.
- **ReLU:** Outputs the input value only when it is greater than 0, otherwise outputs 0.
- **Tanh:** Maps the input to a range between -1 and 1.

#### 2.1.2 Types and Characteristics of Neural Networks

Neural networks can be categorized into various types based on their structure and connection patterns:

- **Feedforward Neural Network:** Signals propagate from the input layer to the output layer without cyclic connections.
- **Recurrent Neural Network (RNN):** Signals can flow cyclically within the network, allowing it to process sequential data.
- **Convolutional Neural Network (CNN):** Specialized for processing grid-like data, such as images.
- **Transformer Model:** Based on attention mechanisms, used for processing sequential data, such as text and speech.

Different types of neural networks have distinct characteristics:

| Type of Neural Network | Characteristics |
|---|---|
| Feedforward Neural Network | Simple and efficient, suitable for classification and regression tasks |
| Recurrent Neural Network | Capable of processing sequential data but prone to gradient vanishing and explosion issues |
| Convolutional Neural Network | Possesses translational invariance, suitable for image processing tasks |
| Transformer Model | Based on attention mechanisms, adept at handling long sequence data |

### 2.2 Deep Learning Models

#### 2.2.1 Convolutional Neural Network (CNN)

CNN is a deep learning model specifically designed for processing grid-like data. Its architecture includes:

- **Convolutional Layer:** Extracts features from data using convolution kernels.
- **Pooling Layer:** Downsamples the output of the convolutional layer, reducing feature dimensions.
- **Fully Connected Layer:** Used for classification or regression tasks.

The advantages of CNNs are:

- **Translational Invariance:** Robust to translations in images.
- **Local Connectivity:** Processes only local regions of data, reducing computational load.
- **Weight Sharing:** Convolution kernels are shared across the entire feature map, reducing the number of parameters.

#### 2.2.2 Recurrent Neural Network (RNN)

RNN is a deep learning model capable of processing sequential data. Its architecture includes:

- **Hidden Layer:** Retains information from previous sequences.
- **Recurrent Connection:** Feeds the output of the hidden layer back into the input.

The advantages of RNNs are:

- **Sequence Memory:** Can remember information within sequences.
- **Dynamic Modeling:** Can handle sequences of varying lengths.

#### 2.2.3 Transformer Model

The Transformer is a deep learning model based on attention mechanisms. Its architecture includes:

- **Encoder:** Transforms input sequences into a series of vectors.
- **Decoder:** Generates output sequences using the encoder's output.
- **Attention Mechanism:** Allows the model to focus on relevant parts of