帮我解释一下这段话:The connection growth algorithm greedily activates useful, but currently ‘dormant,’ connections. We incorporate it in the following learning policy: Policy 1: Add a connection w iff it can quickly reduce the value of loss function L. The DNN seed contains only a small fraction of active connections to propagate gradients. To locate the ‘dormant’ connections that can reduce L effectively, we evaluate ∂L/∂w for all the ‘dormant’ connections w (computed either using the whole training set or a large batch). Policy 1 activates ‘dormant’ connections iff they are the most efficient at reducing L. This can also assist with avoiding local minima and achieving higher accuracy [28]. To illustrate this policy, we plot the connections grown from the input to the first layer of LeNet-300-100 [7] (for the MNIST dataset) in Fig. 3. The image center has a much higher grown density than the margins, consistent with the fact that the MNIST digits are centered. From a neuroscience perspective, our connection growth algorithm coincides with the Hebbian theory: “Neurons that fire together wire together [29]." We define the stimulation magnitude of the mth presynaptic neuron in the (l + 1)th layer and the n th postsynaptic neuron in the l th layer as ∂L/∂ul+1 m and x l n , respectively. The connections activated based on Hebbian theory would have a strong correlation between presynaptic and postsynaptic cells, thus a large value of (∂L/∂ul+1 m )x l n . This is also the magnitude of the gradient of L with respect to w (w is the weight that connects u l+1 m and x l n ): |∂L/∂w| = (∂L/∂ul+1 m )x l n (1) Thus, this is mathematically equivalent to Policy 1.
时间: 2023-06-24 10:04:50 浏览: 71
这段话描述了一种神经网络训练策略,即通过激活当前处于“休眠状态”的连接来提高训练效果。该策略被称为“连接增长算法”,其核心思想是在保证训练误差下降的前提下,尽可能地激活那些能够快速降低误差的“休眠连接”。在训练开始时,神经网络中只有少量的活跃连接用于梯度传播,因此需要寻找那些“休眠连接”并计算它们对误差的贡献,以此来确定哪些连接应该被激活。这个策略有助于避免局部最小值,并提高准确性。从神经科学的角度来看,这个策略与赫布理论相吻合,即“一起激活的神经元会相互连接”。在具体实现中,通过计算某个神经元的前后连接之间的梯度大小来确定是否激活该连接。这个梯度大小也是连接增长算法中用来衡量连接重要性的指标。
相关问题
请帮我翻译一下:The algorithm in Kingma & Ba, 2015 uses such an update in practice, although their analysis does not
和 Welling's paper proposed a novel way for training generative models using stochastic gradient descent.
答案:Kingma 和 Welling 的论文中提出了一种新的使用随机梯度下降训练生成模型的方法的算法。
Please use the method in DSelection to find the median of medians recursively on the following array A in groups of 5.
As an AI language model, I cannot see the input array A to perform the required operation. However, I can explain the method of DSelection to find the median of medians recursively on an array A in groups of 5.
The DSelection algorithm is a modified version of the QuickSelect algorithm that uses the median of medians to select a pivot element. The median of medians is found by recursively dividing the array into groups of 5 and finding the median of each group. This process is repeated until a single median element is found. This median element is then used as the pivot for the QuickSelect algorithm.
Here are the steps to find the median of medians recursively on an array A in groups of 5 using the DSelection algorithm:
1. Divide the array A into groups of 5 elements each.
2. Find the median of each group by sorting the group and selecting the middle element.
3. Create a new array B containing all the medians found in step 2.
4. Recursively apply steps 1-3 on array B until a single median element is found.
5. Use the median element found in step 4 as the pivot for the QuickSelect algorithm.
By using the median of medians as the pivot, the DSelection algorithm ensures that the worst-case time complexity of QuickSelect is O(n). This makes it a useful algorithm for finding the kth smallest element in an unsorted array.
相关推荐
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)