![](https://csdnimg.cn/release/download_crawler_static/4143297/bg10.jpg)
3.3 Performance Analysis 3 ANALYSIS
The backpropagation algorithm will have to visit all connections, this cannot
be changed and it is therefore not possible to change the running time of the back-
propagation algorithm. However, as described in section 2.3, other more advanced
algorithms exists which could get better results than the backpropagation algo-
rithm. These algorithms do not execute faster than the backpropagation algorithm,
but they adjust the weights more precise, making them reach a result faster.
I have chosen to implement the backpropagation algorithm, because it is simple
and effective enough in most cases. This decision means that I have knowingly not
implemented an important optimization for the training algorithm, which implies
that there is not much use in spending too much time on the other optimization
strategies, because a highly tuned backpropagation algorithm will still be slower
than an untuned RPROP algorithm. In spite of that, a basic level of optimization
is still a desirable feature in the implementation of the backpropagation algorithm.
In conclusion; not much is done about the algorithms (although something could
be done about the training), which means that the running time is still Θ(n), where
n is the number of connections. However, there is still room for optimization of the
overhead involved in executing the actual calculations.
3.3.2 Architectural Optimization
There are many ways of building the architecture (data structures) for a neural
network. The object oriented approach would be to make everything an object and
there are actually good abstract concepts like neurons, synapses etc. which would
make for a great class hierarchy. In Jet’s Neural Library [Heller, 2002] such an
approach has been chosen, with all the advantages and disadvantages of this choice.
There are several major disadvantage of this approach:
• Data itself are not located closely together and cache performance is very bad.
• Algorithms like executing the network has code located in several different
classes, which makes the code hard to optimize and adds an overhead on
several key functions.
• It is difficult to make tight inner loops.
These are obviously problems that could be fixed, while still using the object
oriented approach, but the object oriented approach makes it difficult to do so.
A good architecture for a neural network should not take up too much space and
should not include too deep a level of objects. On the other hand some level of object
abstraction is highly desired. Perhaps a three level hierarchy would be acceptable,
with the outer level consisting of the entire ANN, the next level consisting of the
individual layers and the last level consisting of the single neurons and connections.
A good architecture will also allow for easy access to information like total
number of neurons etc.
3.3.3 Cache Optimization
If a good data architecture is in place much of the work for the cache optimization
is already done. But some work still needs to be done in improving the architecture
and making sure that the algorithms themselves are cache aware.
The architecture should assure that data could be accessed sequentially for good
cache performance. A good example of this is the weights, which should be accessed
sequentially when executing the network. For this reasons the weights should be
aligned in memory in one long array, which could be accessed sequentially.
The algorithms themselves should obviously use this optimized architecture and
access the data sequentially. The algorithms should also assure that all the code,
12