Computing gradient vector and Jacobian matrix in arbitrarily connected neural networks

Wilamowski B. M., Cotton N. J., Kaynak O., DÜNDAR G.

IEEE Transactions on Industrial Electronics, cilt.55, sa.10, ss.3784-3790, 2008 (SCI-Expanded, Scopus) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 55 Sayı: 10
  • Basım Tarihi: 2008
  • Doi Numarası: 10.1109/tie.2008.2003319
  • Dergi Adı: IEEE Transactions on Industrial Electronics
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus
  • Sayfa Sayıları: ss.3784-3790
  • Anahtar Kelimeler: Learning, Neural network
  • Boğaziçi Üniversitesi Adresli: Evet

Özet

This paper describes a new algorithm with neuronby-neuron computation methods for the gradient vector and the Jacobian matrix. The algorithm can handle networks with arbitrarily connected neurons. The training speed is comparable with the Levenberg-Marquardt algorithm, which is currently considered by many as the fastest algorithm for neural network training. More importantly, it is shown that the computation of the Jacobian, which is required for second-order algorithms, has a similar computation complexity as the computation of the gradient for first-order learning methods. This new algorithm is implemented in the newly developed software, Neural Network Trainer, which has unique capabilities of handling arbitrarily connected networks. These networks with connections across layers can be more efficient than commonly used multilayer perceptron networks. © 2008 IEEE.