Improving Mixture of Experts Using Second-Order Optimisation

This paper presents a new algorithm to improve the learning algorithm of mixture of experts (ME) model by using conjugate gradient (CG) as a second-order optimisation technique. The CG technique is combined with back-propagation algorithm to yield an efficient learning algorithm for ME structure. The experts and gating network in the enhanced model is replaced with CG-based multi-layer perceptrons in order to provide faster and more accurate learning algorithm. The performance of proposed method is compared with gradient decent-based ME in several classification and regression problems. The results show that CG-based ME has faster convergence and better performance in the utilised benchmark datasets.