Comparative Analysis of Performances of an Improved Particle Swarm Optimization and a Traditional Particle Swarm Optimization for Training of Neural Network Architecture Space

Abstract

Many studies evaluating the performance of various optimization methods for training Artificial Neural Networks (ANNs) have produced conflicting results. This discrepancy often arises due to the limited application of these methods across a narrow spectrum of ANN architectures and training parameter values. In response to this gap, our study introduces an enhanced Particle Swarm Optimization (PSO) technique, denoted as Reverse Direction Supported Particle Swarm Optimization (RDS-PSO), specifically designed for ANN training. RDS-PSO incorporates two novel parameters, namely alpha and beta, allowing the creation of four distinct RDS-PSO types including the original PSO. Unlike many existing studies, we comprehensively evaluate the performance of these four RDS-PSO types across a diverse set of criteria. These criteria include the architectural space of ANN, training depths for ANN, inertia weight direction for RDS-PSO, and adaptation approaches for the two novel parameters of RDS-PSO. Through 100 iterations for each training case, we conduct an extensive and intricate analysis of ANN training performance on three medical datasets. Our experimental findings reveal that RDS-PSO_3, featuring decreasing inertia weight and cosine adaptation, consistently outperforms other RDS-PSO types. Furthermore, RDS-PSO_3 demonstrates greater reliability, as evidenced by lower standard deviation values, across most ANN architectures.