Vanishing Gradient Problem: An Activation Function that Seeks to Improve VGP

Abstract

The performance of deep learning networks can easily be affected by the phenomenon of vanishing gradients. With this in mind, this paper mainly discusses an activation function different from ReLU and other functions that improve the vanishing gradient problem (VGP). Influenced by parameterized functions and equations, we can set an activation function that allows for manual parameter changes. In this way, we can avoid the vanishing gradient situation when the input received by backpropagation is close to 0. With such an activation function, we can manually avoid vanishing gradients; in the experimental part, this paper uses the MNIST dataset to verify the effectiveness of the proposed method. The experiments demonstrate that the proposed method can alleviate the vanishing gradient phenomenon to a certain extent.