Deep Gaussian Mixture Variational Information Bottleneck

Abstract

On supervised learning tasks, introducing an information bottleneck can guide the model to focus on the more discriminative features in the input, which can effectively prevent overfitting. The deep variational information bottleneck aims to learn a global Gaussian latent variable using the neural network, which compresses mutual information with the input features and retains the most relevant information with the output features as much as possible. However, for the input containing complex semantic information, such as multi-label classification datasets, the latent variable obeying a simple Gaussian distribution may not necessarily capture rich representations. To alleviate this drawback, in this paper, we propose a new approach called Gaussian Mixture Variational Information Bottleneck (GMVIB) where the latent variable follows Gaussian mixture distribution. Then we generate Multi-MNIST and Multi-FashionMNIST, the multi-label classification datasets based on MNIST and FashionMNIST. Our experiments on these datasets show that the proposed approach learns a more efficient embedding representation and achieves competitive performance.