Title

2019 JMLR Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model

Problem

• There is a trade-off between sparsity and smoothness in Dirichlet distributions.

  • The Dirichlet distribution encourages the sparsity in the latent variables. However, authors find that the desired sparsity is at odds with the generalization performance of the model.
  • In the case of variational autoencoders (VAEs), the trade off is that:
    • Either the model is sparse and achieves a good reconstruction error, but has a high KL-divergence term and thus a low log-likelihood of the model
    • Or it avoids low settings of the prior and thereby achieves a better log-likelihood at the cost of a higher reconstruction error.

• There is no differentiable non-centered reparameterization function for the Dirichlet distribution.

Motivation

To identify the trade-off between sparsity and smoothness and provide an adapted neural network architecture to decouple them.

Method

• 横向比较

a 为Dirichlet分布的参数，b为稀疏性系数。DVAE和DVAE Sparse的区别在于前者的b为1。

• 内部探究

  • 模型结构

​ 这里引入batchnorm的原因是为了改善 Component Collapsing 问题。

• ELBO和梯度的推导，详看论文。

• VAE中的 Component Collapsing 问题。

  • What?

    This is the problem of a local minimum, which leads to parameters that are equal to the prior and thus, many or all topics are the same. Topic redundancy

  • Why?

    It occurs because it is often easier for the model to minimize the KL-divergence than to minimize the reconstruction error.

  • How to deal with it?

    • use much lower learning rates (NVDM)
    • annealing of the KL-divergence (Bowman et al., 2016)
    • batch normalization and dropout (Srivastava and Sutton, 2017, 2018)

Experiment

• 除了 perplexity 和 NPMI（coherence）以外，还引入了 topic redundancy（越低越好）：

• DVAE，coherence最高，但perplexity却非常高， topic redundancy一般。

• DVAE Sparse，coherence一般，但是perplexity与DVAE和其他基于近似Dirichlet分布的方法相比较低，且topic redundancy低。

• 分析：

  • 以Dirichlet分布为先验的模型在coherence上都好于以高斯分布为先验的模型（NVDM）。This further underlines the argument by Srivastava and Sutton (2017) that the sparsity induced by the Dirichlet indeed leads to higher topic coherence.
  • DVAE的perplexity却非常高是因为 KL-divergence 收敛到 0，则可以通过引入稀疏性系数，即DVAE Sparse来缓解。这表明了the coupling of sparsity and smoothness is responsible for the high perplexity scores. Also perplexity scores are not always meaningful when comparing different topic models.
  • 计算KL散度有解析式计算、采样计算两种方法。后者有利于增加coherence。

Conclusion

• Authors have introduced a variational autoencoder with a Dirichlet prior called DVAE.
• A trade-off between sparsity and smoothness was identified, leading to high perplexity as well as topic coherence scores.
• To test our hypothesis about the reason for the high perplexity scores, a new model called DVAE Sparse was introduced. DVAE Sparse decouples sparsity and smoothness and thus enables lower perplexity scores on the one hand while on the other hand the topic coherence is still competitive.
• The topic quality was also measured in terms of topic redundancy which is a good indicator for component collapsing as opposed to topic coherence.