Title

2019 ICLR Variational autoencoders with jointly optimized latent dependency structure

Problem

• latent variable models are often formulated with simple (e.g. Gaussian) distributions, making independence assumptions among the latent variables. That is, each latent variable is sampled independently. Ignoring the dependencies between latent variables limits the flexibility of these models, negatively impacting the model’s ability to fit the data.

  • e.g. VAEs are typically implemented with basic graphical structures and simple, unimodal distributions (e.g. Gaussians). Independence assumptions such as these may be overly restrictive, thereby limiting modeling capabilities.

• Models with these more flexible latent dependency structures have been shown to result in improved performance. However, despite the benefits of including additional structure in these models, their dependency structures have so far been predefined, potentially limiting the performance of this approach.

• Structure learning is a difficult task with a long history in the graphical models community. Unfortunately, the underlying objectives are often limited to graphs of a particular form (e.g., limited tree width), prohibitively expensive, or difficult to integrate with the gradient-based optimization techniques of modern neural networks.

Motivation

Rather than relying on pre-defined fully-connected structures (Kingma et al., 2016) or chain structures (Sønderby et al., 2016), authors seek to automatically learn the latent dependency structure as part of the variational optimization process.

Method

In this work, authors propose a method for learning dependency structures in latent variable models. In particular, authors introduce a set of binary global variables to gate the latent dependencies. The whole model (including its structure) is jointly optimized with a single stochastic variational inference objective.

• 横向比较

• VAE中，各个隐变量之间满足平均场假设。

• Ladder VAE中，各个隐变量与其上一个隐变量相关，类似于一阶马尔科夫链。条件概率分布是高斯分布，其参数由神经网络得到。

• Graph VAE中，隐变量的组织形式更为灵活。但是各隐变量也只是依赖于之前产生的隐变量，而不包括之后的。

• 内部探究

• 在图中节点数量有限的情况下，图结构的所有可能性是可数的，所有的可能性都是全连接图的子图。因此可以通过引入一个{0， 1}的指示变量 $c_{ij}$ 表示节点i和j之间是否有依赖关系。

• 本文假定隐变量与其父节点之间的条件分布$p_{\theta}\left(\mathbf{z}_{n} | \mathbf{z}_{\mathrm{pa}(n)}, \mathbf{c}_{\mathrm{pa}(n), n}\right)$为高斯分布，那如何计算参数$\widehat{\psi}_{n}=\left(\widehat{\boldsymbol{\mu}}_{n}, \widehat{\mathbf{\Sigma}}_{n}\right)$？通过递归的将各个父节点递归的相乘，并乘上各自对应的指示变量 $c$，再输入MLP后得到。即图中的 top-down inference 部分。

• 另一方面，除了计算隐变量之间的条件分布以外，我们还要计算模型的后验分布$p_{\theta}(\mathbf{z} | \mathbf{x}, \mathbf{c})$ ，VAE的做法都是提出一个提议分布$q_{\phi}(\mathbf{z} | \mathbf{x}, \mathbf{c})=\prod_{n=1}^{N} q_{\phi}\left(\mathbf{z}_{n} | \mathbf{x}, \mathbf{z}_{\mathrm{pa}}(n), \mathbf{C}_{\mathrm{pa}(n)}\right)$ 做近似。这里就是将数据x输入encoder得到一些feature，然后在输入一个MLP计算出提议分布的参数$\widetilde{\psi}_{n}$，即图中的bottom-up inference部分。

• 最后利用一个precision-weighted fusion将两个分布联合起来得到最后的隐变量分布：

• 还有一个问题是如何确定 $c_{ij}$ 的取值。因为 $c_{ij}$ 是离散的，因此需要用一些技巧使得能够对它求导。这里作者用的是 Gumbel-Softmax estimator from Jang et al. (2017)：

$c_{i, j}=\frac{\exp \left(\left(\log \left(\mu_{i, j}\right)+\epsilon_{1}\right) / \tau\right)}{\exp \left(\left(\log \left(\mu_{i, j}\right)+\epsilon_{1}\right) / \tau\right)+\exp \left(\left(\log \left(1-\mu_{i, j}\right)+\epsilon_{2}\right) / \tau\right)}$

这里实际上将离散变量 $c_{ij}$ 转换为了连续变量 $\mu_{ij}$， 使得可以求解梯度。在优化的过程中，使用了退火算法的思想，当 $\tau$ 大时，$c_{ij}$ 较为平滑，当 $\tau$ 小时，$c_{ij}$ 趋于离散。通过调节 $\tau$ ，我们学习 $\mu_{ij}$ 的值。

Experiment

• Analysis on latant variables' structure and node dimensions

• Quantitative Comparations

Conclusion

Authors presented a novel method for structure learning in latent variable models, which uses dependency gating variables, together with a modified objective and discrete differentiation techniques, to effectively transform discrete structure learning into a continuous optimization problem.

In our experiments, the learned latent dependency structures improve the performance of latent variable models over comparable baselines with predefined dependency structures.

Code

https://github.com/ys1998/vae-latent-structure