# Inference of Latent Class Models

In Infinite Latent Feature Models and the Indian Buffet Process, the authors point out that the Gibbs sampling inference in a latent class model depends on the following full conditional distribution of latent class $c_i$ (Equation 17):

$P(c_i = k | C_{-i}, X) \propto p(X|C) P(c_i=k | C_{-i})$

The left-hand side of this equation is a general form. For LDA, it is $P(z_i=k | Z_{-i}, W)$.

I am interested in the right-hand side, because it looks not trivial to compute $p(X|C)$. Do we really need to compute it? I do not think so, because $z_i$ is independent with $W_{-i}$ given $Z_{-i}$.

An example of above statement is the derivation of LDA’s full conditional distribution, where we did the following expansion:
$P(z_i = k | Z_{-i}, W) = \frac{P(z_i, Z_{-i}, w_i, W_{-i})}{P(Z_{-i}, w_i, W_{-i})} = \frac{P(z_i, Z_{-i}, w_i, W_{-i})}{P(Z_{-i}, W_{-i})}$

In the right-most part of the equation, $w_i$ is omitted because given $Z_{-i}$, it is only possible to generate $W_{-i}$ but not $w_i$.

Continuing the derivation, we have

$\frac{P(Z, W)}{P(Z_{-i}, W_{-i})} = P(z_i, w_i | Z_{-i}, W_{-i}) = p(w_i|z_i) p(z_i | Z_{-i})$

This leads to the LDA Gibbs sampling rule appearing in the literature.  So Eqn. 17 can be rewritten as

$P(c_i = k | C_{-i}, X) \propto p(x_i | c_i) P(c_i=k | C_{-i})$

Advertisements