Inference in Finite Mixture Models by Gibbs Sampling and Sum-Product

By Gibbs Sampling
Click the thumbnail to the left to download a 3-page note on inference in finite mixture model using Gibbs sampling. This note uses an example of estimating a mixture of “circles” in images, which, I thought about on my way to office by the subway. LaTeX source of this note is here.

Indeed, the posterior distribution of latent variables are easy to estimate, and there is no need to do approximate inference using Gibbs sampling. (This is also the reason people learn finite mixture models using EM.) What I wrote in this note is a preliminary step before understanding inference in infinite mixture model using Gibbs sampling.

By Sum-Product
Denote the likelihood of a mixture model by

p(x) = \sum_{k=1}^K \theta_k P_k(x)

where θ={θ1, … θK} is the prior of latent component indicators, and Pk(x) is the k-th component.
Given N data points X={x1, … xN}, and denoting the latent component indicators by Z={z1, … zN}, the factor graph contains N disjoint sub-graphs, each is a chain Ri–zi–Li, where Ri is a function representing the prior of zi: Ri(zi)=θzi; and Li is a function representing the likelihood given zi: Li(zi)=Pzi(xi).