.md .pdf

Contents

Mixture of multinoullis

A mixture model where the data consists of D-dimensional bit vectors. The class conditional density is a product of Bernoullis

$$p(x_i|z_i = k, \theta) = \prod_{j=1}^D Ber(x_{ij}| \mu_{ij})$$

• $\mu_{ij}$ is the probability that bit j turns on in cluster k.

The mean and covariance of this mixture model is given by:

$$E[x] = \sum_x \pi_k \mu_k$$

$$cov[x] = \sum_{k} \pi_k [\Sigma_i + \mu_k\mu_k^T] - E[x]E[x]^T$$

• $\Sigma_k = diag(\mu_{jk}(1 - \mu_{jk}))$

Here the mixture distribution can capture correlations between variables.

Clustering

We can use a mixture model to create a generative classifier, where we model each class-conditional density $p(x|y=c)$ by a mixture distribution.

We fit a mixture model, and compute $p(z_i = k| x_i, \theta)$ which represetns the posterior robability that the point i belongs to cluster k. This is known as the responsibility of cluster k for point i and can be computed as:

$$r_{ik} \triangleq p(z_i = k| x_i, \theta) = \frac{p(z_i = k | \theta) p(x_i| z_i = k, \theta)}{ \sum_{k' = 1}^K p(z_i = k' | \theta) p(x_i| z_i = k', \theta) }$$

This is also known as soft clutering. We can transform it into hard clustering by using the MAP estimate by approximating

$$\hat{z_i} = \arg \max_k r_{ik} = \arg \max_k \log p(x_i | z_i = k , \theta) + \log p(z_i = k | \theta)$$

Mixtures of experts

We can use mixtures models for discriminative classification or regression. In this example we fit 3 different linear regression functions, each applying to different part of the input space.

$$\begin{split} p(y_i | x_i, z_i = k, \theta) = N(y_i| w^Tk x_i, \sigma^2_k) \\ p(z_i|x_i,\theta) = Cat(z_i | S(V^Tx_i)) \end{split}$$

This is called a mixture of experts.

Area and Volume Computing the marginal likelihood (evidence)