Jump to content

EM algorithm and GMM model

From Wikipedia, the free encyclopedia

In statistics, EM (expectation maximization) algorithm handles latent variables, while GMM is the Gaussian mixture model.

Background

[edit]

In the picture below, are shown the red blood cell hemoglobin concentration and the red blood cell volume data of two groups of people, the Anemia group and the Control Group (i.e. the group of people without Anemia). As expected, people with Anemia have lower red blood cell volume and lower red blood cell hemoglobin concentration than those without Anemia.

GMM model with labels

is a random vector such as , and from medical studies[citation needed] it is known that are normally distributed in each group, i.e. .

is denoted as the group where belongs, with when belongs to Anemia Group and when belongs to Control Group. Also where , and . See Categorical distribution.

The following procedure can be used to estimate .

A maximum likelihood estimation can be applied:

As the for each are known, the log likelihood function can be simplified as below:

Now the likelihood function can be maximized by making partial derivative over , obtaining:

[1]

If is known, the estimation of the parameters results to be quite simple with maximum likelihood estimation. But if is unknown it is much more complicated.[2]

GMM without labels

Being a latent variable (i.e. not observed), with unlabeled scenario, the Expectation Maximization Algorithm is needed to estimate as well as other parameters. Generally, this problem is set as a GMM since the data in each group is normally distributed. [3][circular reference]

In machine learning, the latent variable is considered as a latent pattern lying under the data, which the observer is not able to see very directly. is the known data, while are the parameter of the model. With the EM algorithm, some underlying pattern in the data can be found, along with the estimation of the parameters. The wide application of this circumstance in machine learning is what makes EM algorithm so important.

EM algorithm in GMM

[edit]

The EM algorithm consists of two steps: the E-step and the M-step. Firstly, the model parameters and the can be randomly initialized. In the E-step, the algorithm tries to guess the value of based on the parameters, while in the M-step, the algorithm updates the value of the model parameters based on the guess of of the E-step. These two steps are repeated until convergence is reached.

The algorithm in GMM is:

Repeat until convergence:

   1. (E-step) For each , set
   
   2. (M-step) Update the parameters
   
      
      

[1]

With Bayes Rule, the following result is obtained by the E-step:

According to GMM setting, these following formulas are obtained:

In this way, a switch between the E-step and the M-step is possible, according to the randomly initialized parameters.

References

[edit]
  1. ^ a b Ng, Andrew. "CS229 Lecture notes" (PDF).
  2. ^ Hui, Jonathan (13 October 2019). "Machine Learning —Expectation-Maximization Algorithm (EM)". Medium.
  3. ^ Tong, Y. L. (2 July 2020). "Multivariate normal distribution". Wikipedia.