User:Biserhong

Probabilistic principal component analysis (PPCA) formulates the principal component analysis procedure (PCA) as a maximum likelihood solution of a probabilistic latent variable model. When only the first few leading dimensions are needed, an EM algorithm can be used for efficiency instead of evaluating the sample covariance matrix. A Bayesian method can be utilized to automatically find the dimensionality of the principal subspace in situations where it is not clearly defined. The probabilistic PCA model can be used to sample from the distribution obtained by generating an observed variable in the original d dimensions from a linear transformation of a lower dimensional latent variable plus Gaussian noise ^[1]. Probabilistic PCA overcomes several problems of standard approaches to principal component analysis - it allows to deal with missing data, whereas normally incomplete points must be discarded, defines a proper density model, so that we can estimate whether new data points are fit well by the model, and provides an efficient way to deal with high-dimensional data. ^[2]

Probabilistic PCA is a generative latent variable model. It describes an observed d-dimensional data vector ${\displaystyle \mathbf {y} }$ as produced by a mapping of a q-dimensional latent (hidden) variable ${\displaystyle \mathbf {x} }$ plus shift to some mean vector ${\displaystyle \mathbf {\mu } }$ plus some additive Gaussian noise ${\displaystyle \mathbf {\epsilon } }$:

${\displaystyle \mathbf {y} =\mathbf {Wx} +\mathbf {\mu } +\mathbf {\epsilon } }$

Generally , q < d. Thus, the d × q matrix W denotes a linear transformation from the lower dimensional latent space to the higher dimensional data space. The parameter ${\displaystyle \mathbf {\mu } }$ allows the data model to have a mean different from zero. The latent variable ${\displaystyle \mathbf {x} }$ is distributed according to a normal distribution with zero mean and an isotropic covariance matrix ${\displaystyle I}$:

${\displaystyle x\ \sim \ {\mathcal {N}}(0,\,I).\,}$

Here, the matrix ${\displaystyle I}$ is q-dimensional (i.e. the latent variable has q dimensions) and is a design parameter that needs to be specified according to the application. The noise model is assumed to be independent of ${\displaystyle \mathbf {x} }$ and is also distributed according to a Gaussian distribution:

${\displaystyle \mathbf {\epsilon } \ \sim \ {\mathcal {N}}(0,\,\mathbf {\Psi } ).\,}$

The covariance matrix ${\displaystyle \mathbf {\Psi } }$ is diagonal, which means the observed variables ${\displaystyle \mathbf {y} }$ are conditionally independent given the latent variables ${\displaystyle \mathbf {x} }$. For a general matrix ${\displaystyle \mathbf {\Psi } }$ it is not possible to obtain a closed-form analytic solution. This is possible only if ${\displaystyle \mathbf {\Psi } }$ is of the form:

${\displaystyle \mathbf {\Psi } =\sigma ^{2}\mathbf {I} }$

Given the above formulation, and since the linear transformation W of the normally distributed random variable ${\displaystyle \mathbf {x} }$ is still normally distributed, the observed vectors ${\displaystyle \mathbf {y} }$ are distributed as follows:

${\displaystyle \mathbf {y} \ \sim \ {\mathcal {N}}(\mathbf {\mu } ,\,\mathbf {C} ),\,}$ where ${\displaystyle \mathbf {C} =\mathbf {WW^{T}} +\mathbf {\Psi } }$

where we have the term ${\displaystyle \textstyle \mathbf {WW^{T}} }$ because the covariance of the linear transformation of ${\displaystyle \mathbf {x} }$ is:

${\displaystyle \operatorname {Cov} (\mathbf {Wx} )=\mathbf {W} \operatorname {Cov} (\mathbf {x} )\mathbf {W^{T}} =\mathbf {WW^{T}} }$

Therefore, we obtain a probability distribution over the data space given a latent variable ${\displaystyle \mathbf {x} }$:

${\displaystyle p(\mathbf {y} |\mathbf {x} )\,=(2\pi \sigma ^{2})^{-d/2}\exp \left(-{\frac {1}{2\sigma ^{2}}}||{\mathbf {y} }-{\mathbf {W} }{\mathbf {x} }-{\mathbf {\mu } }||^{2}\right).}$

The we integrate over ${\displaystyle \mathbf {x} }$ to get the marginal distribution of ${\displaystyle \mathbf {y} }$:

${\displaystyle p(y)=\int p(\mathbf {y} |\mathbf {x} )\,p(\mathbf {x} )\,d\mathbf {x} =(2\pi )^{-d/2}|\mathbf {C} |^{1/2}\exp \left(-{\frac {1}{2}}({\mathbf {y} }-{\mathbf {\mu } })^{T}{\mathbf {C} }^{-1}({\mathbf {y} }-{\mathbf {\mu } })\right).}$

Using Bayes' rule we can also obtain the posterior distribution of the latent variable ${\displaystyle \mathbf {x} }$ ^[3]

• Advanced Vision homepage

Category:Image processing Category:Artificial intelligence