We virtually add a few observations to each component
: causes the weights to never be zero
Dirichlet distribution:
: adds some regularization on the variances
… unfold
Dirichlet Processes
From GMM to DPGMM
Plain GMM
the number of component needs to be know
need to try multiple ones and do model selection
(prior are not handled by EM)
Dirichlet Process GMM
a GMM with an infinity of components
with a proper prior
cannot use EM for inference (infinite vectors)
What is a Dirichlet Process, finally
Dirichlet Process
What?
a distribution over distributions
a prior over distributions
two parameters
a scalar , the “concentration”
a “base” distribution (any type)
a draw from a DP is a countably infinite sum of Diracs
Related formulations
Definition: complicated :)
Stick breaking process (GEM)
as shown on the graphical model
e.g., a prior on the values of the weights
Chinese Restaurant Process (CRP)
how to generate the , one by one, in sequence
Polya Urn
Definition Example (Wikipedia)
Draws from the Dirichlet process DP(N(0,1), alpha). Each row uses a different alpha: 1, 10, 100 and 1000. A row contains 3 repetitions of the same experiment.
Stick breaking process (GEM)
Each “atom” drawn from
Infinite vector drawn from a GEM process
;
;
;
...
and
Denoted
then is a draw from
(GEM for Griths, Engen and McCloskey)
Polya Urn?
Chinese Restaurant Process (CRP)
Gibbs sampling friendly
easy to get
1 / 31 − Rémi Emonet − Infinite Mixture Models with Dirichlet Process