B0776
Title: Frugal Gaussian clustering of huge imbalanced datasets through a bin-marginal approach
Authors: Filippo Antonazzo - Inria (Austria)
Christophe Biernacki - Inria (France)
Christine Keribin - INRIA - Paris-Saclay University (France) [presenting]
Abstract: Clustering conceptually reveals all its interest when the dataset size considerably increases since there is the opportunity to discover tiny but possibly worthwhile clusters which were out of reach with more modest sample sizes. However, clustering is practically faced with computer limits with such high data volume, since it possibly requires extremely high memory and computation resources. In addition, the classical subsampling strategy, often adopted to overcome these limitations, is expected to fail heavily in discovering clusters in the highly imbalanced cluster case. Our proposal first consists in drastically compressing the data volume by just preserving its bin-marginal values, thus discarding the bin-cross ones. Despite this extreme information loss, we then prove the identifiability property for the diagonal mixture model and also introduce a specific EM-like algorithm associated with a composite likelihood approach. This latter is extremely more frugal than a regular but unfeasible EM algorithm expected to be used on our bin-marginal data, while preserving all consistency properties. Finally, numerical experiments highlight that this proposed method outperforms subsampling both in controlled simulations and in various real applications where imbalanced clusters may typically appear.
