Microclustering: When the Cluster Sizes Grow Sublinearly with the Size of the Data Set
Repository Usage Stats
254
views
views
93
downloads
downloads
Abstract
Most generative models for clustering implicitly assume that the number of data points
in each cluster grows linearly with the total number of data points. Finite mixture
models, Dirichlet process mixture models, and Pitman--Yor process mixture models make
this assumption, as do all other infinitely exchangeable clustering models. However,
for some tasks, this assumption is undesirable. For example, when performing entity
resolution, the size of each cluster is often unrelated to the size of the data set.
Consequently, each cluster contains a negligible fraction of the total number of data
points. Such tasks therefore require models that yield clusters whose sizes grow sublinearly
with the size of the data set. We address this requirement by defining the \emph{microclustering
property} and introducing a new model that exhibits this property. We compare this
model to several commonly used clustering models by checking model fit using real
and simulated data sets.
Type
Journal articlePermalink
https://hdl.handle.net/10161/11816Collections
More Info
Show full item recordScholars@Duke
Rebecca Carter Steorts
Associate Professor of Statistical Science
You can find more information about my research group and work at:https://resteorts.github.io/Recent
papers of mine can be found at https://arxiv.org/search/?query=steorts&searchtype=all&source=header

Articles written by Duke faculty are made available through the campus open access policy. For more information see: Duke Open Access Policy
Rights for Collection: Scholarly Articles
Works are deposited here by their authors, and represent their research and opinions, not that of Duke University. Some materials and descriptions may include offensive content. More info