RSQRT: An heuristic for estimating the number of clusters to report

  • Authors:
  • John Carlis;Kelsey Bruso

  • Affiliations:
  • Computer Science and Engineering, University of Minnesota, 4-192 Keller Hall, 200 Union St. SE, Minneapolis, MN 55455, USA;Unisys Corporation, 2470 Highcrest Rd, Roseville, MN 55113, USA

  • Venue:
  • Electronic Commerce Research and Applications
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Clustering can be a valuable tool for analyzing large data sets, such as in e-commerce applications. Anyone who clusters must choose how many item clusters, K, to report. Unfortunately, one must guess at K or some related parameter. Elsewhere we introduced a strongly-supported heuristic, RSQRT, which predicts K as a function of the attribute or item count, depending on attribute scales. We conducted a second analysis where we sought confirmation of the heuristic, analyzing data sets from the UCI machine learning benchmark repository. For the 25 studies where sufficient detail was available, we again found strong support. Also, in a side-by-side comparison of 28 studies, RSQRT best-predicted K and the Bayesian information criterion (BIC) predicted K are the same. RSQRT has a lower cost of O(log log n) versus O(n^2) for BIC, and is more widely applicable. Using RSQRT prospectively could be much better than merely guessing.