Migrating a Multiterabyte Archive from Object to Relational Databases
Computing in Science and Engineering
Category detection using hierarchical mean shift
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
| Hi-index | 0.00 |
Astronomy is about to undergo a major paradigm shift, with datasets becoming larger and more homogeneous, designed for the firsttime in a top-down fashion. In a few years, it might be much easierfor astronomers to "dial-up" a part of the sky, when they need arapid observation, rather than wait for several months to access a(sometimes quite small) telescope. With several projects inmultiple wavelengths underway-such as the SDSS, Galex, 2MASS,GSC-2, POSS2, Rosat, First, and Denis projects, each surveying alarge fraction of the sky-the concept of having a digital sky, withmultiple, terabyte-size databases interoperating seamlessly nolonger seems outlandish. As more and more catalogs are added andlinked to the existing ones in coming years and query enginesbecome more sophisticated, astronomers will have to be just asfamiliar with mining data as with observing on telescopes.As amajor part of that effort, the Sloan Digital Sky Survey willdigitally map about half of the northern sky in five filter bandsfrom ultraviolet to near infrared; we expect it to detect over 200million objects in this area. Simultaneously, the project willmeasure redshifts for the brightest 1 million galaxies. (A redshiftis the spectral displacement of a celestial body toward longerwavelengths caused by the Doppler effect or the source'sgravitational field; the higher the redshift value, the moredistant the object from Earth, hence the farther back in time.) Indoing so, the SDSS will revolutionize astronomy, increasing theamount of information made available to researchers by severalorders of magnitude. The resultant archive available for scientificresearch will be large (exceeding several terabytes) andcomplex-including textual information, derived parameters,multiband images, and spectra. The catalog will let astronomersstudy the universe's evolution in greater detail and should serveas the standard reference for the next several decades. As thisarticle describes, the potential scientific impact of the survey isstunning, but to realize its potential, data must be turned intoknowledge. As I'll indicate, this is not easy, for the survey'sinformation content will be several times larger than the entiretext contained in the Library of Congress.