Image credit: Popular Science, Modern Mechanix
The trick to finding a needle in a haystack is not brute force (like Jim Moran above), but a clever sorting mechanism -- whether a metal detector, or some sifting machine, or another unique scheme.
In astronomy, some of the most interesting objects are some of the rarest. This is often because the interesting objects are those that are changing rapidly (at least in a cosmic sense). For example, stars like the sun are everywhere in the sky, because they live a middle-age life for ten billion years, nearly the entire age of the Universe. But stars ending their lives, making beautiful planetary nebulae, are very rare -- the phase only lasts ten thousand years, a blink in the cosmic eye. So, compared to stars like the sun, planetary nebulae are rare. Another rare object is a supernova, or exploding star. These are only visible for a year or so before fading from sight, and the stars that make them are rare. So, in our skies right now, there are no supernovae in the Milky Way galaxy. We have to look at distant galaxies to find supernovae.
But planetary nebulae and supernovae are pretty easy to find, relatively speaking. Planetary nebulae are big and glow in very specific colors of light, so you can design a search to take pictures of the sky in those colors, and you'll find lots of planetaries. Supernovae are very bright, and so can be seen far away. So, we just look at more and more distant galaxies until we see a supernova -- in essence, we are searching hundreds of billions of stars at once, looking for a "new", bright star in a galaxy. Both of these are clever ways of searching through the haystack of the sky for that elusive needle.
I am part of a group at the University of Texas Astronomy Department that is looking for a specific kind of white dwarf star in a specific patch of the sky. White dwarfs are faint, and bright ones are rare (because they have to be close by). And the patch of sky we are looking in is large, by astronomy standards -- about the size of both of your hands held at arms' length. A typical astronomical camera can only image part of the sky as big as a part of your pinky finger's fingernail. Out of the hundreds of thousands of stars in that patch of the sky, we expect to find about ten of these white dwarfs. So, how can we find them?
The brute force method, like searching each strand of hay for the needle, is long and complex. We would have to take pictures of the entire region of the sky (dozens of nights on telescopes available to us). Then we would analyze the images and look for stars that have just the right colors. Then we go back to the telescope and take spectra of all of those stars, splitting the light up into its component colors, and determining what each one is. We then would have to analyze each spectrum. And, lastly, we have to return to the telescope to double-check each of our best candidates. So, we're talking a few weeks (at least!) on the telescope, and hundreds of person-hours to find ten objects.
Thankfully, there is an easier way -- an automated sorting machine known as the Sloan Digital Sky Survey. The Sloan survey has taken images of one-quarter of the sky, and automated software analyzed each object. Interesting objects are targeted for follow-up work, such as the spectra, which automated routines then analyze. As of right now, the Sloan database has image analysis of 280 million stars and galaxies, and spectra of 1.2 million of those objects.
So, yesterday afternoon I sat at a coffee shop and used the Sloan database query tools to find a dozen interesting white dwarfs in our patch of sky. A grand total of less than one person-hour went into my search. Granted, Sloan has used hundreds of nights of telescope time and countless thousands of person-hours to create and maintain the database. But that enormous undertaking allows small teams such as the team I'm part of to do searches that used to be prohibitively time-intensive. Computers and clever software allow a single person to search through 280 million objects for the dozen he or she is interested in, and those searches only take minutes, not years.
So, I found the needles I was looking for in the haystack of the sky, all thanks to computers and a giant team dedicated to making this possible.