Asteroids do hit the Earth from time to time, and this causes severe problems for anything living on the Earth. But asteroids do not careen randomly through the solar system. They obey Newton's Law of Gravity, just like every other planet, and so their paths are very predictable. The danger to the Earth is not from known asteroids, but from one we don't know about. Still, astronomers are looking for potentially dangerous asteroids, such as Spacewatch, and we'll find them if they are there.
I once had a person email me after the close approach of another asteroid; this person was scared to go to sleep in case an asteroid hit the Earth while she was sleeping. I tried to tell her not to worry; I don't know if she actually believed me. We need to be vigilant, but we don't need to worry about asteroids.
Toutatis (here in an image made with radar in 1992) swings by the Earth every four years or so. Due to the gravitational pull of planets, it is not possible to figure out if Toutatis will ever hit the Earth, but we do know that it's not going to hit in the next few thousand years. Enough time for another cup of coffee before astronomers go out to save the world. :)