Tuesday, July 29, 2008

The science of conflicting results

One of the things about science that tends to get the public most frustrated are conflicting results. As a fictional example, one day you might read a story about how eating sharp cheddar cheese can prevent stomach ulcers and reduce your cholesterol. Then a month later, after adding copious amounts of sharp cheddar to your diet, you read that another group claims that sharp cheddar cheese may help with ulcers, but does nothing for cholesterol. And another two months later, you might read that the sharp cheddar worsens both ulcers and cholesterol. And yet later you read that, no, cheddar is fine after all.

In astronomy, one of the places this often pops up is with water on Mars. One month you can read that Mars used to have a lot of water and was once suitable for life, another month you'll read that it had a little water that was highly unsuitable for life, and another time you'll read that Mars never had much water and has always been a barren planet. Arrgh!

I saw another clear example over the last two nights. I was watching Shark Week on the Discovery Channel. On Sunday, I saw a Mythbusters episode that examined many shark myths. Two myths they claimed to confirm were that (a) flashlights attract sharks, and (b) thrashing in the water attracts more sharks than playing dead. Then, last night, I saw the overly-dramatic "Surviving Sharks" episode, which found that (a) reef sharks are repelled by bright lights at night, and (b) playing dead is more likely to get you eaten than thrashing about. What gives?

As frustrating as it may seem, in all cases like this you are seeing science at work. Science is so much more than running a study and seeing the result. It also involved making sure that the experiment is properly designed, replicating results with a similar experiment, and replicating results with a different experiment. And these processes take time and effort from many groups, who often disagree (sometimes quite vocally). The science-savvy person has to learn some patience, in the belief that time will tell us what the real conclusions are.

When conclusions disagree, it can indicate many things. Maybe an experiment was not properly designed. Or maybe some factor that the experimenters did not control is important in determining the result. For example, in many medical studies, disagreements arise over the size of a study (the number of people looked at) and the makeup of a sample. Perhaps the fictional experiment that finds that eating copious quantities of cheddar cheese lowers you cholesterol is flawed, because a higher fraction of the people in that study exercise regularly.

In the case of Mars, many of the disagreements arise over soil chemistry. On Earth, there are many minerals in rocks that are known to form in the presence of water; since water is and always has been abundant on Earth, we assume those minerals are formed in water. But does that translate to Mars? Are there ways to make certain minerals without an abundance of water? Could just a humid day have the same effect? In some cases the answer is yes, in some cases the answer is no, and in some cases, we don't know. Similar arguments rage over whether water is needed to carve certain land features on Mars, or whether wind, lava, and even liquid carbon dioxide could do the same thing. These are the boundaries of Mars science. Mistakes will be made, hypotheses formed and rejected, and results of space missions debated for years to come until one idea proves itself to stand up against all challenges.

As for the sharks, there are some obvious differences in the experiments that could explain the contradictory results. If you get a chance to see both the Mythbusters and Surviving Sharks, take a close look and try and figure out what the differences are that may matter. What about both experiments are the same? What is different? Can you think of ways of improving the experiments to control more of the variables?

These are also questions you can ask yourself whenever you see conflicting science results. A lot of time you don't get enough information in a news article to answer such questions. But rest assured that other scientists are looking into these issues.

No comments:

Post a Comment