Friday, June 10, 2005
Misappropriating the Method
I regularly find it hard to believe that we can live in the most technologically advanced society that the world has ever known, yet still be completely out of touch with the true nature of science. Amazing, yes -- but impossible to deny. We consume the fruits of this process without the slightest understanding of how these fruits are cultivated. And because we don't know what science is, we often fail to recognize the impostors.
Over the last couple of weeks, I've run across two excellent examples of this phenomenon. Not only are these examples educational, they're also pretty darn funny. So, let's take a look, shall we?
Last Friday I got caught up watching Bill O'Reilly amidst my mindless channel surfing. He had teased the upcoming segment by claiming that he would have psychiatrists dueling it out over the subject of gay marriage. Well, for me, that's a lock. Seems that I have a bit of a thing for psychology. Plus, I couldn't imagine a reputable psychological argument against gay marriage. How could I pass that up?
Unfortunately, the debate failed to live up to the hype. As it turned out, the disagreement stemmed from the American Psychiatric Association's decision to take a position with respect to gay marriage. One shrink thought it was OK for the APA to take positions on political issues, one did not. Color me bored.
Apparently, Bill was similarly dissatisfied with what was shaping up to be a procedural discussion. His solution was to attack the science that justified the APA's position.
However, behind all the hilarity is an important point. Notice the usage of the word "study" in the previous quote. Bill uses it three times to refer to research that supposedly challenges the conclusions of the APA. Dr. Stotland also uses the term, this time to ask Bill what the hell he's talking about. Clearly these two, and the larger television audience, understand the implication of this word. A "study" is a scientifically rigorous examination of a given phenomenon. Moreover, because such an investigation is scientifically rigorous, we are all expected to give its conclusions some credence. This isn't just some idiot mouthing off on the op-ed page. This is objective science.
In reality, though, one of these two people actually understands what a study really is, and it isn't Bill O'Reilly.
Immediately after Bill referred to this "study", I started poking around on the Internet in order to learn more about it. Like Dr. Stotland, I had never heard of this research and I was interested to discover what, if any, merit it had. While doing so I came across several sources that do a number on the conclusions of this "study" (here and here). But beyond the questionable analysis and logical fallacies, a far more dramatic fact becomes clear.
You see, it's hardly surprising that Dr. Stotland had never heard of the study when you realize that it doesn't exist! That's right. In fact, the "study" to which Bill refers is actually this article published in The Weekly Standard.
So, either Bill O'Reilly is woefully misinformed (a definite possibility) or he has absolutely no idea what makes a study scientific. You see, any idiot can hole up in his living room, conduct experiments, and draw conclusions. He can even write up those conclusions, sprinkling his document with a thousand SAT words mined out of Roget's Thesaurus. But, it really isn't science until his findings have undergone a fair degree of peer review. Generally this means submitting your work to an established scientific journal where your methods and analysis are examined by others conversant in the field. If they believe your work to be sound, they will permit its publication. It is only at this point that it becomes a "study" in the sense we have come to understand.
Before that, you're just an idiot with too much time on your hands.
I uncovered this second example while reading about hearings held recently by the Kansas State Board of Education considering the introduction of Intelligent Design into the state's scientific curriculum. Of course, the ID movement presents a rather "target-rich" environment for those of us looking to skewer scientific frauds. But this little anecdote made an impression upon me. Massimo Pigliucci, Ph.D., is an evolutionary biologist teaching at the State University of New York, Stony Brook. As such, he occasionally duels with ID advocates who claim that their theory should be considered alongside Darwin's. Apparently, this led him to once cross swords with William Dembski, a leading Intelligent Design proponent. This event provided an incredibly revealing moment, which Pigliucci describes as follows.
Yet, he strangely has no research that he'd like to conduct. That's pretty darn unusual. Any scientist worth her salt has a million ideas brewing inside her head that she would test at the first opportunity. For most scientists, formulating experiments isn't the hard part -- paying for it is. With Dembski, not so much. And this is, of course, the problem with the entire ID movement. It's literally drowning in cash, but spends all of its money on advocacy -- and none on research!
As they say, if you want to understand something, follow the money.
While it's fun and all to expose the embarrassing ignorance on display in these two examples, I do have a more profound point to make. As I mentioned earlier, we live in a highly technical society. Our existence would have a dramatically different flavor without the benefits that science has provided to us. And while we have paid a price for some of this scientific advancement, a great deal of it has been a boon to our society. We are made aware of this fact every single day.
Moreover, the record of success that science has posted over the last 500 years has given us all reason to trust it. Faith typically isn't something we associate with science, yet for nonpractitioners that's basically what their enthusiasm for science amounts to. Again and again we have seen science come to the rescue in order to free us from the practical struggles of existence. Over time, we have learned that our faith in science is warranted.
Due to our societal trust in science, advocates of all stripes have strived to employ it in service of their goals. People tend to listen to and trust scientists, therefore having one on your side has obvious benefits. Advocates lace their pitches with references to data and scientific analysis, hoping that this will increase the persuasiveness of their argument. For years, the tobacco industry supported narrowly tailored research so that they might scientifically buttress their claim of the cigarette's harmlessness. The petroleum industry does likewise to deflect concerns over global climate change. Even groups that are openly hostile to science will use it to advance their positions when it suits them. The persuasive power of science is too powerful to pass up.
However, just calling something science doesn't make it science. It is a very specific process that has evolved in order to avoid error, confirmation bias, and outright chicanery. If you don't follow the method, your results don't deserve the veneration we typically reserved for truly scientific conclusions. Without the method, you are trusting the investigator, not science.
But most people aren't aware of these fine distinctions and O'Reilly, the ID movement, and many, many others realize this. They understand that they will win converts by scientifically backing their position. If they are unable to do this honestly, they will simply produce fraudulent claims and label them scientific, fully aware that few will see through the charade. It is deeply dishonest, but its effectiveness is proven and therefore they cannot eschew the practice.
That is, until such a time as the general public comes to understand that "science" is more than a word, more than a test tube, and more than a lab coat. Once we collectively begin to understand this fact and challenge those who improperly don the mantle of science, this practice will wither away.
But, that may take some time. Until then, keep an eye peeled for advocacy dressed up as science. Clothes may make the man, but such trappings do little for poor research. It might look great on the outside, but if it isn't science all the way through, it's just junk.
No matter what you call it.
Over the last couple of weeks, I've run across two excellent examples of this phenomenon. Not only are these examples educational, they're also pretty darn funny. So, let's take a look, shall we?
Last Friday I got caught up watching Bill O'Reilly amidst my mindless channel surfing. He had teased the upcoming segment by claiming that he would have psychiatrists dueling it out over the subject of gay marriage. Well, for me, that's a lock. Seems that I have a bit of a thing for psychology. Plus, I couldn't imagine a reputable psychological argument against gay marriage. How could I pass that up?
Unfortunately, the debate failed to live up to the hype. As it turned out, the disagreement stemmed from the American Psychiatric Association's decision to take a position with respect to gay marriage. One shrink thought it was OK for the APA to take positions on political issues, one did not. Color me bored.
Apparently, Bill was similarly dissatisfied with what was shaping up to be a procedural discussion. His solution was to attack the science that justified the APA's position.
O'REILLY: Are you familiar with the Swedish study on marriage, ma'am?Now, it's a little easy to get distracted here because they're so much funny stuff going on. I mean, how can you not laugh when a bloviating TV host attempts to call a professional scientist ignorant in her own field? That's what I call comedy.
DR. NADA STOTLAND (vice president, American Psychiatric Association): No -- which study are you referring to?
O'REILLY: OK. They did a sociological study in Sweden that said that marriage between men and women declined drastically since gay marriage was legalized there. And now up to, I don't know, 60, 65 percent of all Swedes are not married. So the institution of marriage, basically, in that country collapsed, because there was no tradition to it.
And you know, I was wondering before the -- your organization made your decision to come out and say this is a good thing, whether you took a look at that study. [Emphasis mine]
However, behind all the hilarity is an important point. Notice the usage of the word "study" in the previous quote. Bill uses it three times to refer to research that supposedly challenges the conclusions of the APA. Dr. Stotland also uses the term, this time to ask Bill what the hell he's talking about. Clearly these two, and the larger television audience, understand the implication of this word. A "study" is a scientifically rigorous examination of a given phenomenon. Moreover, because such an investigation is scientifically rigorous, we are all expected to give its conclusions some credence. This isn't just some idiot mouthing off on the op-ed page. This is objective science.
In reality, though, one of these two people actually understands what a study really is, and it isn't Bill O'Reilly.
Immediately after Bill referred to this "study", I started poking around on the Internet in order to learn more about it. Like Dr. Stotland, I had never heard of this research and I was interested to discover what, if any, merit it had. While doing so I came across several sources that do a number on the conclusions of this "study" (here and here). But beyond the questionable analysis and logical fallacies, a far more dramatic fact becomes clear.
You see, it's hardly surprising that Dr. Stotland had never heard of the study when you realize that it doesn't exist! That's right. In fact, the "study" to which Bill refers is actually this article published in The Weekly Standard.
So, either Bill O'Reilly is woefully misinformed (a definite possibility) or he has absolutely no idea what makes a study scientific. You see, any idiot can hole up in his living room, conduct experiments, and draw conclusions. He can even write up those conclusions, sprinkling his document with a thousand SAT words mined out of Roget's Thesaurus. But, it really isn't science until his findings have undergone a fair degree of peer review. Generally this means submitting your work to an established scientific journal where your methods and analysis are examined by others conversant in the field. If they believe your work to be sound, they will permit its publication. It is only at this point that it becomes a "study" in the sense we have come to understand.
Before that, you're just an idiot with too much time on your hands.
I uncovered this second example while reading about hearings held recently by the Kansas State Board of Education considering the introduction of Intelligent Design into the state's scientific curriculum. Of course, the ID movement presents a rather "target-rich" environment for those of us looking to skewer scientific frauds. But this little anecdote made an impression upon me. Massimo Pigliucci, Ph.D., is an evolutionary biologist teaching at the State University of New York, Stony Brook. As such, he occasionally duels with ID advocates who claim that their theory should be considered alongside Darwin's. Apparently, this led him to once cross swords with William Dembski, a leading Intelligent Design proponent. This event provided an incredibly revealing moment, which Pigliucci describes as follows.
I once debated ID proponent William Dembski at the NY Academy of Science, and pointedly asked him what he would do if he got a grant from the National Science Foundation. He couldn't come up with anything…Now remember, Dembski claims to be, first and foremost, a scientist. Unlike some, he is not a religious zealot. He's just searching for the truth -- just like every other man and woman wearing a lab coat.
Yet, he strangely has no research that he'd like to conduct. That's pretty darn unusual. Any scientist worth her salt has a million ideas brewing inside her head that she would test at the first opportunity. For most scientists, formulating experiments isn't the hard part -- paying for it is. With Dembski, not so much. And this is, of course, the problem with the entire ID movement. It's literally drowning in cash, but spends all of its money on advocacy -- and none on research!
As they say, if you want to understand something, follow the money.
While it's fun and all to expose the embarrassing ignorance on display in these two examples, I do have a more profound point to make. As I mentioned earlier, we live in a highly technical society. Our existence would have a dramatically different flavor without the benefits that science has provided to us. And while we have paid a price for some of this scientific advancement, a great deal of it has been a boon to our society. We are made aware of this fact every single day.
Moreover, the record of success that science has posted over the last 500 years has given us all reason to trust it. Faith typically isn't something we associate with science, yet for nonpractitioners that's basically what their enthusiasm for science amounts to. Again and again we have seen science come to the rescue in order to free us from the practical struggles of existence. Over time, we have learned that our faith in science is warranted.
Due to our societal trust in science, advocates of all stripes have strived to employ it in service of their goals. People tend to listen to and trust scientists, therefore having one on your side has obvious benefits. Advocates lace their pitches with references to data and scientific analysis, hoping that this will increase the persuasiveness of their argument. For years, the tobacco industry supported narrowly tailored research so that they might scientifically buttress their claim of the cigarette's harmlessness. The petroleum industry does likewise to deflect concerns over global climate change. Even groups that are openly hostile to science will use it to advance their positions when it suits them. The persuasive power of science is too powerful to pass up.
However, just calling something science doesn't make it science. It is a very specific process that has evolved in order to avoid error, confirmation bias, and outright chicanery. If you don't follow the method, your results don't deserve the veneration we typically reserved for truly scientific conclusions. Without the method, you are trusting the investigator, not science.
But most people aren't aware of these fine distinctions and O'Reilly, the ID movement, and many, many others realize this. They understand that they will win converts by scientifically backing their position. If they are unable to do this honestly, they will simply produce fraudulent claims and label them scientific, fully aware that few will see through the charade. It is deeply dishonest, but its effectiveness is proven and therefore they cannot eschew the practice.
That is, until such a time as the general public comes to understand that "science" is more than a word, more than a test tube, and more than a lab coat. Once we collectively begin to understand this fact and challenge those who improperly don the mantle of science, this practice will wither away.
But, that may take some time. Until then, keep an eye peeled for advocacy dressed up as science. Clothes may make the man, but such trappings do little for poor research. It might look great on the outside, but if it isn't science all the way through, it's just junk.
No matter what you call it.
