STATISTICS AND THE MYSTIQUE OF SCIENCE
A May 7th article in the New York Times (you need to register to read the article) discusses a study about school vouchers that was used in the 2000 elections and subsequently by government policy makers to justify dismantling public education in favour of the voucher system. Because to read the article you must register with the New York Times, I'll post some of the article here:
"In August of 2000, in the midst of the Bush-Gore presidential race, a Harvard professor, Paul E. Peterson, released a study saying that school vouchers significantly improved test scores of black children. Professor Peterson had conducted the most ambitious randomized experiment on vouchers to date, and his results — showing that blacks using vouchers to attend private schools had scored six percentile points higher than a control group of blacks in public schools — became big news."
This led to the professor appearing in newspapers and on television news shows across the country. Bush used the report. Gore had to answer for his stance against vouchers. In 1999, Brother Jeb Bush had railroaded vouchers through the Florida education system (as part of his plan to dismantle it, IMHO) taking the lead in the United States and now those vouchers have been found as unconsititutional, violating separation of church and state principles. The Republicans were looking for reasons to do what they wanted to do already and the professor from Harvard handed them just what they wanted to hear. The news media, of course, were their typical uncritical selves:
"Then, three weeks later, Professor Peterson's partner in the study, Mathematica, a Princeton-based research firm, issued a sharp dissent. Mathematica's report emphasized that all the gains in Professor Peterson's experiment, conducted in New York City, had come in just one of the five grades studied, the sixth, and that the rest of the black pupils, as well as Latinos and whites of all grades who used vouchers, had shown no gains. Since there was no logical explanation for this, Mathematica noted the chance of a statistical fluke. 'Because gains are so concentrated in this single group, one needs to be very cautious,' it said. Several newspapers wrote about Mathematica's report, but, coming three weeks after the first round of articles, these did not have the same impact."
This is the price the US pays for an uncritical news media that publishes press releases rather than asks difficult questions about scientific studies. But there's more:
"David Myers, the lead researcher for Mathematica, is hesitant to criticize Professor Peterson. ('I'm going to be purposely vague on that,' he said in an interview.) But he did something much more decent and important. After many requests from skeptical academics, he agreed to make the entire database for the New York voucher study available to independent researchers. A Princeton economist, Alan B. Krueger, took the offer, and after two years recently concluded that Professor Peterson had it all wrong — that not even the black students using vouchers had made any test gains."
It turns out that when Krueger looked at the data, he added back 292 cases that had been excluded from the first study. That additional data changed the results drastically. But is this bad science on the part of Peterson? Well, yes and no. This is the normal ins and outs of research AND it is the reason why ONE STUDY IS NOT CONCLUSIVE:
"Some background. In 1997, 20,000 New York City students each applied for a $1,400 voucher to private school through a project financed by several foundations. A total of 1,300 were selected by lottery to get a voucher, and 1,300 others — the controls, who had wanted a voucher but were not selected — were tracked in public schools. When the first test results came back, the vouchers made no difference in test scores for the 2,600 students as a whole. So the original researchers tried breaking the group down by ethnicity and race, and that's when they noted the sixth-grade test gains for the black voucher group. But there was a problem. The original researchers had never planned to break out students by race. As a result, their definition of race was not well thought out: it depended solely on the mother. In their data, a child with a black mother and a white father was counted as black; a child with a white mother and a black father was counted as white. When the father's race is considered, 78 more blacks are added to the sample. Professor Krueger also found that 214 blacks had been unnecessarily eliminated from the results because of incomplete background data. These corrections by Professor Krueger expanded the total number of blacks in the sample by 292, to 811 from 519."
If reporters don't dig and ask the right questions, then things are heavily dependent upon the scientists, who unlike their mystique are human and have egos:
"As for Professor Peterson of Harvard, the star of newspapers and TV news in 2000 remains curiously mum these days. In a brief interview, he declined to comment on Professor Krueger's or Mathematica's criticisms. He said he stood by his conclusion that vouchers lifted black scores, and would 'eventually' respond in a 'technical paper.' But he said he would not discuss these matters with a reporter. 'It's not appropriate,' he said, 'to talk about complex methodologies in the news media.'"
Let's see, it was appropriate to issue press releases regarding his conclusions, but it is not appropriate to explain how he got to those conclusions. This is junk science, my friends. You can't have it both ways.
So why did I appreciate this article so much?
Because I'm sick of hearing about how science and medicine just KNOWS that being fat is bad. The NYT's said about policy making (unmarking, of course, the media's role in this fiasco):
"It is scary how many prominent thinkers in this nation of 290 million were ready to make new policy from a single study that appears to have gone from meaningful to meaningless based on whether 292 children's test scores are discounted or included. "
In so-called "obesity" research this happens all the time.
As it turns out, scientific study have social contexts: the prestige of the researcher, the economic benefit from the results, the politics of the results affect how the study is reported. But even more fundamental: such things as the definition of terms and the inclusion or exclusion of data and the collection processes affect the results.
That is why good scientific method includes replication of results before any conclusions can be drawn, especially conclusions that lead to cause and effect links. That is why good scientific method includes skepticism of every single result of every single study.
I've run out of time this morning, but look for a lesson in cause and effect in the next couple of days. I think the time has come to talk about critically thinking about scientific studies and statistics, but I want to do it justice, though,
so to be continued...
Posts
