| Biostatistics Rules: A cautionary tale of good intentions and bad numbers |
|
Numbers may be the backbone of biomedical research, but it is how those numbers are derived -- rather than the exact identify of the figures themselves -- that may determine whether they will stand up to the rigors of reproducibility so fundamental to academic science. If the methods are thorough, then the progress of science can be served. However, if the statistics are shady, the research can fall apart, sometimes with disastrous consequences. An example of biostatistics gone awry is the recent suspension of three clinical trials at Duke after a letter penned by 33 prominent biostatisticians asked for an investigation of their scientific validity. One of those biostatisticians, Michael Kosorok, professor and chair of the department of biostatistics at UNC-Chapel Hill and director of the Biostatistics Core at the NC Translational and Clinical Sciences (NC TraCS) Institute, says that people first became suspicious of the genomics study underlying the trials because it had achieved success far too easily. “Not that I’m saying that these kinds of studies can’t yield good results,” said Kosorok. “It’s just that we as biostatisticians know things are fishy when discoveries are made easily. And genomics without good statistical principles is guaranteed to be fishy.” Kosork’s friends Keith Baggerly and Kevin Coombes of MD Anderson Cancer Center were the first to crack the case. Intrigued by Duke’sresults, they asked the researchers for access to their data so they could attempt similar analyses. But they wouldn’t give it to them, and the biostatisticians had to do something called biostatistic forensics (cue CSI theme music) to unearth how the Duke researchers had arrived at their statistical findings. Despite their best attempts, Baggerly and Coombes were unable to reproduce Duke’s results. “When you are dealing with a lot of data, the chances of seeing something that is not real are extremely high, upwards of 99.9 percentin fact” said Kosorok. “It is almost guaranteed that you will see false information. But if you don’t know this sort of thing, if you aren’t firmly rooted in a foundation of statistics, you just get excited with everything you see in the data. Well, these researchers didn’t incorporate this type of thinking and got excited with everything they saw, so a lot of false numbers were feeding into the trial.” What’s more, the forensic biostatisticians uncovered serious statistical errors and data management issues in Duke’s work, which was supposed to use genomics to predict which chemotherapy is best for each cancer patient. In findings published in the journal Annals of Applied Statistics, the biostatisticians cautioned that “poor documentation can shift from an inconvenience to an active danger when it obscures not just methods but errors.” Tables were off alignment and patients were literally assigned the wrong treatment, potentially harming participants, said Kosorok. Accordingto The News & Observer, Michael Cuffe, vice president for medical affairs at Duke responded to concerns of patient safety by saying that individuals whose treatment relies on this genomics approach were not at risk, as the treatment they receive follows standard medical procedures. Kosorok, however, personally believes that the secrecy of the researchers involved was in fact dangerous because it led to a number of very serious errors being propagated as nobody was cross-checking the data. “There can be this sort of cowboy mentality among some scientists that they want to do everything themselves, keep everything privateand get all the credit,” said Kosorok. “And that is very dangerous when you are dealing with things that require careful reproducibility and evaluation like clinical trials because you can’t make mistakes, it is just too harmful for patient lives. And you can’t just base your feelings on what happened with one patient or in one situation, you really have to look at the picture as a whole, in a way that actually reflects the potential reality that you are facing.” Duke’s Internal Review Board conducted its own investigation into the work last year and resumed the trials in January after concluding thatthe approach was “viable and likely to succeed.” But their data was never released for an independent evaluation and validation, raising one of the main issues stated in the letter from the 33, questioning how the IRB could determine viability without such outside scrutiny. The plot thickens. The trials were halted again after allegations that one of the trials’ key investigators, Anil Potti, falsely claimed to be a Rhodes Scholar on his applications for federal funding and resigned his post at Duke. One of Potti’s collaborators has asked to retract at least one of the scientific publications leading to the trials. And the American Cancer Society and the British journal The Lancet Oncology continue to investigate why other researchers have not been able to replicate the groundbreaking findings of the scientists. “I think that the best research absolutely requires trust, having other researchers in multiple areas of expertise that you can collaborate withand trust to do the right thing,” said Kosorok. “Trust is essential, because if you know everything there is to know either the problem you are working on is too simple or you do not understand the issues correctly. “What happened at Duke is really a double-edged sword. It demonstrates just how important biostatistics is to medical research, but it also tells us that we have to be even more careful in what we do or we can make those same mistakes. Even though we know better, we need to make sure we practice what we preach.”
For more information on “Pottigate,” visit:
|
| By Marla Broadfoot |
