Among the first physicians to have considered the implications of statistical probability in medical research was Donald Mainland, MD, of Dalhousie University in Halifax, Canada. He appears to have been the first to report modern statistical methods, in articles published in the Canadian Medical Association Journal and in the British Medical Journal in the 1930s.
Since then, medical research has increasingly adopted and developed principles of experimental design and statistical analysis. Biostatistics has been essential in moving medical research from anecdotal case reports to experiments with control groups, and finally to the large-scale, randomized trials that are now the preferred standard of scientific proof. In addition, the evidence-based medicine movement has established the benefits of applying the best evidence to resolving clinical problems. Evidence-based medicine is predominantly literature-based medicine and so depends heavily on the quality of published research, especially on the proper use and complete reporting of statistical methods and results.
But there is a problem. Lots of them, actually.
The overall quality of the medical literature is poor. Much research is poorly designed, poorly executed, poorly analyzed, or poorly reported, to the point that the results and conclusions are often not valid. In terms of statistics, I think there are three broad concerns.
Concern #1: Many researchers still use only basic statistics, if they use any at all.
Several studies have found that the number of scientific articles that use no or only basic statistics—descriptive statistics, t- tests, chi-square tests—is surprisingly if not disturbingly large:
- Of 229 articles published in the Journal of Family and Community Medicine between 1994 and 2010, 29% used no or only summary statistics, including 14% that reported odds or risk ratios; 68% used only chi-square tests; and 29% used only t-tests.
- A 2004 study of 144 articles from six pharmacy journals found that 28% used only descriptive statistics. Of the 99 articles using statistical tests, 77 used only basic tests: 33 used chi-square tests, 26 used t tests, and 18 used correlation analyses.
- A 1996 study of 219 articles in two radiology journals found that 47% used no or only descriptive statistics and that 31% used only one statistical method. Only19% used more advanced statistics, such as regression analyses or multi-way analysis of variance.
- Among 215 articles published in the Journal of Foot and Ankle Surgery between 2004 and 2008, 16% used no descriptive statistics. Of the 146 articles using statistical tests, 30% were t tests and 21% were chi-square tests.
- In articles from the New England Journal of Medicine published in 1978 and 1979, 27% used no or only descriptive statistics, 44% used only tests, and 27% used only chi-square tests. Similar studies of the journal in 1989 and 2005 found no substantial change in these percentages.
In all fairness, the use of more sophisticated statistical methods is increasing, especially in some journals and fields of medicine, and understanding some of the literature now requires knowing more about statistics than ever before. But the overall level of statistical sophistication is not great.
Concern #2: Researchers who use statistics make lots of mistakes when using or reporting them
Studies of the statistical quality of journal articles have consistently found high error rates in the application, analysis, interpretation, or reporting of statistics. Since the first such study—the earliest I found was published in 1959—error rates of at least 50% and as high as 80% have been reported. Further, about half these errors are so great as to call the authors’ conclusion into question. The problem is made worse by the fact that most of these studies are of the world’s leading peer-reviewed general medical and specialty journals.
Although errors have been reported for more complex statistical procedures, paradoxically, many errors are in basic, not advanced, statistical methods. Perhaps advanced methods are suggested by consulting statisticians, who perform the analyses competently, but it is also true that, as above, authors are far more likely to use only elementary statistical methods, if they use any at all. Still, articles with even major errors continue to pass editorial and peer review and to be published in leading journals.
In fact, the problem of poor statistical reporting in the biomedical literature is long-standing, worldwide, present in all fields of medicine, potentially serious, and not at all apparent to many readers. At the same time, most of these errors are related to topics included in most introductory statistics books. It seems strange indeed that a problem seemingly so important, so widespread, and so long-standing should continue, despite apparently being so basic in nature.
Concern #3: Nobody seems to be doing much about the first two concerns.
The hesitancy to use statistics and the inability to use them correctly will likely continue in the absence of major educational efforts. To be sure, the problem is unfortunate at best, unforgivable at worst, but understandable in either case. Statistics as a field of study attracts few people who are already attracted to medicine. It is full of subtleties and complexities that require much time to appreciate and more time to master, and it is often taught by those oriented to mathematics, not to medicine. In their undergraduate training, most health care providers do take at least one statistics class, but they rarely learn or remember what they want or need to know to understand or write research publications.
But there’s more. These problems will also continue in the absence of suitable reporting guidelines that are 1) widely adopted by authors and 2) rigorously enforced by journals.
More than 30 years ago, several authorities recommended that statistical reporting guidelines be adopted by the medical community. To be sure, several individual articles on such guidelines were published in biomedical journals, but these have usually been general in nature, small in scope, and limited in distribution.
In fact, the first comprehensive set of statistical reporting guidelines for authors, editors, and reviewers appears to have been How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers, published by the American College of Physicians in 1997 and updated in a second edition in 2006.
Journals likewise have not been proactive about improving statistical reporting. Although both the AMA Manual of Style and the Council of Science Editor’s Scientific Style and Format now include chapters on reporting statistics, most journals have still not included in their instructions for authors more than a paragraph or two about the topic.
Not until 2013 was there a comprehensive set of reporting guidelines designed specifically to be included in a journal’s instructions for authors. Called the SAMPL Guidelines (Statistical Analyses and Methods in the Published Literature), they were soon listed on the EQUATOR network site as “Key Reporting Guidelines.” Despite wide publicity, few journals have adopted them.
But here’s the final, awkward, truth: neither How to Report Statistics nor the SAMPL guidelines were introduced by a clinician, statistician, researcher, educator, or journal editor: the people who actually need to interpret, evaluate, and report statistics or to teach these skills to others. This fact speaks volumes about the general level of disinterest in statistics.
At the same time, this disinterest can be an opportunity for medical writers and editors who take the time to learn something about understanding and reporting statistics. Learning to interpret and evaluate statistics is not the same as calculating them. The learning curve is not nearly as steep, or as necessary, and some basic knowledge goes a long way. The terms and concepts needed to understand the statistical methods and results in perhaps 80% of the medical literature can be taught in about 2 days. Learning how to apply this knowledge when writing and editing research manuscripts takes longer, but not much, and the value of adding this skill to one’s professional services is substantial.
Learn more about Mr. Lang
