Elsevier

Injury

Volume 44, Issue 1, January 2013, Pages 6-11
Injury

Lies, damn lies and statistics: Errors and omission in papers submitted to INJURY 2010–2012

https://doi.org/10.1016/j.injury.2012.11.005Get rights and content

Abstract

Introduction

Many reviews of published papers in the medical literature have reported errors in statistical methods and presentation.

Methods

100 successive papers submitted to INJURY and sent for initial statistical review between December 2010 and January 2012 were analysed. The comments made on the papers were categorised and summarised.

Results

Suggestions for improvement were made for 90 of the papers. An inappropriate analysis was identified in 47. Other errors were seen in 45 papers including 9 wrong p-values for the method used. Simple numerical mistakes were common (19%). An inadequate description of some element of the study was a problem in 22 papers and additional limitations to be described in Discussion were recommended in 26. Numerically most comments were made about some element of the presentation of results.

Discussion

Many of the errors identified are easily avoided. Guidance on some common issues is presented.

Conclusions

Statistical and numerical errors are common in papers submitted to INJURY and requiring statistical review. Following the advice in Discussion and using reporting guidelines should reduce the number of papers requiring corrections.

Introduction

Medical statistics is often considered a young discipline, though the use of statistical diagrams to present information pertaining to health certainly dates to Florence Nightingale with a series of publications in the 1850s.1 Epidemiology also has its roots in that period with the pioneering work of John Snow on the modes of communication of cholera.2 Modern medical statistics owes much to Austin Bradford Hill in the middle of the twentieth century. He brought the ideas of randomised experimentation and statistical inference into medicine from the agricultural areas where the methods had been developed. Since then the role of statistics in medical research has been expanding and evolving. The practice of medical statistics has been revolutionised by the growing availability of low-price computing and the development of statistical packages for analysis. Many of these packages are designed for easy access by statistical novices, bringing the possibility of conducting sophisticated statistical analyses within the ambit of all researchers. This very positive development also has associated risks. It is now possible to present a totally erroneous analysis in a way that is superficially convincing.

Statistical review of papers in medical journals was quite unusual until a series of review papers from the 1970s onwards, demonstrated that a majority of papers in leading journals had worrying statistical errors. An early example showed that of 62 papers published in the British Medical Journal within the first quarter of 1976, 32 had errors while 5 actually came to false conclusions.3 Since then, statistical review has been an increasingly common feature of many medical journals and the major medical journals now ask for statistical review of all, otherwise acceptable, papers with a statistical content.

The requirement for papers to meet rising standards for their scientific and statistical content has been an impetus for the development of guidelines to help authors prepare their papers for publication. Randomised controlled trials were the first subject for such guidelines with the CONSORT statement, subsequently updated.4 This has been extended to also deal with cluster randomised trials.5 There are now also guidelines for diagnostic accuracy (STARD),6 meta-analysis (PRISMA7), and observational studies (STROBE)8 among others. The Council of Science Editors website is a good access point for guidelines.9

Statistical guidelines are, of necessity, somewhat general. It is difficult to imagine that anyone will ever be able to anticipate all of the complexities that may arise in any research study, so a cookbook approach to statistics will always have its limitations. There are certain principles that should be followed, however, and Altman and colleagues were among the first to delineate these.10

The reviews of statistical quality have usually been restricted to published papers because of difficulty in gaining access to the original submissions. Thus, it is reasonable to anticipate that errors in the papers submitted to journals will be even higher than in the ‘sanitised’ versions that are published after peer review. Experience of reviewing papers for Injury indicates a broad spectrum of statistical expertise among authors. The purpose of this paper is to document the issues that have been raised during statistical review in order to provide potential authors with a better idea of the expectations of a statistical reviewer and to minimise the number of drafts before acceptance of their paper. Some errors are common and many of them are present in papers by authors who otherwise appear to have a fairly strong statistical background. We are hopeful therefore that even those who feel confident in their statistical abilities will take the time to consider these findings.

For those at the less experienced end of the spectrum, we hope this paper will help to put the statistical issues into a better perspective and help to de-mystify a subject which is widely regarded as ‘difficult’. It is, of course, a subject that can readily become quite technical when applied to a complex data set. In these circumstances it is worth remembering this definition of the statistical method; it is a method for the elucidation of data that is affected by a multitude of causes. Statistics should be acting as an aid to understanding our data and then to facilitate the communication of that understanding to others. Non-statisticians may be surprised how often statistical reviewers suggest a reduction in the statistical content of a medical paper rather than suggest additional statistical work.

Section snippets

Methods

All papers submitted to one of the authors (RP) for blinded statistical review between December 2010 and January 2012 were included in the review, irrespective of whether or not the paper was subsequently published in INJURY. This period was chosen to give a sample size of 100 for convenience of presentation. The comments in the original statistical review were subsequently categorised by the same person, who is a medical statistician with 40 years of experience. These categories were used to

Results

Some suggestions for improving the paper were made in 90 of the 100 papers reviewed. Of these an inappropriate analysis of the kinds listed in Table 1 were identified in 47. The most common problem was the incorrect analysis of 2 × 2 contingency tables. There were 65 papers presenting such data and 14 (22%) used an unsuitable method.

As of much concern as the use of an inappropriate method of analysis is the presentation of erroneous results. This was detected in 45 papers (Table 1). Many tests of

Discussion

Appropriate analysis, both quantitative and qualitative, is a keystone to obtaining valid evidence from research. Classical statistical analysis is one form of quantitative analysis. Regardless of how good the method is and how reliable the results are, unless the statistical analysis is appropriate, incorrect conclusions can be drawn. In the world of evidence-based practice, decisions should only be made where the analysis of the results of any paper is robust and the nature of the statistical

Conclusion

This paper has documented the range of statistical problems that have been identified on initial review of papers submitted to INJURY. Authors of papers with statistical content are recommended take the points raised in Discussion into account when preparing their analysis plans. Additionally they should take advantage of the various guidelines that are available for reporting specific types of study.

Conflict of interest statement

Professor Prescott is Statistical Advisor to INJURY. Mr Civil is Non-Orthopaedic Editor of INJURY.

References (10)

There are more references available in the full text version of this article.

Cited by (0)

View full text