Wikipedia vs Peer-Reviewed Medical Literature for Information About the 10 Most Costly Medical Conditions

Methods

The 10 most costly conditions in the United States by public and private expenditure in 2008—the year that the most complete data were available for the present study—were identified from the publicly available database from the Agency for Healthcare Research and Quality.¹³ We then identified 10 Wikipedia articles that we believed most closely related to each of those conditions. Because Wikipedia articles are dynamic and subject to frequent changes and updates, we printed the selected articles on April 25, 2012, for our research purposes.

In a blinded process, we randomly selected 10 reviewers to examine 2 of the selected Wikipedia articles. Each reviewer was an internal medicine resident or rotating intern at the time of the assignment. This arrangement created redundancy, giving the study 2 independent reviewers for each article. Also, by using physicians as reviewers, we ensured a baseline competency in medical literature interpretation and research. We used a Web-based randomizer (http://www.random.org) to assign the selected Wikipedia articles to each reviewer. Reviewers were asked to identify every assertion (ie, implication or statement of fact) in the Wikipedia article and to fact-check each assertion against a peer-reviewed source that was published or updated within the past 5 years. Reviewers were sent an e-mail containing examples of assertions (eg, “diuretics are the initial drug of choice for essential hypertension without co-morbidities”). The authors instructed the reviewers to use UpToDate (http://www.uptodate.com/) as the initial means by which to search for peer-reviewed sources. If UpToDate did not produce adequate results, then each reviewer was instructed to use PubMed (http://www.ncbi.nlm.nih.gov/pubmed), Google Scholar (http://scholar.google.com/), or a search engine of their choice. Each reviewer then reported concordance or discordance between Wikipedia and the peer-reviewed sources. Two researchers who did not participate in the original review process then compared both reviews of each article for similar assertions as well as dissimilar assertions and tallied the concordance and discordance for each.

The null hypothesis of the study was that there would be concordance between the Wikipedia article and the peer-reviewed sources (P>.05). The alternative hypothesis was that there would be discordance (ie, no concordance) between the Wikipedia article and the peer-reviewed sources (P<.05). A McNemar test for correlated proportions was conducted for the assertions that were similar, dissimilar, or both, as assessed by the blinded reviewers.^{14(pp171-178)}

Results

The Agency for Healthcare Research and Quality¹³ listed the following 10 conditions as the costliest: heart disease, cancer, mental disorders, trauma-related disorders, osteoarthritis, chronic obstructive pulmonary disease/asthma, hypertension, diabetes, back problems, and hyperlipidemia. The corresponding Wikipedia articles^15-24 are listed in Table 1. Examples of the descriptive terms we used to categorize the findings of each reviewer are listed on Table 2.

Reviewers found a statically significant discordance between Wikipedia and peer-reviewed sources for assertions that were similar (P<.05) in all but 1 of the conditions: trauma-related disorders (ie, concussions). The same was true for all assertions found by the blinded reviewers of the articles (P<.05 for all conditions except concussions). In 4 articles—major depressive disorder, osteoarthritis, chronic obstructive pulmonary disease, and diabetes mellitus—there was a statistically significant discordance between Wikipedia articles and peer-reviewed sources for dissimilar assertions. The interpretation of the P value is true for similar assertions between the 2 reviewers as well as for dissimilar assertions (Table 3).

Discussion

A few studies^12,25-27 have compared Wikipedia articles with standard peer-reviewed sources and have shown it to be roughly equivalent to these sources. The most notable study, by Giles,²⁵ compared Wikipedia with the Encyclopedia Britannica. Other authors^12,26,27 have compared Wikipedia with textbooks and national databases and showed comparable results. In contrast, other researchers^28-30 have determined that Wikipedia is unsuitable as a reference for drugs. Except for psychiatric conditions,²⁶ scientific research has never, to our knowledge, focused on Wikipedia's content on prevalent medical conditions. A recent study by Azer³¹ concluded that Wikipedia is not a reliable information source for medical students in gastroenterology and hepatology.

The present study demonstrated that most Wikipedia articles on the 10 most costly conditions in the United States contained assertions that are inconsistent with peer-reviewed sources. Because our standard was the peer-reviewed published literature, it can be argued that these assertions on Wikipedia represent factual errors.

A perplexing finding in our study was that most of the dissimilar assertions found by the reviewers failed to demonstrate discordance. A reporting bias may have plausibly occurred: each article reviewer was either an internal medicine resident or a rotating intern physician at the time of the review and may not have believed that every assertion was worth reporting. For example, the diabetes mellitus Wikipedia article stated that it is a condition in “which a person has high blood sugar.” One reviewer might have accurately recorded this statement as an assertion, whereas another might have assumed the statement to be common knowledge and erroneously not recorded it as an assertion. These incongruent criteria for assertions may explain the difference found between reviewers.

Although 9 of 10 articles demonstrated discordance between Wikipedia articles and the peer-reviewed sources, the article on concussions did not. This finding may have occurred because Wikipedia has a number of different contributors to each article and the contributors to this particular article were more expert.

The present study had 5 main limitations. First, it did not address errors of omission, but rather was designed to detect assertional errors. It is possible that the Wikipedia article did not contain important information about a topic. However, we opted not to examine errors of omission because of the subjectivity involved with determining what should be included in a review article on a specific medical topic. Second, the present study would have been stronger if more than 2 reviewers were assigned to each article. A future study design could use additional reviewers with more varied specializations to strengthen its findings. Third, we used any peer-reviewed reference as a standard that included an initial search through a subscription-only service (UpToDate). Fourth, we used physicians-in-training rather than content experts as reviewers, which may have created a bias that the present study was not designed to measure. Lastly, we did not check the assertions in the peer-reviewed sources, a limitation that may prove important because peer-reviewed sources are often not in agreement. Future studies might also include how the convenience of Wikipedia may influence perception of the reliability of the information found.

Conclusion

Most Wikipedia articles for the 10 costliest conditions in the United States contain errors compared with standard peer-reviewed sources. Health care professionals, trainees, and patients should use caution when using Wikipedia to answer questions regarding patient care.

Our findings reinforce the idea that physicians and medical students who currently use Wikipedia as a medical reference should be discouraged from doing so because of the potential for errors.