Clinical and Health Affairs
Teaching Numeracy to Physicians-in-Training
Quantitative Analysis for Evidence-Based Medicine
By Robert M. Jacobson, M.D.
Abstract
Many physicians admit to having some degree of innumeracy—difficulty understanding and working with numbers. Yet, increasingly, physicians in all specialties are committing to practicing evidence-based medicine (EBM) and, as a result, must learn to discern quantitative differences and address statistical significance. Although no one expects a practicing physician to be able to evaluate a statistician’s choice of methods or conduct an independent rendering of a clinical study’s analysis, practitioners of EBM must learn how to assess the importance of results found in a clinical study. Since 2001, Mayo Clinic has been teaching its residents in pediatric and adolescent medicine the skills required for EBM. This article describes the 5 steps involved in practicing EBM, focusing on the interpretation of study results.
Public health officials and epidemiologists frequently note that physicians in general admit to having some degree of innumeracy or mathematical illiteracy.1 Unlike illiteracy, innumeracy affects well-educated, highly intelligent individuals who must make complex decisions and conduct complicated transactions. Yet, innumeracy can lead to confused decision-making, poorly written policies, and even unscientific thinking.
Innumeracy often manifests when physicians must deal with probability and statistics. Most physicians understand the importance of these concepts. They know that much of what they encounter in their practice is marked by probability and uncertainty. Physicians understand that all medications carry known and unknown risks of adverse events and that no laboratory test is without false-positives and false-negatives. They expect to see references to statistical methods when reading about clinical studies, and they know to look for P values less than 0.05.
Nonetheless, most physicians feel uncomfortable reviewing the statistical sections of research reports. Recent studies have demonstrated, if anything, that the gap between what is published in the statistical methods sections of studies and what is understood by typical readers has widened—not narrowed—despite attempts by both medical schools and residency programs to teach medical statistics.2,3
This chasm occurs at a time when physicians are embracing evidence-based medicine (EBM).4-6 One of the premises of EBM is that published studies—rather than experience, routine, anecdote, or recollection—are the basis for what we know and do not know about medicine. To determine the validity and significance of a study, one has to understand the design and the statistical methods used to analyze the data. EBM is now routinely taught in medical schools and residencies. Continuing medical education programs and practice-guideline developers rely on EBM as an arbiter for disputes and look to it as a mark of quality. Despite its inherent weaknesses and limitations, EBM is here to stay.
Therefore, physicians must be equipped to evaluate medical evidence numerically. This means that despite their feelings of inadequacy for mathematical thinking, physicians must be able to conduct a quantitative analysis of the evidence presented in published reports.
Teaching Numeracy
To make physicians more comfortable with numbers, we began teaching pediatric residents at Mayo Clinic techniques for interpreting medical research in 2001. We abandoned our twice-monthly journal club and instituted a series of lectures, small-group activities, and large-group interactions. Our goal was to help physicians-in-training practice EBM independently by providing them with the skills they will need to make sense of published studies—including the sections on statistical methods and the presentation of quantitative results. These skills are necessary to every field of medicine, not just pediatrics. Future physicians should begin learning them in medical school and continue developing them throughout their careers.
Our residents take part in an ongoing series of meetings, each an hour in length, held throughout the 3 years of the program. We meet approximately every other week. Each academic year (starting in July with the arrival of the first-year residents) begins with a series of 10 seminars that introduce both qualitative and quantitative topics. The first 5 sessions address basic principles and are designed for first-timers, although 2nd- and 3rd-year residents routinely attend and participate. The next 5 sessions consist of more intense discussions about the nuances of study design and statistical methods. The material discussed in the seminars is then made available on the department’s website so residents can access it anytime.
We teach the residents a method for conducting an EBM investigation that includes these 5 steps:
1. Construct an answerable clinical question,
2. Search for studies that answer the question,
3. Evaluate the study methods qualitatively,
4. Evaluate the results quantitatively, and
5. Apply the evidence in clinical practice.
After the 10 didactic sessions, residents work in pairs to research a clinical topic using this approach. They then present their findings to the entire group. This gives the presenters a chance to demonstrate what they have learned about applying quantitative analysis and the other residents the chance to consider the investigation thoughtfully.
Dealing with Numbers
Of the 5 steps necessary for applying EBM to clinical problems, the 4th step is the one that requires the reader to understand the presentation of numbers, calculations, and statistics in a study. We begin by introducing students to new, sophisticated means for expressing the results of studies: numbers needed to treat (NNT), numbers needed to harm (NNH), and likelihood ratios (LRs). These replace myriad terms used by statisticians and researchers including absolute and relative risk reduction, odds ratio, and predictive indices.
Numbers needed to treat or NNT tells how many persons we would have to treat using a new intervention to see improvement in 1 patient on average. For example, let’s say Drug A is a novel therapy that has a cure rate of 70%. This compares well with the standard therapy, Drug B, which has a cure rate of 60%. In this case, we would calculate that the NNT is 10. In other words, we would need to treat 10 patients with the novel therapy to see 1 patient benefit as compared with the standard therapy.
In general, the lower the NNT, the better. Solid proposals for switching to new therapies based on their improved benefit usually involve NNTs of 2 to 5. A perfect NNT value would be 1, of course, meaning that every patient who is treated will benefit from the change to the new therapy. Acceptable values for many studies range from 2 to 15. An upper limit might be 50 or 100. For preventive therapies, the NNT value can be much higher. In studies of preventive therapies, the minimally important NNT might be 100, 500, or even 1,000.
To give the residents a sense of similar NNTs, we provide them with benchmarks from the field. For example, Permethrin, when compared with placebo for the treatment of head lice, has an NNT of 2. Fluoxetine for depression in adolescents as compared with placebo has an NNT of 5. Amoxicillin versus placebo to cure otitis media at 4 days has an NNT of 8, while amoxicillin for otitis media at 11 days has an NNT of infinity. A course of antenatal steroids given to preterm infants to prevent respiratory distress syndrome compared with none at all has an NNT of 11. Intensive therapy for insulin-dependent diabetes mellitus neuropathy has an NNT of 15.
Numbers needed to harm or NNH is a measure of the number of patients who need to be exposed to a treatment to experience an adverse event such as a side effect. The measure also can be used to express the negative impact of an etiologic or prognostic risk factor. NNH is calculated similarly to NNT.
For studies of diagnostic tests, readers use likelihood ratios, which express the impact of a test’s results, whether positive or negative, on the likelihood of a patient having a particular disease or condition.
We teach residents how to calculate the NNT, NNH, and LR using the data provided in the study, as quite often, the study itself does not provide these clinically useful renderings of the results. On the other hand, the math is not daunting and can be done on the back of an envelope.
We also teach them to obtain a confidence interval for these quantitative measures. The confidence interval is the range that we are confident contains the true value. We typically use a 95 percent confidence interval, meaning that if we ran the study 100 times the same way, 95 times in 100, the confidence interval that we calculated from that study would include the true value.
For example, having ascertained a study’s NNT, the reader should construct the confidence interval for that NNT. If the confidence interval for an NNT of 4 ranges from 3 to 8, then one could feel pretty confident that a treatment is likely to result in improvement. But if the NNT is 4 and the confidence interval ranges from 2 to 6,132, one should be concerned that the true NNT may be far beyond a respectable 5 or 10 or even 15. A very large confidence interval might indicate that a sample size was too small and suggest the study should be repeated or combined with other studies in a meta-analysis. Alternatively, given an NNT of 150 with a confidence interval ranging from 100 to 10,000, the reader can conclude no more studies need to be done. In that particular case, there is no evidence a repeat study, even a larger one, would move the NNT into an acceptable range.
Just as many studies do not provide the NNT, neither do they provide the confidence interval or enough detail to make calculating the confidence interval straightforward. We teach the residents how to access and use online resources to calculate the confidence intervals for the NNT in such instances.7
Measuring Our Impact
Charles H. Mayo, M.D., once said, “Probably in the not distant future, we will crawl out of our old methods of education, as a snake sheds its skin, and reorganize a new plan.”8 We believe this has occurred. Determining the best approach to diagnosis or therapy no longer can be taught using textbooks; it now requires learning to access and evaluate studies from the ever-expanding library of medical literature.
The challenge in teaching these skills is to give students enough information so that they can intelligently interpret the numbers in a study and yet not make the process so complicated that they will avoid the undertaking. How do we know if we are successful in teaching our residents these skills? The ultimate proof is our residents’ application of the methods we teach outside the classroom. To assess their competence in applying these techniques, we survey residents at the beginning and the end of each year. Meanwhile, such groups as the Ambulatory Pediatric Association are attempting to develop even better measures that can reliably demonstrate whether our teaching techniques are having long-term effects. We know our residents are more confident in their ability to understand scientific studies and statistical methodology, and we see them demonstrating their abilities on the wards and in the office. We believe that, by helping our residents become more skillful in assessing the studies quantitatively, they are becoming better readers of medical literature, and, in turn, better practitioners of medicine. MM
Robert M. Jacobson is chair of the department of pediatric and adolescent medicine and professor of pediatrics in the College of Medicine at Mayo Clinic.
References
1. Paulos JA. Innumeracy—Mathematical Illiteracy and its Consequences. New York: Farrar, Straus, and Giroux Hill and Wang division, 2001.
2. McNally P, Loftus BG. Knowledge of statistical methods and their implications for clinical practice: a survey of paediatricians. Ir Med J. 2005:98(10):240-2.
3. Hellems MA, Gurka MJ, Hayden GF. Statistical literacy for readers of Pediatrics: A moving target. Pediatrics. 2007:119(6):1083-8.
4. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA. 1992;268(17):2420-5.
5. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence-based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71-2.
6. Jacobson RM. Pediatrics and evidence-based medicine revisited. J Pediatr. 2007;150(4):325-6.
7. Graphware Software Quickcalcs. Available at: www.graphpad.com/quickcalcs/index.cfm. Accessed October 15, 2007.
8. Mayo CH, Mayo WJ. Aphorisms. Rochester, Minnesota: Mayo Foundation, 1988.