The Role of Numeracy in Understanding the Benefit of Screening Mammography

Methods

Study Design and Sample

In December 1995, we drew a simple random sample of 500 women from a registry of female veterans maintained at the Department of Veterans Affairs Medical Center, White River Junction, Vermont. These women were mailed one of four questionnaires, which differed only in how the same information on average risk reduction with mammography was presented. Reminder letters were sent to nonrespondents after 2 weeks, and new copies of the questionnaire were mailed after 4 weeks. The study was closed in February 1996. Because of errors in the registry, 26 entries in our sample were not usable (10 of the listed persons were deceased, and 16 either were not female veterans or were listed twice). Thus, our possible respondent pool totaled 474, of which 302 (64%) returned the questionnaire.

The returned surveys sometimes contained unanswered questions. Such blanks may represent information about the respondent's abilities, or they may be a marker of an unmotivated respondent or one who has become fatigued from the survey burden. To remove those who were unmotivated or fatigued, we required completion of four of the five questions on the last page of the survey for inclusion in the analyses. Only 15 women did not meet this criterion. Thus, our final sample comprised 287 women, for a 61% completion rate. Unanswered numeracy questions were considered wrong answers. Restricting the sample to women who answered these questions yielded results similar to those seen with the entire sample.

Questionnaire Design

Assessment of Numeracy

Numeracy was assessed with three questions and was scored as the total number of correct responses. The first question assessed basic familiarity with probability: “Imagine that we flip a fair coin 1,000 times. What is your best guess about how many times the coin would come up heads in 1,000 flips? ____times out of 1,000.”

The second question asked respondents to convert a percentage (1%) to a proportion (10 in 1000): “In the BIG BUCKS LOTTERY, the chance of winning a $10 prize is 1%. What is your best guess about how many people would win a $10 prize if 1000 people each buy a single ticket to BIG BUCKS?____person(s) out of 1,000.”

The third question reversed this task, asking the respondent to convert a proportion (1 in 1000) to a percentage (0.1%): “In ACME PUBLISHING SWEEPSAKES, the chance of winning a car is 1 in 1,000. What percent of tickets to ACME PUBLISHING SWEEPSAKES win a car?____%.”

Presentation of Risk Reduction Data

We randomly assigned each of the 500 women to receive one of four questionnaires; the questionnaires differed only in how the information on risk reduction was framed (Figure 1). Risk reduction data were framed as a 33% relative risk reduction, a 33% relative risk reduction together with the baseline risk for death from breast cancer in the next 10 years, a 4 in 1000 absolute risk reduction, or a 4 in 1000 absolute risk reduction together with the baseline risk for death from breast cancer in the next 10 years.

Assessment of Perceived Benefit

To assess their perceived risk for death from breast cancer with and without mammography (Figure 1), women were asked to do the following:

“Imagine 1,000 women exactly like you. Of these women, what is your best guess about how many will die from breast cancer during the next 10 years if …

they are not screened every year for breast cancer by mammogram

_____out of 1000”

they are screened every year for breast cancer by mammogram

_____out of 1000”

Calculation of Accuracy

To see how accurately respondents applied the risk reduction data that they were given, we compared their perceived risk for death from breast cancer with mammography with their perceived risk for death without mammography (Figure 1). Accuracy was judged by the ability to adjust perceived risk in accordance with the risk reduction data presented. Accuracy was judged solely by the change between perceived risk with mammography and perceived risk without mammography, which we calculated from the responses to these two questions. Thus, women could grossly overestimate their risk while still accurately applying the risk reduction information. In fact, many women did overestimate their perceived risk without mammography, as observed in previous work [23].

For the two groups presented with the absolute risk reduction, we subtracted each woman's perceived risk with mammography from her perceived risk without mammography. For example, a woman who perceived her risk to be 100 out of 1000 without mammography and 96 out of 1000 with mammography indicated an absolute risk reduction of 4 out of 1000. Because the women in these groups were told that mammography decreases the risk for death from breast cancer by 4 in 1000, only women whose responses indicated an absolute risk reduction of 4 in 1000 were judged to be accurate.

For the two groups presented with the relative risk reduction, we determined the percentage reduction between the perceived risk without mammography and the perceived risk with mammography. For example, a woman with a perceived risk of 100 out of 1000 without mammography and 67 out of 1000 with mammography indicated a 33% risk reduction. The women in these groups were told that mammography decreases the risk for death from breast cancer by 33%. To allow for rounding off, women whose responses indicated a relative risk reduction between 30% and 40% were judged to be accurate.

Statistical Analysis

For the assessment of both numeracy and accuracy, the percentage of the sample that had correct answers for each measure was calculated as the number of respondents with correct responses divided by the total number of respondents (n = 287). Chi-square tests and Kruskal-Wallis tests were used to compare participant characteristics across the four groups. All comparisons were two-sided and were considered statistically significant at a P value of less than 0.05. We used multiple logistic regression to assess the relation between accuracy in applying the risk reduction information (dependent variable) and numeracy (independent variable) after adjusting for age, income, education, and the framing of the information provided.

Role of the Funding Sponsor

This research was funded by the Department of Veterans Affairs Fellowship Program in Ambulatory Care and the Department of Defense's Breast Cancer Research Program. Neither department had any role in the gathering, analysis, or interpretation of the data or in deciding whether to submit the report for publication.

Results

Study Sample

Overall, the study sample consisted of older female veterans, almost all of whom had at least a high school education. Most reported having annual household incomes of less than $25 000, and fewer than one fourth were currently employed (Table 1). Most reported having had at least one mammogram, and 9% had a history of breast cancer. No significant differences were seen in any characteristic across the four groups.

Assessment of Numeracy

Almost half of the women (46%) answered the coin-flip question (which asked how many times a coin would come up heads in 1000 flips) incorrectly, raising questions about basic understanding of probability. Incorrect answers ranged from 0 to 800 and were largely underestimates (one third of the sample thought that 1000 flips of a fair coin would result in <300 heads). The most common incorrect answers were 25, 50, and 250.

Women also had trouble converting between percentages and probability expressions. The difficulty was greater for the conversion of 1 in 1000 to 0.1% than for the conversion of 1% to 10 in 1000. Although 46% of 287 respondents were unable to convert 1% to 10 in 1000, 80% were unable to convert 1 in 1000 to 0.1%.

The total number of correct responses to these three simple tasks were as follows: Thirty percent of respondents had 0 correct answers, 28% had 1 correct answer, 26% had 2 correct answers, and 16% had 3 correct answers. The percentage of women with 3 correct answers did not differ significantly across the four groups (range, 10% to 22%).

Relation between Data on Risk Reduction and Accuracy

After receiving quantitative risk reduction data about the benefit of mammography, most women did not apply this information correctly when asked to estimate their risk for death from breast cancer with and without mammography (Figure 2). Accuracy in applying risk reduction information was poor for all four groups: Seventeen percent of the group receiving the relative risk reduction with baseline risk were accurate, 10% of those receiving relative risk reduction without baseline risk were accurate, 33% of the group receiving absolute risk reduction with baseline risk were accurate, and 7% of those receiving absolute risk reduction without baseline risk were accurate. As Figure 2 shows, inaccurate responses tended to be in one direction: Most women overestimated the effectiveness of screening mammography.

Presenting the baseline risk for death from breast cancer significantly enhanced accuracy in the groups that received the absolute risk reduction (P < 0.001), although adding the baseline risk did not provide significantly different accuracy for the relative risk reduction groups (P = 0.2).

Relation between Numeracy and Accuracy

Higher numeracy scores were associated with greater accuracy in applying risk reduction information. As the number of correct responses to the three numeracy questions increased, the percentage of women who accurately gauged the risk reduction of mammography increased linearly (Figure 3). Accuracy was strongly related to numeracy: The accuracy rate was 5.8% (95% CI, 0.8% to 10.7%) for a numeracy score of 0, 8.9% (CI, 2.5% to 15.3%) for a score of 1, 23.7% (CI, 13.9% to 33.5%) for a score of 2, and 40% (CI, 25.1% to 54.9%) for a score of 3. Thus, compared with a numeracy score of 0, the odds ratio for accuracy was 1.6 (CI, 0.5 to 5.2) for a score of 1; 5.1 (CI, 1.8 to 14.5) for a score of 2; and 10.9 (CI, 3.7 to 32.3) for a score of 3.

Because women with higher levels of numeracy were younger and reported having higher levels of education and income, we performed multiple logistic regression analyses to assess the independent effect of numeracy on the accuracy of applying risk reduction information. Adjustment for age, income, level of education, and framing of the information had little effect on the relation between accuracy and numeracy: Compared with a numeracy score of 0, the odds ratio for accuracy was 1.3 (CI, 0.3 to 4.7) for a score of 1; 7.1 (CI, 2.2 to 23.4) for a score of 2; and 13.1 (CI, 3.6 to 48.0) for a score of 3.

Discussion

Because of the variable interpretation of qualitative descriptions of risk by both physicians [4] and patients [24], many have argued for the use of quantitative presentations to enhance patient comprehension. In our study, however, few women were able to apply quantitative information about the benefit of mammography to their perceived risk for death from breast cancer. The inability to make use of such risk reduction information was strongly related to numeracy-which was often limited in our sample, although almost all of the participants had at least a high school education.

How information is framed greatly affects its interpretation [8-20]. For example, both patients and physicians presented with two equally efficacious medications or treatments preferred the one whose benefit was given in relative as opposed to absolute terms [11, 14, 17, 20]. Other work on framing has shown that the same treatment looks more attractive if its risk is expressed in terms of a gain (95% survive) rather than a loss (5% die) [8-1012, 13, 15, 16, 18-20]. Similarly, our study found that framing mattered. Even more important than presenting the risk in a relative or absolute format, however, was adding information on baseline risk. The large improvement in accuracy achieved by adding baseline risk to absolute risk reduction data highlights the importance of grounding patients in their underlying risk.

Although framing was a significant predictor of accuracy, numeracy was more strongly related to the accurate use of quantitative information. Even with our “best” presentation (absolute risk reduction with baseline risk), only one third of the sample accurately applied the information. Our results raise questions about the effective use of quantitative messages and suggest that low levels of numeracy may make a focus on framing premature.

Our study has several possible limitations. First, the response rate was not 100%. However, our 61% completion rate is very good for a mailed survey. If incomplete response had any effect, it probably led us to underestimate the effect of numeracy because it is reasonable to believe that women who chose not to respond may have had more difficulty with numbers. Second, our sample of female veterans may not be representative of the population as a whole. Our sample reported slightly higher income and education levels compared with the overall female population in the United States [25]. These demographic characteristics were associated with higher levels of numeracy in both our study and the National Adult Literacy Survey [21]. On the other hand, our sample was older than the general population, a characteristic associated with lower numeracy. Although the net effect on generalizability is difficult to estimate, our findings and those of the National Adult Literacy Survey suggest that poor numeracy is very common.

Finally, our results may reflect inadequacies in literacy [21, 22], a common problem, as well as numeracy. Women who are unable to read could not have assimilated the information provided. The consistency and appropriateness of responses to the other survey questions (such as those about demographic characteristics and risk factors for breast cancer), however, argue that most women were able to read and understand most of the questions in the survey.

Our finding that women who could not complete the numeracy tasks were less able to use quantitative risk information is not surprising. To apply the information provided about mammography, respondents had to both extract the relevant data from the presentation and perform an arithmetic calculation. In the National Adult Literacy Survey [21], almost half of the respondents could not complete a similar task (for example, they could not calculate the difference between a regular and a sale price from an advertisement).

The inability to apply the quantitative information on risk reduction is troubling. For persons with strong numeracy skills, quantitative messages may convey information concisely and with great precision. However, problems with numeracy are common. For many persons, quantitative expressions may have no meaning or may represent useless and potentially confusing information. Our study highlights problems in communicating the benefit of a screening test to patients.

Ways to address such problems include using better numeric formats, combining numeric and verbal formats, and using visual displays. Work in cognitive psychology suggests that quantitative presentations are more accessible when chance is expressed by using frequency formats rather than probabilities (for example, 10 out of 1000 rather than 1%) [26]. Our findings support this idea: The absolute risk reduction formats (examples of frequency formats) overall outperformed the relative risk reduction formats (which use percentages). The number needed to treat [27] is another example of a simple frequency format that may be particularly useful for communicating with patients.

Messages using both verbal and numeric formats may be especially useful because they combine the accessibility of written language with the precision of numbers. Although verbal formats have been criticized because of imprecision (for example, “small chance” may have a wide range of meaning for different patients and may mean something very different to physicians) [4, 24, 28], they have the distinct advantage of using ordinary language. Much work is needed to learn how to standardize the meaning of ordinary words in communicating chance.

Alternatively, visual displays that do not explicitly rely on quantitative precision may more effectively communicate the risk for disease or the benefit of treatment. Such displays are commonly used outside of the physician's office. A recent article on mammography [29], for example, used shaded dots in addition to traditional graph formats (such as histograms and line graphs) to express a variety of risks. Some work in the psychology literature [30] has focused on how the use of “risk ladders” (in which risks are represented by their vertical position on a ladder) affects a person's perception of risk. The little work available in the medical literature shows that survival curves could be used to elicit patient preferences for treatment (although patients interpreted these curves differently than physicians did) [31] and that women with breast cancer found a “decision board” (which showed the risks and benefits of chemotherapy on an adjustable pie chart) helpful in making decisions about chemotherapy [32]. Formal evaluation of common visual formats is needed, particularly for persons with low levels of numeracy.

As a result of attempts to promote informed decision making, patients are receiving more quantitative information about risks for disease and benefits of treatment. Our results suggest that current formats for presenting such information may be ineffective. If effective communication of quantitative information is important, we can either better educate society to improve numeracy or try to develop communication strategies that overcome innumeracy.

Dr. Black: Department of Community and Family Medicine, Dartmouth Medical School, Hanover, NH 03755.