A Summary to Communicate evidence from systematic reviews to the public improved understanding and accessibility of information: a randomized controlled trial

Journal Club

A Summary to Communicate evidence from systematic reviews to the public improved understanding and accessibility of information: a randomized controlled trial

19 November 2014

Facilitated by PhD candidate Peter Coxeter

Background

People are increasingly demanding to better understand health information to manage their health, although the evidence for treatment benefits and harms (currently written in technical scientific language for other health professionals and researchers) is not accessible to patients, consumers, or members of the public. Research to date has focused on the specifics of how to present and communicate the benefits and harms of treatments.

Authors developed a new format for a plain language summary (PLS) for one Cochrane systematic review (Hemilä and Chalker 20131): a popular topic (Vitamin C for the prevention and treatment of common cold). The new PLS format was established through robust qualitative development that explored issues in qualitative and quantitative presentations of benefit/risk, as well as confidence intervals and tables (Glenton et al. 20102). Research indicates that patients want not only to know how many people will improve or be harmed by treatment, but also how “certain” those numbers are (ie. quality of the evidence).

Paper presented

Santesso et al. 2014 3 conducted a Randomised Controlled Trial (RCT) via the internet in August 2009 to compare the new PLS format to the current narrative format. The study was funded by the Opportunities Fund of the Cochrane Collaboration and ethics approval was granted. Four Cochrane review groups across diverse disease areas in 5 countries (Canada, Norway, Spain, Argentina and Italy) recruited members of the public and patients though a message on Cochrane group Web sites and an email invitation disseminated to local consumer groups and forwarded via local patient, researcher and health professional networks. The PLS were written in English and then translated into Norwegian, Spanish and Italian.

“PICO” elements of the research question:

Participants:      Members of the public and patients ≥16 years of age (who expressed a willingness to participate in the study).

Intervention:      New PLS format for a systematic review. The new PLS format has a more structured presentation using a question and answer approach and communicates information about benefits and harms with reference to both the magnitude of effect and the quality of evidence. Information is presented separately in tables as numbers and symbols and in standardised qualitative statements. Readers can use the qualitative statements, the tables, or both to understand the information.

Comparison:     Current PLS format (revised to include similar background information and language to the new PLS, but maintained current narrative format).

Outcome(s):     Primary – Proportion of people who correctly understood the benefits and harms of the intervention and quality of evidence (5 multiple choice questions, each with 5 response options). The proportion of people who correctly answered a question was averaged over the 5 questions and compared between formats.

Secondary – Proportion of people who correctly understood each of the 5 questions; overall number of correct answers for these 5 questions; comprehension of the purpose and producer of the summary (multiple choice with 3 options); usability, ease of understanding and accessibility (ie. extent to which main findings are easy to find, understand and use by someone making a decision) (7-point Likert scale, 1=strongly disagree to 7=strongly agree); preference for one format over the other (7-point scale, strong preference for one format at each end).

Demographic information – age, language spoken and read, education, and frequency searching the internet for health information.

The “RAMbo” mnemonic to critically appraise the study:

Randomisation: Eligible participants centrally randomised to the new or current PLS format by a staff member unaware of participants demographic or other characteristics. The sequence was generated using block randomisation (40 permuted blocks of 4) generated from third-party software (http://www.randomization.com). A local investigator sent an email to the participant with a link to the online questionnaire and PLS format to which they had been allocated. Participants were not made aware of which PLS was the current or new format. The majority of participants were female (73%), well represented across 10-year age categories, with diverse levels of education – slightly less than one-third held a high school diploma (30%) or a university degree (30%) – and a broad range of health information seeking behaviour on the internet. No systematic differences in baseline characteristics between study groups were observed.

Attrition: 193 people were randomised and allocated to intervention (n=97) or control (n=96) groups; 154 (80%) commenced the study (14 intervention and 17 control participants did not respond to the randomisation email to complete questionnaire); and 143 (74%) completed the study (3 intervention and 8 control participants did not complete questionnaire after logging in) and were analysed.

Measurement (blinded or objective): The survey questionnaire was self-completed by participants online (SurveyMonkey) and the analysis was conducted by a statistician blinded to the PLS format tested.

Results

More participants who received the new PLS format correctly answered the comprehension questions (primary outcome) than those who received the current PLS format (53% vs. 18%; P<0.001). The difference remained statistically significant after adjusting for education level, language and internet experience in a linear regression modelling.

Analyses for each question showed that in 4 of the 5 questions (with exception of the question about the meaning of qualitative statements and understanding risk), the differences in the proportion of people answering the questions correctly were >40% and statistically significant. The median number of questions answered correctly out of 5 was significantly greater in participants who received the new PLS format compared with the current PLS (3, interquartile range [IQR]: 1-4) vs. 1 [IQR: 0-1); P<0.001].

A greater proportion of respondents who received the current PLS format correctly understood the purpose of the summary (ie. results were not about one large study), although overall understanding was fairly low and not statistically significant. More respondents receiving the PLS understood the summary was produced by the Cochrane Collaboration (89% vs. 67%; P<0.001).

More participants exposed to the new PLS responded that the information was reliable, easy to find and understand, and presented in a way to help make a decision and presented the most important effects. All comparisons were statistically significant with the exception of the question regarding ease of understanding.

Across both study arms, respondent had a greater preference for the new PLS format over the existing format (median 3 [“somewhat prefer”], IQR: 1-6), although participants generally preferred the format to which they were exposed first.

Authors’ Discussion

Evidence communicated in qualitative statements, as well as in numbers, improved understanding of benefits and harms and the quality of the evidence. The new format was found to be accessible and usable, and preferred by most participants in several countries. However, only up to 65% answered most comprehension questions correctly and fewer than 50% correctly answered questions about the quality of evidence. The authors note the GRADE working group’s Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) project will conduct further research about communication of evidence and recommendations from guidelines with patients and the public. Several trial limitations were noted by the study authors. High attrition was observed (74% completed the study) and attributed to the two-step approach to of initial invitation and second contact for randomisation. Importantly, there were no systematic differences observed between study groups. Recruitment was conducted in high and middle income countries and the generalizability of results to people living in low income countries is unknown. Recruitment methods may also have led to a selection bias toward people with a special interest in health information. The authors sufficiently justified a possible unfair advantage to respondents being asked the specific number of people experiencing an outcome and quantitative information provided only in the new PLS format.

Journal club commentary

The new PLS evaluated was substantially revised from the current one in the Cochrane systematic review, although it maintained the narrative format. We would have preferred a three-arm RCT (New PLS format vs. revised current PLS vs. current PLS). What discussed what proportion of correct responses is realistic for the public (or indeed health professionals) to attain? Also, how might people or health professionals be taught to assist their understanding of this format? More respondents felt the new PLS format somewhat to strongly disagreed with the statement “The information is reliable (Figure 4, pg. 8), and this poses the question of whether introducing understanding of the quality of evidence paradoxically introduces a level of uncertainty. We also wondered whether the wording of questions may have introduced bias – for example, the question on the comprehension of the purpose of the summary [“This summary is about the results of a large study”; Yes, No, I’m not sure] might be correctly interpreted as a single study or a systematic review and meta-analysis. The evaluation of ‘creative language’ used in standardised qualitative statements about treatment effects might also introduce bias. Despite carefully defining standardised qualitative statements in relation to distinct levels of evidence (eg. ‘will’=high level of evidence; ‘probably’=moderate level of evidence) they were combined on one occasion which was counter-intuitive (eg. “Probably will not decrease how long a cold lasts…”.

Acknowledgement

Thanks to Associate Professor Tammy Hoffmann for identifying the article for Journal club discussion.

References

    1. Hemilä and Chalker. Vitamin C for preventing and treating the common cold. Cochrane Database of Systematic Reviews, 2013, Issue 1. Art. No.:CD000980.pub4).
    2. Glenton C, Santesso N, Rosenbaum, et al. Presenting the results of Cochrane systematic reviews to a consumer audience: A Qualitative study. Medical Decision Making. 2010; 30(5): 566-577.
    3. Santesso N, Rader T, Nilsen ES, et al. A Summary to Communicate evidence from systematic reviews to the public improved understanding and accessibility of information: a randomized controlled trial. Journal of Clinical Epidemiology. 2014; doi:10.1016/j.jclinepi.2014.04.009.