JMIR J Med Internet Res Journal of Medical Internet Research 1438-8871 JMIR Publications Inc. Toronto, Canada v18i5e100 27226241 10.2196/jmir.5145 Original Paper Original Paper Readability Analysis of the Package Leaflets for Biological Medicines Available on the Internet Between 2007 and 2013: An Analytical Longitudinal Study Moorhead Anne Hameen-Anttila Katri Diviani Nicola Piñero-López María Ángeles PharmD,PhD 1 http://orcid.org/0000-0002-4159-0436 Modamio Pilar PharmD,PhD 1 http://orcid.org/0000-0003-3193-6285 Lastra Cecilia F PharmD,PhD 1 http://orcid.org/0000-0002-2983-1284 Mariño Eduardo L PharmD,PhD 1
Clinical Pharmacy and Pharmacotherapy Unit Department of Pharmacy and Pharmaceutical Technology University of Barcelona Faculty of Pharmacy Av. Joan XXIII s/n Barcelona, 08028 Spain 34 934024544 34 934024544 emarino@ub.edu
http://orcid.org/0000-0003-4386-9315 1 Clinical Pharmacy and Pharmacotherapy Unit Department of Pharmacy and Pharmaceutical Technology University of Barcelona Barcelona Spain Corresponding Author: Eduardo L Mariño emarino@ub.edu 052016 25 05 2016 18 5 e100 18 9 2015 10 10 2015 23 11 2015 15 2 2016 ©María Ángeles Piñero-López, Pilar Modamio, Cecilia F Lastra, Eduardo L Mariño. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 25.05.2016. 2016

This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.

 Background

The package leaflet included in the packaging of all medicinal products plays an important role in the transmission of medicine-related information to patients. Therefore, in 2009, the European Commission published readability guidelines to try to ensure that the information contained in the package leaflet is understood by patients.

 Objective

The main objective of this study was to calculate and compare the readability levels and length (number of words) of the package leaflets for biological medicines in 2007, 2010, and 2013.

 Methods

The sample of this study included 36 biological medicine package leaflets that were downloaded from the European Medicines Agency website in three different years: 2007, 2010, and 2013. The readability of the selected package leaflets was obtained using the following readability formulas: SMOG grade, Flesch-Kincaid grade level, and Szigriszt’s perspicuity index. The length (number of words) of the package leaflets was also measured. Afterwards, the relationship between these quantitative variables (three readability indexes and length) and categorical (or qualitative) variables were analyzed. The categorical variables were the year when the package leaflet was downloaded, the package leaflet section, type of medicine, year of authorization of biological medicine, and marketing authorization holder.

 Results

The readability values of all the package leaflets exceeded the sixth-grade reading level, which is the recommended value for health-related written materials. No statistically significant differences were found between the three years of study in the readability indexes, although differences were observed in the case of the length (P=.002), which increased over the study period. When the relationship between readability indexes and length and the other variables was analyzed, statistically significant differences were found between package leaflet sections (P<.001) and between the groups of medicine only with regard to the length over the three studied years (P=.002 in 2007, P=.007 in 2010, P=.009 in 2013). Linear correlation was observed between the readability indexes (SMOG grade and Flesch-Kincaid grade level: r2=.92; SMOG grade and Szigriszt’s perspicuity index: r2=.81; Flesch-Kincaid grade level and Szigriszt’s perspicuity index: r2=.95), but not between the readability indexes and the length (length and SMOG grade: r2=.05; length and Flesch-Kincaid grade level: r2=.03; length and Szigriszt’s perspicuity index: r2=.02).

 Conclusions

There was no improvement in the readability of the package leaflets studied between 2007 and 2013 despite the European Commission’s 2009 guideline on the readability of package leaflets. The results obtained from the different readability formulas coincided from a qualitative point of view. Efforts to improve the readability of package leaflets for biological medicines are required to promote the understandability and accessibility of this online health information by patients and thereby contribute to the appropriate use of medicines and medicine safety.

 package leaflet readability biological medicines online health information Introduction

Today, health care professionals are no longer the only source of information for patients on matters related to health because the new information and communication technologies have increased the capacity of patients to seek information independently [1-3]. That is why patient education is growing in importance; in particular, written health-related materials available on the Internet [4], such as package leaflets provided by manufacturers.

One aspect that influences understanding of written information is literacy, which is defined as “using printed and written information to function in society, to achieve one’s goals, and to develop one’s knowledge and potential” [5]. On the basis of this concept, health literacy is defined as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” [6]. The capacity of the individual is an important aspect in the definition of health literacy, referring to both innate and acquired skills [7].

Health literacy plays an important role in the evaluation of the online health information [8]. An improvement in health literacy leads to a greater ability of patients to understand how to use medicines appropriately and that is why adequate levels of health literacy can improve medication safety and reduce adverse drug reactions [9,10].

Another aspect that also influences the understanding of written information is its readability, which can be measured using readability formulas. By the 1980s, there were some 200 published readability formulas and a large number of studies that attested to their validity [11]. Nowadays, these formulas are widely applied to measure the readability of health-related written materials [12-15]. It is recommended that the readability level of health-related written materials for end users is not higher than the sixth (age 11-12 years) [7,16] or seventh (age 12-13 years) grade [17].

Among health-related written materials, the package leaflet is “a leaflet containing information for the user which accompanies the medicinal product” [18]. It is very important for the transmission of drug-related information to citizens and it can help to supplement and reinforce the information received from health professionals [19], possibly leading to increased adherence and a consequent decrease in health care costs [4,20].

Several studies have highlighted problems associated with the quality of the information contained in the European model of the package leaflet [21,22]. This is in contrast with current European regulations, according to which the contents of the package leaflet must be clear and understandable, enabling the users to act appropriately, and it must be clearly legible in the official language or languages of the Member State [23]. To achieve this aim, and in accordance with Article 65(c) of Directive 2001/83/EC, in 2009, the European Commission published the Guideline on the Readability of the Labelling and Package Leaflet of Medicinal Products for Human Use, which offers guidance on the writing of the labeling and package leaflet to facilitate understanding and the accessibility of their content [24]. The guidance recommends, among other things, the use of simple words with few syllables and avoidance of long sentences. These explicitly refer to variables included in readability formulas (word length and sentence length).

The European Commission has final authority for authorizing the commercialization of human medicines within the European Union via a centralized authorization procedure. This centralized procedure is compulsory for biological medicines, among others [25]. In 1982, an insulin product became the first recombinant medicine approved for sale [26]. Since then, many biological medicines have been authorized for the treatment of a wide variety of diseases. The biological product market is projected to grow over the coming years and that may affect the introduction both of new products and of existing products for additional indications [27]. In January 2013, there were 171 biological medicines authorized in the European Union [28], 12 of which were biosimilars [29].

In a previous study, the readability levels of package leaflets of some biological medicines (n=33) downloaded from the European Medicines Agency (EMA) website was studied between 2007 and 2010. No variation in readability was found between the two years—the readability of the package leaflets was above the recommended values for health-related materials—and there were differences between the readability levels of the different sections of package leaflets [30]. In view of these findings, it seems important to extend the study to 2013 to see whether the guidance on readability has had an effect in package leaflets until that date.

Thus, the main objective of the current study was to calculate and compare the readability level and length (number of words) of the package leaflets of biological medicines in 2007, 2010, and 2013. We hypothesize that there will be an improvement in the readability of package leaflets, especially in 2013, 4 years after publication of the guidance on readability. We considered 4 years to be sufficient time for manufacturers and marketing authorities to have applied the recommendations in the guidelines. We also wanted to analyze a potential link between readability and length. Moreover, the possible influence of some categorical variables on readability and length was also studied.

 Methods Type of Study and Inclusion/Exclusion Criteria

We designed and performed an analytical longitudinal study. The study sample consisted of biological medicine package leaflets that were authorized by the EMA by January 2007 and continued to be authorized in December 2013. Of these, one medicine per active substance was chosen.

The following types of medicines were excluded from the study so as not to introduce bias into the results: vaccines because they are used as a prophylactic measure and, moreover, most are administered by physicians to pediatric patients leading to reduced importance of package leaflets; insulin products because diabetic patients usually know their illness and medicines very well and package leaflet readability is secondary; and botulinum toxin because it was the only biological medicine at the time of the study that contained a toxin as an active substance.

 Data Collection

The package leaflets were downloaded from the EMA website [31] at three different times: January 2007, July 2010, and August 2013. The same pharmaceutical form was chosen in all three years for each medicine.

 Sample Characteristics

As in our previous study [30], the sample of package leaflets (n=36) was divided into five groups depending on the source of the drug [32]: monoclonal antibody (mAb) products, cytokines, therapeutic enzymes, recombinant blood-related products, and recombinant hormones (see Multimedia Appendix 1).

Five of the six sections according to the EMA package leaflet model/template [33] were evaluated: (1) what X is and what it is used for, (2) what you need to know before you take (or use) X, (3) how to take (or use) X, (4) possible side effects, and (5) how to store X. The section (6) “contents of the pack and other information” was excluded because it is considered by patients less important than the other sections [22] and its content was highly similar for all the package leaflets. The “annex” section, which provides information about instructions for use, was also evaluated in the package leaflets where it appeared (10 leaflets in 2007, 11 in 2010, and 12 in 2013).

The evaluated sections of the package leaflets were copied as plain text into individual Microsoft Word 2007 files. Before calculating quantitative variables, the following modifications were made:

Titles, subtitles, citations, tables, graphs, images, references, header tables, figure captions, and the brand name of the medicines were deleted.

All abbreviations, unit and magnitude symbols, numbers, and acronyms were replaced by their full version because when applying the readability formulas these must be treated as if read aloud [34].

Bullets (eg, dashes, numbers, asterisks) were deleted.

Compound words and numbers were considered as a single word.

 Quantitative Variables: Readability Indexes and Length

The quantitative variables calculated for the package leaflets were the length (number of words) and three readability indexes: SMOG grade [34], Flesch-Kincaid grade level [35], and Szigriszt’s perspicuity index [36].

The readability indexes were chosen taking into account the following criteria:

SMOG grade and Flesch-Kincaid grade level are commonly used in recently published health care literature and they have been validated by different methods [37]. This can make it useful to compare the results obtained with the two formulas.

SMOG grade is recommended for use in health-related written materials [1,37,38] because it is the only formula with 100% expected comprehension and is based on more recent criteria for determining reading grade level. For this reason, SMOG grade values are usually higher than Flesch-Kincaid grade level values when both formulas are applied to the same text [37].

The qualitative interpretation of Szigriszt’s perspicuity index was designed to assess the readability of written materials in Spanish, which is the language of the package leaflets analyzed in this study.

The SMOG grade formula has as a variable the number of words with three or more syllables, whereas the Flesch-Kincaid grade level and Szigriszt’s perspicuity index use the number of words per sentence and the number of syllables per word. SMOG grade was calculated manually following the author’s instructions, Flesch-Kincaid grade level was calculated using Microsoft Word 2007 software, and Szigriszt’s perspicuity index was calculated following the author’s instructions (before applying this formula, both the number of words and the number of sentences were obtained using Microsoft Word 2007 software and the number of syllables per word was obtained from previously calculated Flesch-Kincaid grade level values).

SMOG grade and Flesch-Kincaid grade level indicate reading grade level (according to the number of years of schooling required after the age of 6 years to understand the text). Thus, sixth-grade level is equivalent to age 11 to 12 years, seventh grade level is equivalent to age 12 to 13, and so on. However, Szigriszt’s perspicuity index has seven qualitative ranges that relate the score obtained and the quality of the text: very easy (85-100), easy (75-85), rather easy (65-75), standard (50-65), rather difficult (35-50), difficult (15-35), and very difficult (0-15) [36]. As the Szigriszt’s perspicuity index score increases, the ease of reading the text also increases. Therefore, the readability is directly proportional to Szigriszt’s perspicuity index and inversely proportional to SMOG grade and Flesch-Kincaid grade level.

The length (numbers of words) of the texts was the fourth variable obtained from the package leaflets because more words on package leaflets can decrease the capacity to find certain information and decrease motivation to read the package leaflet and confidence in using the medicine correctly after reading it [39]. The length was obtained using Microsoft Word 2007 software.

To obtain these variables, the whole text was evaluated in this study to avoid bias that could be introduced in the choice of samples.

 Categorical Variables

The influence of some categorical variables on readability and length was studied. These variables were:

Year of downloading the package leaflet (three groups): 2007 (n=36), 2010 (n=36), and 2013 (n=36).

Section of package leaflet (six groups): (1) what X is and what it is used for (n=36), (2) what you need to know before you take (or use) X (n=36), (3) how to take (or use) X (n=36), (4) possible side effects (n=36), (5) how to store X (n=36), and annex (n=10 in 2007, n=11 in 2010, n=12 in 2013).

Group of medicine according to its source (five groups): mAb products (n=6), cytokines (n=11), therapeutic enzymes (n=4), recombinant blood-related products (n=9), and recombinant hormones (n=6).

Date of first authorization of medicine (two groups): 1995-1999 (n=16) and 2000-2002 (n=20).

Marketing authorization holder with the real names replaced by letters (five groups in 2007 and four groups in both 2010 and 2013): A (n=3; only in 2007), B (n=3), C (n=5 in 2007, n=4 in both 2010 and 2013), D (n=3), and E (n=3). For this variable, we only considered laboratories with at least three authorized medicines.

For the second variable (section of package leaflet), the values per section were taken into account; for the rest of the variables, the mean of each readability index per package leaflet and the total length of each package leaflet were considered to have a single value per package leaflet.

 Statistical Analysis

Statistical calculations and box plots were performed using Deducer (version R 2.15.0) and R-Commander (version R 3.1.0).

The test of normality used in all the comparative studies was the Shapiro-Wilk test. Taking into account the normality results, the following statistical tests were used to examine the relationship between quantitative and categorical variables: (1) repeated-measures analysis of variance and the Friedman test to compare different groups of each quantitative variable according to package leaflet download date; (2) the Kruskal-Wallis test to compare different groups of quantitative variables according to package leaflet section, source of biological medicine, and marketing authorization holder; and (3) the Mann-Whitney U test, to compare different groups of quantitative variables according to date of first authorization of the medicine. Furthermore, the degree of correlation between the readability indexes and length was estimated via the coefficient of determination. A P value less than .05 was considered significant in all the hypothesis tests.

Nonstatistical calculations and the other graphs (both line graphs and scatterplots) were generated using Microsoft Excel 2007 software.

 Results Effect of Package Leaflet Year on Readability and Length

Table 1 shows readability (mean values for package leaflet) and length results for package leaflet and year studied.

All the SMOG grade and Flesch-Kincaid grade level values exceeded the recommended readability levels for health-related written materials (all SMOG grade and Flesch-Kincaid grade level values were much higher than 6). In addition, all the Szigriszt’s perspicuity index values were less than 75; that is, no package leaflet was easy to understand according to this scale. The respective statistical analysis of the results in Table 1 is presented in Table 2.

No statistically significant differences were found in the readability values between the three years (P=.40 in SMOG grade, P=.22 in Flesch-Kincaid grade level, P=.20 in Szigriszt perspicuity index), but differences emerged in terms of the length (P=.002), with the length of the package leaflets increasing over the 6-year period (Figure 1).

Mean values of the three readability indexes and total length for package leaflet (n=36) and year studied.

Medicine SMOG grade, mean (SD) Flesch-Kincaid grade level, mean (SD) Szigriszt’s perspicuity index, mean (SD) Length
2007 2010 2013 2007 2010 2013 2007 2010 2013 2007 2010 2013
Agalsidase alfa 16.6 (3.5) 16.6 (3.5) 16.6 (3.5) 16.4 (3.8) 16.3 (4.0) 16.0 (3.6) 58.4 (12.3) 58.9 (13.5) 60.2 (12.6) 905 907 940
Agalsidase beta 18.4 (3.1) 18.0 (5.1) 18.4 (4.8) 19.5 (3.6) 18.5 (6.2) 18.2 (5.6) 45.4 (11.6) 50.5 (21.7) 51.6 (19.3) 986 851 1131
Anakinra 14.8 (1.7) 14.8 (1.7) 15.2 (1.5) 13.3 (1.9) 13.4 (1.8) 13.6 (1.8) 68.4 (8.0) 68.1 (8.1) 67.3 (7.9) 1349 1429 1584
Basiliximab 17.0 (3.6) 17.2 (4.7) 17.2 (4.7) 16.6 (3.8) 16.5 (5.3) 16.7 (5.6) 56.3 (13.5) 58.5 (19.7) 58.0 (20.6) 975 949 970
Choriogonadotropin alfa 16.8 (1.9) 16.8 (1.9) 17.6 (2.6) 16.3 (2.8) 16.3 (2.8) 17.4 (3.6) 56.0 (12.1) 56.0 (12.1) 52.9 (15.6) 1217 1217 1705
Darbepoetin alfa 16.3 (1.8) 16.2 (1.3) 16.5 (1.4) 15.4 (2.3) 15.0 (2.4) 15.5 (2.5) 60.8 (8.9) 62.1 (10.4) 59.9 (10.4) 2326 2125 2324
Desirudin 16.2 (1.3) 17.6 (2.7) 17.6 (2.7) 15.2 (1.6) 16.8 (3.2) 16.8 (3.2) 60.9 (6.7) 55.9 (11.9) 55.9 (11.9) 1253 1167 1167
Epoetin beta 16.6 (2.1) 16.4 (2.7) 16.4 (1.7) 16.2 (2.7) 15.5 (2.6) 15.6 (2.6) 56.6 (10.4) 60.0 (10.5) 59.8 (10.5) 2587 2529 2579
Eptacog alfa (activated) 15.2 (1.8) 16.2 (1.8) 16.2 (1.8) 14.5 (2.1) 15.5 (2.3) 15.4 (2.2) 63.7 (8.2) 62.1 (7.1) 62.7 (6.7) 1158 1426 1431
Eptotermin alfa 18.6 (3.9) 18.6 (3.5) 16.8 (1.6) 18.1 (5.4) 18.1 (5.1) 16.1 (1.8) 52.8 (17.4) 53.7 (15.5) 59.0 (6.0) 1432 1608 1057
Etanercept 17.2 (2.8) 17.5 (2.7) 18.2 (2.9) 16.5 (3.8) 16.5 (3.3) 16.9 (3.4) 58.3 (13.7) 58.6 (12.1) 58.6 (11.4) 3109 3973 4555
Follitropin alfa 17.8 (2.6) 17.8 (2.6) 17.5 (3.1) 17.6 (3.3) 17.6 (3.3) 17.2 (3.8) 51.8 (12.5) 51.8 (12.5) 54.2 (15.4) 2560 2561 3278
Human coagulation factor IX 16.6 (1.9) 16.8 (2.2) 16.8 (2.2) 16.0 (1.9) 16.2 (2.3) 16.2 (2.3) 57.4 (7.3) 57.2 (9.3) 57.1 (9.3) 1737 1874 1874
Human protein C 17.6 (2.9) 17.6 (2.9) 17.8 (3.2) 17.0 (3.4) 17.0 (3.5) 17.0 (3.5) 55.0 (12.0) 55.4 (12.1) 55.5 (12.3) 1761 1760 1768
Imiglucerase 18.2 (2.2) 18.0 (3.5) 18.2 (3.7) 18.0 (2.2) 17.9 (4.1) 17.9 (4.1) 50.4 (8.3) 51.8 (13.7) 51.9 (13.9) 1155 1062 1119
Infliximab 17.8 (2.9) 17.6 (4.8) 17.8 (5.3) 18.3 (3.2) 17.1 (5.6) 17.4 (6.1) 48.4 (12.0) 55.2 (18.7) 54.1 (20.1) 2289 2701 3134
Interferon alfa-2b 17.7 (3.4) 17.8 (3.3) 17.5 (3.1) 16.6 (3.7) 16.6 (3.9) 16.7 (3.8) 57.7 (13.9) 57.5 (14.5) 57.2 (14.4) 3833 4006 4146
Interferon beta-1a 17.6 (2.5) 15.5 (1.8) 15.3 (1.5) 16.8 (2.8) 14.1 (2.2) 14.1 (2.2) 55.4 (10.8) 64.7 (9.3) 64.6 (9.1) 2640 2591 2630
Interferon beta-1b 17.3 (2.7) 17.7 (3.1) 17.7 (3.1) 16.8 (2.7) 17.0 (4.0) 16.9 (4.0) 55.6 (9.8) 55.8 (14.1) 55.8 (14.3) 4868 5463 5450
Lutropin alfa 17.8 (2.9) 17.8 (2.9) 17.8 (3.0) 17.1 (3.4) 17.3 (3.6) 17.1 (3.2) 54.7 (12.4) 53.5 (13.3) 54.0 (11.5) 1794 2110 2346
Moroctocog alfa 18.0 (3.5) 17.6 (1.3) 18.2 (1.9) 17.2 (3.7) 16.5 (2.0) 16.9 (2.2) 54.1 (13.6) 57.0 (8.1) 55.5 (8.7) 1604 2169 2164
Nonacog alfa 17.0 (1.4) 16.8 (1.6) 17.2 (1.9) 16.2 (1.8) 15.7 (2.7) 16.2 (2.7) 58.1 (8.0) 60.5 (10.5) 58.3 (10.4) 2490 2743 2432
Octocog alfa 17.6 (2.9) 17.4 (2.3) 17.4 (2.3) 17.3 (2.6) 17.1 (2.7) 16.8 (2.0) 52.9 (8.6) 53.3 (10.4) 54.8 (7.2) 2029 2403 2249
Palivizumab 15.6 (2.5) 16.6 (3.3) 16.6 (3.3) 15.3 (2.7) 15.9 (4.2) 16.0 (4.3) 61.0 (11.8) 59.3 (18.2) 59.4 (18.2) 548 526 534
Pegfilgrastim 15.5 (1.6) 16.2 (2.6) 15.7 (2.3) 14.3 (2.4) 14.7 (3.0) 14.3 (2.7) 65.4 (9.8) 65.3 (9.7) 66.1 (9.0) 1532 1623 1521
Peginterferon alfa-2a 17.3 (5.4) 15.8 (1.6) 16.2 (2.0) 16.1 (6.5) 14.5 (2.5) 14.8 (2.6) 60.5 (21.2) 64.4 (12.3) 63.8 (11.7) 3005 3545 4151
Peginterferon alfa-2b 17.7 (4.5) 18.5 (5.0) 18.7 (5.8) 17.5 (5.0) 18.1 (6.9) 18.6 (7.3) 53.8 (16.2) 52.6 (18.5) 52.1 (22.2) 3821 4976 5752
Rasburicase 16.2 (1.9) 17.8 (3.4) 18.0 (3.5) 15.7 (3.2) 17.3 (4.1) 17.7 (4.1) 57.4 (14.0) 53.3 (15.8) 52.1 (16.1) 766 714 745
Reteplase 16.8 (2.5) 16.8 (2.5) 16.8 (2.5) 15.9 (2.6) 15.9 (2.6) 15.9 (2.6) 60.4 (9.9) 60.4 (9.9) 60.4 (9.9) 1847 1905 1905
Rituximab 16.2 (2.9) 17.4 (4.5) 18.0 (4.8) 15.0 (3.0) 16.6 (5.3) 17.1 (5.6) 64.0 (10.8) 59.0 (17.3) 57.7 (17.7) 1505 2109 2702
Somatropin 17.0 (2.9) 17.0 (2.9) 17.5 (3.8) 16.7 (3.6) 16.7 (3.6) 17.4 (4.7) 54.9 (15.3) 54.9 (15.1) 53.4 (17.2) 4041 4043 4388
Sulesomab 16.2 (3.3) 16.6 (2.5) 16.6 (3.2) 17.9 (2.2) 17.2 (2.5) 17.1 (3.0) 45.9 (12.2) 51.4 (10.2) 51.6 (11.6) 800 955 962
Tasonermin 19.2 (2.5) 17.8 (3.6) 18.6 (3.8) 19.4 (2.9) 16.9 (4.1) 18.1 (4.3) 47.6 (9.7) 56.9 (18.0) 52.7 (14.5) 2168 1433 1456
Tenecteplase 17.8 (2.8) 18.0 (3.2) 17.4 (1.8) 17.7 (3.0) 17.6 (3.4) 17.0 (2.1) 51.4 (12.4) 51.9 (14.3) 54.6 (10.3) 878 890 1270
Thyrotropin alfa 18.0 (3.6) 17.2 (3.0) 17.4 (3.0) 17.4 (4.4) 16.4 (3.5) 16.7 (3.4) 53.2 (16.8) 56.8 (14.0) 55.7 (13.3) 1219 1240 1286
Trastuzumab 17.0 (3.4) 17.0 (3.7) 17.4 (3.2) 16.5 (3.8) 16.6 (4.0) 17.0 (3.3) 58.7 (13.3) 58.5 (14.3) 56.1 (10.5) 1625 1699 1858

Descriptive statistics of the three readability indexes and length by studied year and results of the hypothesis tests.

Variable and year n Mean (SD) 95% CI Median Range P
SMOG grade .40a
2007 36 17.1 (1.0) 16.8-17.4 17.1 14.8-19.2
2010 36 17.1 (0.8) 16.9-17.4 17.3 14.8-18.6
2013 36 17.2 (0.9) 16.9-17.5 17.4 15.2-18.7
Flesch-Kincaid grade level .22b
2007 36 16.6 (1.3) 16.2-17.1 16.6 13.3-19.5
2010 36 16.5 (1.1) 16.1-16.9 16.6 13.4-18.5
2013 36 16.6 (1.1) 16.2-16.9 16.9 13.6-18.6
Szigriszt’s perspicuity index .20b
2007 36 56.2 (5.2) 54.4-58.0 56.2 45.4-68.4
2010 36 57.3 (4.3) 55.8-58.8 56.9 50.5-68.1
2013 36 57.1 (4.2) 55.7-58.5 56.0 51.6-67.3
Length .002b
2007 36 1939 (1031) 1590-2288 1681 548-4868
2010 36 2091 (1210) 1682-2501 1817 526-5463
2013 36 2238 (1335) 1786-2690 1866 534-5752

a Repeated-measures ANOVA.

b Friedman test.

Evolution of the medians of length of the package leaflets studied.

 Effect of Package Leaflet Section on Readability and Length

Table 3 and Multimedia Appendix 2 describe the readability indexes and length per package leaflet section in the three years studied. Statistically significant differences can be observed between the four variables and the six sections: (1) what X is and what it is used for, (2) what you need to know before you take (or use) X, (3) how to take (or use) X, (4) possible side effects, (5) how to store X, and (6) annex (P<.001).

These results (Table 3, Multimedia Appendix 2) show the following order of readability of the sections of the package leaflets (from easiest to understand to most difficult): (1) section 5, (2) annex, (3) section 3, (4) section 2, (5) section 1, and (6) section 4. According to the Szigriszt’s perspicuity index medians (Table 3) and the scale of perspicuity level, section 4 was rather difficult, section 1 was standard in 2007 and rather difficult in 2010 and 2013, sections 2 and 3 were standard, and section 5 and the annex were rather easy. Thus, sections 1 and 4 were rather difficult to understand in 2010 and 2013, and the readability of section 4 (the most difficult section) decreased during the period studied (Figure 2).

From the median length (Table 3) of the sections of the package leaflets, the following order was determined from longest to shortest: (1) annex, (2) section 2, (3) section 3, (4) section 4, (5) section 1, and (6) section 5. Section 3 decreased in length during the study years, whereas section 2 increased (Figure 2).

Descriptive statistics of the three readability indexes and length by package leaflet section and year studied.

Variable, year, and section n Mean (SD) 95% CI Median Range Pa
SMOG grade
2007 <.001
1. What X is and what it is used for 36 18.4 (2.1) 17.7-19.1 18.0 15-24
2. What you need to know before you take (or use) X 36 17.2 (1.3) 16.8-17.6 17.0 14-19
3. How to take (or use) X 36 16.3 (1.4) 15.8-16.7 16.0 14-21
4. Possible side effects 36 20.3 (2.6) 19.5-21.2 20.0 15-28
5. How to store X 36 13.7 (1.2) 13.3-14.1 14.0 11-16
6. Annex 10 15.0 (0.7) 14.5-15.5 15.0 14-16
2010 <.001
1. What X is and what it is used for 36 18.5 (1.8) 17.9-19.1 19.0 15-23
2. What you need to know before you take (or use) X 36 17.2 (1.3) 16.8-17.6 17.0 14-20
3. How to take (or use) X 36 16.1 (1.2) 15.7-16.5 16.0 14-20
4. Possible side effects 36 20.7 (2.8) 19.7-21.7 20.5 15-28
5. How to store X 36 13.7 (1.0) 13.4-14.1 14.0 11-16
6. Annex 11 14.9 (0.7) 14.4-15.4 15.0 14-16
2013 <.001
1. What X is and what it is used for 36 18.6 (1.9) 17.9-19.2 18.5 15-23
2. What you need to know before you take (or use) X 36 17.4 (1.3) 16.9-17.8 18.0 14-20
3. How to take (or use) X 36 16.2 (1.2) 15.8-16.6 16.0 14-19
4. Possible side effects 36 20.9 (3.3) 19.8-22.0 21.0 16-30
5. How to store X 36 13.9 (1.0) 13.5-14.2 14.0 11-16
6. Annex 12 14.9 (0.7) 14.5-15.3 15.0 14-16
Flesch-Kincaid grade level
2007 <.001
1. What X is and what it is used for 36 18.4 (2.7) 17.5-19.3 18.0 14.4-25.6
2. What you need to know before you take (or use) X 36 16.4 (1.5) 15.9-16.9 16.5 12.6-18.7
3. How to take (or use) X 36 16.0 (1.9) 15.3-16.6 15.7 12.7-20.9
4. Possible side effects 36 20.0 (3.0) 19.0-21.0 19.3 13.8-28.9
5. How to store X 36 13.0 (1.7) 12.4-13.6 12.7 10.0-16.7
6. Annex 10 12.9 (1.0) 12.2-13.6 12.8 11.5-14.9
2010 < .001
1. What X is and what it is used for 36 18.6 (2.3) 17.8-19.4 19.0 14.5-24.8
2. What you need to know before you take (or use) X 36 16.6 (1.5) 16.1-17.1 16.6 13.3-19.7
3. How to take (or use) X 36 15.5 (1.6) 15.0-16.0 15.4 12.0-19.0
4. Possible side effects 36 20.4 (3.4) 19.2-21.5 20.0 13.4-29.6
5. How to store X 36 12.2 (1.3) 11.7-12.6 12.3 9.0-16.0
6. Annex 11 12.8 (0.9) 12.2-13.4 12.8 11.3-14.1
2013 <.001
1. What X is and what it is used for 36 18.5 (2.2) 17.8-19.3 18.3 14.7-23.2
2. What you need to know before you take (or use) X 36 16.7 (1.5) 16.2-17.2 17.0 13.1-20.0
3. How to take (or use) X 36 15.6 (1.7) 15.1-16.2 15.2 12.2-19.5
4. Possible side effects 36 20.5 (3.9) 19.2-21.8 20.1 14.9-32.6
5. How to store X 36 12.5 (1.2) 12.1-12.9 12.7 9.0-15.2
6. Annex 12 12.9 (0.9) 12.3-13.4 13.1 11.2-14.1
Szigriszt’s perspicuity index
2007 <.001
1. What X is and what it is used for 36 49.0 (9.6) 45.8-52.3 51.0 31.9-64.1
2. What you need to know before you take (or use) X 36 56.5 (5.8) 54.5-58.4 56.6 44.7-69.5
3. How to take (or use) X 36 58.0 (8.7) 55.1-61.0 59.7 29.6-71.3
4. Possible side effects 36 44.7 (9.5) 41.5-47.9 45.6 20.0-67.0
5. How to store X 36 69.5 (8.5) 66.6-72.4 70.7 43.1-82.0
6. Annex 10 71.7 (3.9) 68.9-74.5 72.1 64.1-78.2
2010 <.001
1. What X is and what it is used for 36 48.5 (8.1) 45.8-51.3 47.2 36.4-63.4
2. What you need to know before you take (or use) X 36 55.9 (6.0) 53.8-57.9 56.2 43.5-66.1
3. How to take (or use) X 36 60.2 (6.9) 57.8-62.5 60.7 41.0-73.4
4. Possible side effects 36 44.4 (10.4) 40.9-47.9 44.8 17.8-68.6
5. How to store X 36 74.0 (6.5) 71.8-76.2 74.3 54.3-88.8
6. Annex 11 72.1 (3.3) 69.9-74.3 71.2 67.4-78.2
2013 <.001
1. What X is and what it is used for 36 48.8 (8.6) 45.9-51.7 50.0 28.3-63.4
2. What you need to know before you take (or use) X 36 55.9 (5.9) 53.9-57.9 55.9 42.1-68.3
3. How to take (or use) X 36 59.8 (7.5) 57.3-62.3 60.4 39.0-72.6
4. Possible side effects 36 44.2 (11.3) 40.4-48.0 44.8 10.7-63.0
5. How to store X 36 72.7 (5.6) 70.8-74.6 72.2 58.8-88.8
6. Annex 12 71.6 (3.2) 69.6-73.6 70.9 67.4-78.0
Length
2007 <.001
1. What X is and what it is used for 36 166 (75) 140-191 157 62-433
2. What you need to know before you take (or use) X 36 518 (323) 409-628 502 89-1641
3. How to take (or use) X 36 498 (351) 379-617 422 89-1331
4. Possible side effects 36 399 (236) 320-479 331 113-984
5. How to store X 36 102 (39) 88-115 100 32-168
6. Annex 10 922 (519) 551-1294 646 481-2114
2010 < .001
1. What X is and what it is used for 36 163 (93) 131-194 138 54-401
2. What you need to know before you take (or use) X 36 546 (337) 432-660 478 92-1572
3. How to take (or use) X 36 477 (339) 362-592 367 93-1332
4. Possible side effects 36 491 (364) 368-614 367 75-1726
5. How to store X 36 116 (46) 101-132 116 22-229
6. Annex 11 977 (588) 582-1372 645 479-2114
2013 <.001
1. What X is and what it is used for 36 168 (96) 136-201 140 54-399
2. What you need to know before you take (or use) X 36 618 (388) 487-750 530 92-1572
3. How to take (or use) X 36 471 (327) 361-582 350 88-1244
4. Possible side effects 36 532 (426) 388-677 342 75-1976
5. How to store X 36 132 (50) 115-149 127 22-268
6. Annex 12 947 (572) 584-1310 644 479-2114

a Kruskal-Wallis test.

Evolution of the medians by year of SMOG grade, Flesch-Kincaid grade level, Szigriszt's perspicuity index, and length for the package leaflet sections. Section 1: what X is and what it is used for; section 2: what you need to know before you take (or use) X; section 3: how to take (or use) X; section 4: possible side effects; section 5: how to store X; and annex.

 Effect of Source of Biological Medicine on Readability and Length

When comparing readability levels as a function of the source of the medicine, no statistically significant differences were observed between the five groups (mAb products, cytokines, therapeutic enzymes, recombinant blood-related products and recombinant hormones) of package leaflets (SMOG grade: P=.44 in 2007, P=.31 in 2010, P=.61 in 2013; Flesch-Kincaid grade level: P=.46 in 2007, P=.11 in 2010, P=.09 in 2013; Szigriszt’s perspicuity index: P=.34 in 2007, P=.08 in 2013), except when considering Szigriszt’s perspicuity index in 2010 (P=.03). However, differences could be observed between the five groups in the length of the package leaflets in the 3 years studied (P=.002 in 2007, P=.007 in 2010, P=.009 in 2013). These differences in length were mainly due to the difference between therapeutic enzymes and cytokines when applying the Bonferroni posttest for multiple comparisons (Table 4). Cytokines had the longest package leaflets: a median of 2640 words (range 1349-4868) in 2007, 2591 words (range 1429-5463) in 2010, and 2630 (range 1456-5752) in 2013; and therapeutic enzymes had the shortest: a median of 946 words (range 766-1155) in 2007, 879 words (range 714-1062) in 2010, and 1030 (range 745-1131) in 2013.

P values obtained by applying the Bonferroni posttest for multiple comparisons of package leaflet length.

Year and type of medicine P value
mAb products Cytokines Therapeutic enzymes Recombinant blood-related products
2007
Cytokines .048 — — —
Therapeutic enzymes .99 .02 — —
Recombinant blood-related products .99 .06 .20 —
Recombinant hormones .99 .99 .10 .99
2010
Cytokines .37 — — —
Therapeutic enzymes .99 .02 — —
Recombinant blood-related products .99 .47 .11 —
Recombinant hormones .99 .99 .10 .99
2013
Cytokines .78 — — —
Therapeutic enzymes .99 .02 — —
Recombinant blood-related products .99 .20 .03 —
Recombinant hormones .99 .99 .38 .99
Effect of Date of First Authorization on Readability and Length

No statistically significant differences were observed in readability levels and length between the leaflets of medicines authorized in 1995-1999 (n=16) and those authorized in 2000-2002 (n=20) as a function of the year of first authorization of the biological medicine (SMOG grade: P=.81 in 2007, P=.79 in 2010, P=.99 in 2013; Flesch-Kincaid grade level: P=.79 in 2007, P=.81 in 2010, P=.94 in 2013; Szigriszt’s perspicuity index: P=.57 in 2007, P=.78 in 2010, P=.96 in 2013; length: P=.90 in 2007, P=.95 in 2010, P=.90 in 2013). However, differences in the descriptive analysis of the data were observed between the medians of the length of the groups: the medians of length of the medicines authorized in 1995-1999 were higher than those of medicines authorized in 2000-2002 in the three years studied. The medians of length for the medicines authorized in 1995-1999 were 1726 (range 548-4868) in 2007, 2007 (range 526-5463) in 2010, and 2035 (range 534-5450) in 2013, although those of the medicines authorized in 2000-2002 were 1681 (range 766-4041) in 2007, 1730 (range 714-4976) in 2010, and 1813 (range 745-5752) in 2013.

 Effect of Marketing Authorization Holder on Readability and Length

Even though differences were found between groups for the readability indexes in 2007 and for Szigriszt’s perspicuity index in 2010 and 2013 (SMOG grade: P=.01 in 2007; Flesch-Kincaid grade level: P=.01 in 2007; Szigriszt’s perspicuity index: P=.01 in 2007, P=.03 in 2010, P=.04 in 2013), these differences not were observed in the Bonferroni posttest for multiple comparisons (P>.05). This was probably due to a lack of statistical power because the samples were considered too small. For this reason, regarding the comparative study of groups of package leaflets as a function of marketing authorization holder, no statistical evidence was available to show the influence of marketing authorization holder on readability and length.

 Relationship Between Quantitative Variables

The results of studying the relationship between the three readability indexes and length are presented in Table 5.

Correlations between the three readability indexes and length.

Variable SMOG grade Flesch-Kincaid grade level Szigriszt’s perspicuity index Length
r2 Pa r2 Pa r2 Pa r2 Pa
SMOG grade — — .92 <.001 .81 <.001 .05 <.001
Flesch-Kincaid grade level — — — — .95 <.001 .03 <.001
Szigriszt’s perspicuity index — — — — — — .02 .003

a ANOVA.

Although the P value was significant for all pairs of variables, taking into account the coefficient of determination, only a linear correlation between the readability indexes was evident. This correlation can be observed in the respective scatterplots (Figure 3).

Scatterplots relating SMOG grade, Flesch-Kincaid grade level, Szigriszt's perspicuity index, and length.

 Discussion Principal Findings

The objective of this study was to determine the readability and length of package leaflets for biological medicines in three different years. Our results show that none of the package leaflets evaluated met the recommended readability levels for health-related written materials. Therefore, the package leaflets are not in line with European legislation, according to which they must be clearly legible and understandable [23]. This finding could negatively affect patient understanding of the information contained in package leaflets and result in a reduction of adherence intentions [40,41]. A recent study in which patient information materials were evaluated in terms of readability and variety of content found that these materials did not promote health literacy and were only accessible to a proportion of higher skilled patients, which could ultimately increase inequalities in health [42].

Moreover, no improvement in readability was observed over the 6-year period analyzed. This result was not in line with our expectations because the European Commission Guideline [24] recommended the use of words with few syllables and avoidance of long sentences in 2009. In accordance with this guideline, the readability of package leaflets should have increased, but this is not the case.

Furthermore, the number of words in package leaflets increased between 2007 and 2013, which is also a problem for patients. This trend has existed in Europe for a number of years [39]. The observed increase in the length of the package leaflets over the 6-year period studied could be a consequence of prevention policies of the EMA and the time since first authorization that may lead to there being more information related to pharmacovigilance.

When the sections of package leaflets were compared, differences between them were observed in both readability and length in all the years studied. The most difficult sections contain information about therapeutic indications (section 1: what X is and what it is used for) and side effects of the medicines (section 4: possible side effects). This information is considered very important by patients [22] because it highlights the importance of promoting an understanding of the need for medicines and individual risk for side effects when taking medicines [43], and allows them to undertake a rational benefit-risk assessment of their medication [44]. Herber et al [44] stated that “package leaflets need to convey potential risk information in a language that is less frightening while retaining the information content required to make informed decisions about the prescribed medication.” In contrast, the most understandable section (section 5: how to store X) is considered less important by patients [22].

Regarding the length of sections, the shortest section was section 5 (how to store X) and it was also the most understandable. In contrast, the annex section was the longest section, but it was more understandable than other sections because it has shorter sentences and contains fewer technical terms. Moreover, section 4 (possible side effects) was not the longest section, but it was the least understandable as mentioned previously. This is because in most package leaflets it contains a long list of difficult medical words, which can prevent appropriate interpretation by patients [39].

In relation to the studied correlation between the three readability indexes and total length, only the readability indexes showed an acceptable linear relationship, which was not observed between the readability indexes and length. Thus, we can conclude that the readability of package leaflets is not associated with their length. Fitzsimmons et al [1] assessed the readability of online consumer-oriented Parkinson’s disease information using the Flesch-Kincaid grade level and SMOG grade formulas. They found that webpage length was not associated with readability, suggesting that reading difficulty of websites evaluated was independent of word count. Nevertheless, it is recommendable to reduce the length of package leaflets to make them easier to read and to motivate the patient to access and understand them [1,39]. Indeed, some authors have proposed an alternative template for European package leaflets [45].

 Limitations

Our study has some limitations. First of all, one limitation is the lack of a test of package leaflets according to target patient groups. According to Article 59(3) of Directive 2004/27/EC, package leaflets should reflect the results of user testing to ensure readability. Methods to assess package leaflets using patients have been published [24,46,47], but there are also different studies published in which the results obtained by applying readability formulas and those using user testing have been consistent [48]. Thus, before undertaking a user test or any other form of user consultation [24], the use of formulas can be considered a first step to predict understandability and to initially identify readability problems independently of the type of patient.

A second limitation of this research is the lack of assessment of other characteristics of package leaflets that also improve their readability, such as font, figures, design, and layout [16,24,47,49]. It is important to point out that there are several indirect instruments that assess these characteristics in health-related written materials, such as Suitability Assessment of Materials (SAM) [16] and Patient Education Materials Assessment Tool (PEMAT) [50].

Lastly, not all biological medicines were considered in this study. A sample of 36 biological medicines authorized in both 2007 and 2013 was analyzed taking into account the inclusion and exclusion criteria established. This sample constituted 36 of 61 (approximately 60%) medicines that complied with these criteria in January 2007 and only 36 of 126 (approximately 30%) in January 2013 [28].

Nevertheless, three readability formulas were used (and one of them especially for Spanish) to provide an objective measure of difficulty [11] of reading package leaflets. Moreover, the criteria for obtaining readability indexes have been explained in detail, allowing future comparisons with other studies. Furthermore, to avoid interpretation problems due to a bias in the selection of text samples, all the information contained in the package leaflets was used to calculate the quantitative variables. Finally, this longitudinal study considered adaptation to EU guidelines over a 6-year period of the same health-related written materials.

 Future Implications and Conclusions

Most studies to date have shown that the materials targeted at patients are written with readability levels that make it difficult for the materials to be understood by most people. Package leaflets are this type of written material; therefore, they are no exception to what has been confirmed.

We continue to believe that it would be more than reasonable for applicants and marketing authorization holders, who are responsible for drawing up the package leaflets according to the EMA instructions, to be more conscious of the need to improve the readability of package leaflets together with a decrease of their length. This would increase the usefulness of package leaflets and access to the information they contain by patients.

In the same way, it is advisable for those responsible for drawing up the package leaflets to measure the readability with some of the formulas applied here (which we have shown to be highly correlated), as a method adopted in parallel to direct methods using patients. In this way, taking the measures as soon as possible and, depending on the results, package leaflets could be revised if necessary. Future studies are needed to explain the reasons why the readability of package leaflets has not improved over the years considered in this study, even though the EU legislation and guidelines have changed.

Finally, we suggest an alternative type of written medicine-related information targeted at patients that covers their information needs because the current package leaflet in the EU has proved to be a barrier that has not introduced significant changes in either its readability or length. The alternative written material should be more concise (shorter) than the current package leaflets and it should be written in clear language that is comprehensible for patients.

 Multimedia Appendix 1

Biological medicines studied (n=36).

 Multimedia Appendix 2

Descriptive statistics of SMOG grade, Flesch-Kincaid grade level, Szigriszt's perspicuity index, and length for each package leaflet section and year studied. Section 1: what X is and what it is used for; section 2: what you need to know before you take (or use) X; section 3: how to take (or use) X; section 4: possible side effects; section 5: how to store X; and annex.

 Abbreviations EMA

European Medicines Agency

 EU

European Union

 mAb

monoclonal antibody

MAP-L acquired, analyzed and interpreted data, and drafted and approved the final submitted manuscript. PM conceived of the study, analyzed and interpreted data, and drafted and approved the final submitted manuscript. CFL made suggestions for data interpretation and approved the final submitted manuscript. ELM conceived of the study, analyzed and interpreted data, and drafted and approved the final submitted manuscript.

None declared.

 Fitzsimmons PR Michael BD Hulley JL Scott GO A readability assessment of online Parkinson's disease information J R Coll Physicians Edinb 2010 12 40 4 292 296 10.4997/JRCPE.2010.401 21132132 Patel PP Hoppe IC Ahuja NK Ciminello FS Analysis of comprehensibility of patient information regarding complex craniofacial conditions J Craniofac Surg 2011 07 22 4 1179 1182 10.1097/SCS.0b013e31821c00e4 21778840 00001665-201107000-00003 Beck F Richard J Nguyen-Thanh V Montagni I Parizot I Renahy E Use of the internet as a health information resource among French young adults: results from a nationally representative survey J Med Internet Res 2014 05 16 5 e128 10.2196/jmir.2934 24824164 v16i5e128 PMC4051740 Laugesen J Hassanein K Yuan Y The impact of Internet health information on patient compliance: a research model and an empirical study J Med Internet Res 2015 17 6 e143 10.2196/jmir.4333 26068214 v17i6e143 PMC4526934 Kutner M Greenberg E Jin Y Paulsen C The Health Literacy of America’s Adults: Results From the 2003 National Assessment of Adult Literacy 2006 09 2015-07-29 Washington, DC National Center for Education Statistics http://nces.ed.gov/pubs2006/2006483.pdf US Department of Health and Human Services, Office of Disease Prevention and Health Promotion National Action Plan to Improve Health Literacy 2010 2015-07-31 Washington, DC US Department of Health and Human Services, Office of Disease Prevention and Health Promotion http://www.health.gov/communication/hlactionplan/pdf/Health_Literacy_Action_Plan.pdf Nielsen-Bohlman L Panzer AM Kindig DA Health Literacy: A Prescription to End Confusion 2004 Washington, DC The National Academy Press Diviani N van den Putte B Giani S van Weert JC Low health literacy and evaluation of online health information: a systematic review of the literature J Med Internet Res 2015 05 17 5 e112 10.2196/jmir.4018 25953147 v17i5e112 PMC4468598 Wolf MS Bailey SC Improving prescription drug labeling N C Med J 2007 09 68 5 340 342 18183755 Ngoh LN Health literacy: a barrier to pharmacist-patient communication and medication adherence J Am Pharm Assoc (2003) 2009 09 49 5 e132 e146; quiz e147 10.1331/JAPhA.2009.07075 19748861 9716532212Q45P74 DuBay WH The Principles of Readability 2004 2015-08-07 Costa Mesa, CA Impact Information http://almacenplantillasweb.es/wp-content/uploads/2009/11/The-Principles-of-Readability.pdf Gromoske AN Oldridge N Brondino MJ Readability of 3 heart disease health-related quality of life questionnaires J Cardiopulm Rehabil Prev 2011 07 31 4 245 248 10.1097/HCR.0b013e31821c13c6 21606846 Stossel LM Segar N Gliatto P Fallar R Karani R Readability of patient education materials available at the point of care J Gen Intern Med 2012 09 27 9 1165 70 10.1007/s11606-012-2046-0 22528620 PMC3514986 Misra P Agarwal N Kasabwala K Hansberry DR Setzen M Eloy JA Readability analysis of healthcare-oriented education resources from the American Academy of Facial Plastic and Reconstructive Surgery Laryngoscope 2013 01 123 1 90 96 10.1002/lary.23574 23023924 Edmunds MR Barry RJ Denniston AK Readability assessment of online ophthalmic patient information JAMA Ophthalmol 2013 12 131 12 1610 1616 10.1001/jamaophthalmol.2013.5521 24178035 1763220 Doak CC Doak LG Root JH Teaching Patients with Low Literacy Skills. 2nd edition 1996 Philadelphia, PA JB Lippincott Company MedlinePlus 2013 2015-08-11 How to write easy-to-read health materials.http://www.nlm.nih.gov/medlineplus/etr.html European Parliament and Council of the European Union Directive 2001/83/EC of the European Parliament and of the Council of 6 November 2001 on the Community code relating to medicinal products for human use J Eur Comm 2001 11 311 67 128 Hamrosi KK Raynor DK Aslani P Enhancing provision of written medicine information in Australia: pharmacist, general practitioner and consumer perceptions of the barriers and facilitators BMC Health Serv Res 2014 04 14 183 10.1186/1472-6963-14-183 24754890 1472-6963-14-183 PMC4000453 Lam WY Fresco P Medication adherence measures: an overview Biomed Res Int 2015 2015 217047 10.1155/2015/217047 26539470 PMC4619779 Gustafsson J Kälvemark S Nilsson G Nilsson JL Patient information leaflets--patients' comprehension of information about interactions and contraindications Pharm World Sci 2005 02 27 1 35 40 15861933 Fuchs J Hippius M Schaefer M A survey of package inserts use by patients Hosp J Pharm Eur 2005 07 21 29 31 European Parliament and Council of the European Union Directive 2004/27/EC of the European Parliament and of the Council of 31 March 2004 amending Directive 2001/83/EC on the Community code relating to medicinal products for human use. J Eur Comm 2004 04 30 136 34 57 European Commission. Enterprise and Industry Directorate General Guideline on the Readability of the Labelling and Package Leaflet of Medicinal Products for Human Use 2009 01 12 2015-08-25 http://ec.europa.eu/health/files/eudralex/vol-2/c/2009_01_12_readability_guideline_final_en.pdf European Medicines Agency. Authorisation of medicines 2015-11-11 http://www.ema.europa.eu/ema/index.jsp?curl=pages/about_us/general/general_content_000109.jsp Strohl WR Knight DM Discovery and development of biopharmaceuticals: current issues Curr Opin Biotechnol 2009 12 20 6 668 672 10.1016/j.copbio.2009.10.012 19896824 S0958-1669(09)00140-2 Walsh G Biopharmaceutical benchmarks 2014 Nat Biotechnol 2014 10 32 10 992 1000 10.1038/nbt.3040 25299917 nbt.3040 European Medicines Agency 2015-07-29 Send a question to the European Medicines Agency.http://www.ema.europa.eu/ema/index.jsp?curl=pages/about_us/landing/ask_ema_landing_page.jsp&mid= European Medicines Agency European public assessment reports.http://www.ema.europa.eu/ema/index.jsp?curl=pages/medicines/landing/epar_search.jsp&mid=WC0b01ac058001d124 Piñero-López MA Modamio P Lastra CF Mariño EL Readability assessment of package inserts of biological medicinal products from the European medicines agency website Drug Saf 2014 07 37 7 543 554 10.1007/s40264-014-0188-1 24942755 European Medicines Agency 2015-07-29 http://www.ema.europa.eu/ema/ Walsh G Pharmaceutical biotechnology products approved within the European Union Eur J Pharm Biopharm 2003 01 55 1 3 10 12551698 S0939641102001650 European Medicines Agency 2015-07-29 Product-information templates.http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/document_listing/document_listing_000134.jsp&mid=WC0b01ac0580022c59 McLaughlin GH SMOG grading - a new readability formula J Read 1969 05 12 639 646 Kincaid JP Fishburne RP Rogers RL Chissom BS Research Branch Report 1975 02 2015-11-09 Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for navy enlisted personnel.http://www.dtic.mil/dtic/tr/fulltext/u2/a006655.pdf Szigriszt Pazos F Universidad Complutense de Madrid 1993 2015-07-29 Sistemas predictivos de legibilidad del mensaje escrito: fórmula de perspicuidad (PhD thesis).http://biblioteca.ucm.es/tesis/19911996/S/3/S3019601.pdf Wang LW Miller MJ Schmitt MR Wen FK Assessing readability formula differences with written health information materials: application, results, and recommendations Res Social Adm Pharm 2013 09 9 5 503 516 10.1016/j.sapharm.2012.05.009 22835706 S1551-7411(12)00077-0 Wilson M Readability and patient education materials used for low-income populations Clin Nurse Spec 2009 01 23 1 33 40; quiz 41 10.1097/01.NUR.0000343079.50214.31 19098513 00002800-200901000-00010 Fuchs J Werner S Scheunpflug C Götze EA Elstermann K Scheffel K Domnowski A Peter G Hertzsch C Hippius M Excessive medical information increase in package inserts Int J Clin Pharmacol Ther 2010 12 48 12 781 790 21084033 8154 Burgers C Beukeboom CJ Sparks L Diepeveen V How (not) to inform patients about drug use: use and effects of negations in Dutch patient information leaflets Pharmacoepidemiol Drug Saf 2015 02 24 2 137 143 10.1002/pds.3679 25044310 Liu F Abdul-Hussain S Mahboob S Rai V Kostrzewski A How useful are medication patient information leaflets to older adults? A content, readability and layout analysis Int J Clin Pharm 2014 08 36 4 827 834 10.1007/s11096-014-9973-2 24986267 Protheroe J Estacio EV Saidy-Khan S Patient information materials in general practices and promotion of health literacy: an observational study of their effectiveness Br J Gen Pract 2015 03 65 632 e192 e197 10.3399/bjgp15X684013 25733441 65/632/e192 PMC4337308 Tong V Raynor DK Blalock SJ Aslani P Consumer interpretation of ramipril and clopidogrel medication risk information - implications for risk communication strategies Patient Prefer Adherence 2015 07 9 983 988 10.2147/PPA.S86414 26185427 ppa-9-983 PMC4501347 Herber OR Gies V Schwappach D Thürmann P Wilm S Patient information leaflets: informing or frightening? A focus group study exploring patients' emotional reactions and subsequent behavior towards package leaflets of commonly prescribed medications in family practices BMC Fam Pract 2014 10 15 163 10.1186/1471-2296-15-163 25277783 1471-2296-15-163 PMC4287479 Pander Maat H Lentz L Improving the usability of patient information leaflets Patient Educ Couns 2010 07 80 1 113 119 10.1016/j.pec.2009.09.030 19854022 S0738-3991(09)00446-7 Fuchs J Hippius M Inappropriate dosage instructions in package inserts Patient Educ Couns 2007 07 67 1-2 157 168 10.1016/j.pec.2007.03.009 17451909 S0738-3991(07)00103-6 Krass I Svarstad BL Bultman D Using alternative methodologies for evaluating patient medication leaflets Patient Educ Couns 2002 05 47 1 29 35 12023098 S0738399101001719 Smith SG Wolf MS Obichere A Raine R Wardle J von Wagner C The development and testing of a brief ('gist-based') supplementary colorectal cancer screening information leaflet Patient Educ Couns 2013 12 93 3 619 625 10.1016/j.pec.2013.08.013 24007765 S0738-3991(13)00320-0 PMC3863947 Kish-Doto J Scales M Eguino-Medina P Fitzgerald T Tzeng JP McCormack LA O'Donoghue A Oguntimein O West SL Preferences for patient medication information: what do patients want? J Health Commun 2014 19 Suppl 2 77 88 10.1080/10810730.2014.946114 25315585 Shoemaker SJ Wolf MS Brach C Agency for Healthcare Research and Quality 2013 2015-07-29 The Patient Education Materials Assessment Tool (PEMAT) and User’s Guide.http://www.ahrq.gov/professionals/prevention-chronic-care/improve/self-mgmt/pemat/