We constructed a corpus of tweets relevant to HPV following the approach described in a study by Chen et al [29]. Figure 1 shows the flowchart of this data collection process. We first collected tweets that contained any of the 2 terms related to HPV (namely, “HPV” and “papillomavirus”) using the Twitter streaming API from November 11, 2014 to August 8, 2016, and further filtered out vaccine-irrelevant tweets using a statistical classifier [40]. This support vector machine classifier aims to distinguish vaccine-relevant tweets from vaccine-irrelevant tweets. It was trained on 1899 manually annotated tweets and achieved good performance (precision=0.96; recall=0.91; F₁=0.93) [40]. The time frame reflects image tweets collected from a previous vaccine images tweet analysis [29]. We then downloaded images for original image tweets, excluding retweets that had duplicate images.

We next used Face++ [33], a Web-based face recognition tool, to automatically categorize images containing faces and facial properties (eg, gender, age, and race). We selected Face++ because it has a high reported accuracy in the literature and supports race identification. We found that 25.8% of images had at least one face.

In addition, we obtained the locations of these image tweets using the CARMEN geolocation tool [41]. CARMEN infers the location of a tweet from the user’s profile and geotags in the metadata of the tweet. CARMEN has previously resolved location for 44.45% of tweets and correctly labeled the tweet location to within 250 miles of its true origin 75.27% of the time [41]. For our HPV image tweets, 32.5% of tweets were from the United States, and the location of 48.7% tweets was unknown. Our final dataset contained 456 HPV US-based image tweets containing at least one face.

We obtained 3 face attributes, gender (female or male), age (an integer), and perceived race (white, black, Asian, or ambiguous, which are the only 4 categories provided by Face++), of HPV images from Face++. Face++ has been widely used for face detection, recognition, and face attribute identification in social images like Twitter [35-38] and on major search engine images [34]. As reported previously, Face++ achieved a true positive rate of 85% in face detection with a false positive rate of 0.1 [39], 88% accuracy in gender recognition, and 79% accuracy in the race recognition [42].

Exact age estimation is a difficult task for both machines and humans. Face++ has a mean absolute error of 11.0 for exact age estimation [43] but a much higher accuracy (>93%) when it groups ages into categories (<18 years, 18-35 years, and >35 years) [44]. To allow the estimated age to be more robust, we organized age into 5 groups, namely, infant (0-2 years), child (3-8 years), youth (9-26 years, the recommended age to take the HPV vaccination by CDC), adult (27-64 years), and senior (>64 years).

To further validate the performance of Face++, we manually annotated the perceived gender, race, and age group attributes of all visible faces in each HPV image. The 2 annotators first worked independently and then resolved discrepancies by consensus. This process also helped us to gain insights from the HPV images as well as challenges of classifying the race, which is further detailed in the Discussion section.

We performed a mixed-methods analysis for the demographics reflected in the 3 face attributes. Based on the gender of faces, we first categorized images into 3 groups as follows: images that only have female faces, those that only have male faces, and those that have faces of both genders. We then compared the race and age distribution across these 3 groups. Categories chosen for the race were consistent with the current CDC standards for data collection and included Asian or Pacific Islander, black, and white individuals [45]. Owing to the technical limitations of Face++ of identifying the race and ethnicity, our racial classification system was streamlined (by dropping the racial classification for American Indian and Alaskan Native and the question of Hispanic ethnicity [46]) and embraced ambiguity (by adding a category for “ambiguous”). We acknowledge that the race and gender are nuanced social constructs that are particularly challenging to classify (see Discussion section). In this study, the race and gender, as perceived by researchers, were used to highlight differences in Twitter images. Racial perception is an appropriate measure as we seek to understand how health communications will be perceived. Notably, the complexities of reifying the race and gender are beyond the scope of this study.

Our analysis focused on images containing youths (recommended HPV vaccine recipients) or adults (eg, parents or health care providers). In addition, we examined the association between facial attributes and the source of the image tweet; for example, do government users or users with a health focus tend to post faces that reflect the actual burden of HPV diseases? To answer such questions, we manually examined (first annotated by one author of this paper, and then checked by 2 additional authors) all the source profiles of our HPV dataset (298 unique sources in total). We categorized these sources into 3 types, namely, government (eg, CDC and local health departments), nongovernment organizations (including health-related organizations), and individuals. We further stratified sources into health-related (eg, health care provider) or news-related categories. We then compared facial attributes across the resulting source categories.

Manual annotation identified more faces (1219) than Face++ (999). The discrepancy is primarily caused by images with multiple small, blurry, or nonfront-facing faces. Such faces are still distinguishable by humans but are rather difficult for automatic tools like Face++ to recognize. Because the subject of an image is usually the clearest and largest face, we believe Face++ has detected most important faces in our dataset (see Multimedia Appendix 1).

Table 1 details the distribution of 456 HPV images by the face count. Most images—55.6% according to manual annotation—contained multiple faces. Owing to the lack of face positions in manual annotation, it is difficult to align the annotated faces by human annotators and Face++ when multiple faces are presented in an image. Therefore, we limited the direct comparison of 2 annotations to images with a single face. Face++ achieved accuracy values of 84.7%, 76.4%, and 85.1% with respect to gender, race, and age, respectively, when compared with manual annotation, ignoring images with ambiguous labels. To gain an in-depth understanding of the performance of Face++, the confusion matrix is presented in Figure 2.

For gender, Face++ mistakenly labeled 27 female faces as male when considering manual annotation as ground truth. Of these 27, 12 (44.4%) were actually black women. For the race, Face++ identified 100% of black faces but had difficulty in differentiating white and Asian faces. Our human annotators acknowledged the same difficulty, leading them to label the race of 27 faces as ambiguous. Regarding age, we identified youth and adult age categories to have the greatest discrepancy between Face++ and manual annotation. From manual annotation, we acknowledge the challenge in distinguishing between age cutoffs (eg, 26 vs 27 years old). As such, the actual performance of Face++ on age has an accuracy of >85.1%. Overall, Face++ is reliable in detecting important faces and recognizing the facial properties (age, gender, and race) for our HPV images.

Considering manual annotation is more accurate than Face++, we only detail the results of manual annotation in the following tables. Our results from manual annotation and adoption of Face++ were broadly consistent, highlighting the potential for the automated face analysis utilization in public health research (see Multimedia Appendix 1). We first examined the gender of faces. At the broader image level in our dataset, 53.6% of images had only female faces, 17% had only male faces, 27.9% had both genders, and the remaining 1.6% had ambiguous gender or did not have a face (we excluded these 1.6% images in the following analysis). At the individual face level in our dataset (Table 2), 71.70% (874/1219) of faces were female, 27.89% (340/1219) were male, and the rest 0.41% (5/1219) had ambiguous gender (eg, infants’ faces).

Tables 3 and 4 show the number (percentage) of faces by the age and race, respectively. Overall, youth (616/1219, 50.53%) and adults (578/1219, 47.41%) were the 2 primary age groups, but the detailed distributions varied in each gender group. For instance, within the youth category, the majority of images included faces of both genders (365/616, 59.3%) and female faces (214/616, 34.3%), whereas in the adult category, the largest portion was female images (285/578, 49.3%), followed by both genders (219/578, 37.9%).

Looking at the race (Table 4), the majority of faces (795/1219, 65.22%) were white, followed by black (140/1219, 11.48%) and Asian (71/1219, 5.82%), whereas 17.47% (213/1219) of faces were racially ambiguous. For the overall gender distribution, female faces were predominantly represented across races (331/795, 42.39% white; 71/140, 50.71% black; and 34/71, 47.89% Asian) compared with male faces (7/140, 5.0% black; 75/795, 9.43% white; and 8/71, 11.27% Asian). In addition, we observed that over half (121/213, 56.80%) of faces with ambiguous races appeared in images with both genders.

Finally, we focused on youth and adult age faces and examined the gender and racial breakdown for the 2 age groups (detailed in Table 5). In addition to the consistently higher proportion of white faces across all gender and age categories, we observed a higher percentage (171/219, 78.1%) of white adult faces. In images with black faces, a black adult was rare in images with only male faces (1/74, 1.4%) or those with both genders (11/219, 5.0%). For Asians, we did not observe any Asian youth in images only with male faces.

Figure 3 shows the source categorization results. The authors manually annotated sources into 3 broad categories (ie, organizations, government, and individual). Source categories were further annotated by 3 specific topics (ie, news, health, and other). The majority (161/298, 54.0%) of sources were organizations, followed by individuals (99/298, 33.2%) and government (38/298, 12.8%). Regarding the topic, 13.4% (40/298) of sources were news related (eg, Huffington Post Blog and Business Insider) and 34.6% (103/298) were health related. We found that government sources were primarily health related (34/38, 90%) and organizations were more health related (64/161, 39.8%) than news related (37/161, 23.0%), whereas individual sources did not have a strong health or news focus (91/99, 92% of users had other topics).

Multimedia Appendix 2 shows the breakdown of demographic results by the source. Organization sources used more images with white faces. For instance, 96.6% (28/38) of female youth and 100% (12/12) of adult male images posted by news organization sources were white. These findings were significant, especially given the influence of media in promoting vaccination to reduce disparities (see Discussion).

Governmental health sources primarily posted images that had both genders, whereas news organization sources preferred female faces. Health-related organizations utilize many youth faces in images with both genders, and individual sources had an equal likelihood of posting female or mixed gender images. For the race, governmental health agencies tended to post a large proportion of black female faces in both youth (6/24, 25%) and adult age groups (4/20, 20%), more frequently than other sources (Table 5). Furthermore, governmental health sources posted a high proportion of youth faces of ambiguous race (5/24, 21% female; 3/15, 20% male; and 28/92, 30% both genders).

This study examined the content of HPV Twitter images. Despite reflecting demographics similar to US Census data, our results show a distinct difference between the demographics reflected in Twitter images and the actual burden of disease by the race and gender. Additional Census comparison figures are included in the Multimedia Appendix 1.

First, our analyses demonstrate that male faces are significantly underrepresented in HPV images (9.3%), particularly for youth, the recommended group for the HPV vaccination—34.3% girls versus 6.0% boys (Table 3); this is consistent with the observed lower uptake of the HPV vaccine among males. Currently, 37.5% of male youth were up to date with recommended HPV vaccination series compared with 49.5% for female youth [1], highlighting the need for further targeted HPV vaccine promotion toward male youth. Presenting the HPV vaccination for males as cancer and genital wart prevention has helped increased acceptance among males [47]. Continued efforts of increasing the awareness and representation of both genders in HPV communication are crucial in preventing HPV-associated cancer that affect men and women alike. Second, Twitter images had significantly fewer images of African American individuals even though African American individuals are more likely to be affected by HPV-associated cancer. Consistent with US Census data, our results demonstrate that white faces represented 65.22% (795/1219) of the images compared with 11.48% (140/1219) African American faces and 5.82% (71/1219) Asian faces [48]. However, of critical importance is that African American individuals are disproportionately affected by HPV-related diseases and are less likely to complete the HPV vaccination series compared with their white counterparts [4]. Furthermore, black adults are more likely to use Twitter than other racial groups, making Twitter a pertinent health communication platform to target parents of black youth [27]. Our findings suggest that despite known disparities, health communication efforts on Twitter have not sought to include the representation of African American individuals in targeted communications that could address racial disparities in the HPV vaccination [27].

Of note, images that included both females and males were usually represented by groups of youth (365/616, 59.3%), whereas for female-only and male-only faces, images were typically youth accompanied by adults who were more likely to be male (65.5%). From manual annotation, annotators noted that adults were typically represented by parents or health care providers. From the limited number of male faces in the sample (340/1219, 27.9%), male adults were twice as likely to be represented (74/578, 12.8%) compared with male youth (37/616, 6.0%), highlighting the insufficient communication efforts to target male youth for the HPV vaccination. Third, our results showed that the frequency of Twitter images posted from individual, organization, and government sources also reflected discrepancies in the minority representation. Individual sources shared more images with females (n=117) than males (n=37), suggesting the majority of users perceive HPV as an exclusively female issue. Organization users (health- or news-related) depict almost exclusively white faces.

The influence of Web-based news organizations is significant, especially given that 62% of American individuals access the internet for health information [49]. Although it is not their role to promote health behaviors, Web-based news sources are certainly influential in reporting and detecting emerging trends and outbreaks [22,23]. By depicting mostly white faces in Web-based HPV communication, these organizations might be reinforcing an inaccurate message about who is most at risk and failing to address groups that would also benefit from vaccination.

These findings are distinct from government images, which could be attributed to the differing incentives and roles of the media related to promoting vaccination compared with the government. In addition to having a lower presence tweeting black HPV images, government sources, which were typically local, state, and federal public health agencies, also had an uneven distribution of gender and racial representation within aforementioned images. Males represented in the images were overwhelmingly white—80% (12/15) of youth and 100% (1/1) of adults. Government HPV images with both genders included predominantly racially ambiguous youth (5/24, 21% female; 3/15, 20% male; and 28/92, 30% both gender), which may reflect an intentional choice to reach a more diverse audience, especially given the increasing number of multiracial youth in the Census [48]. However, the greater representation of racially ambiguous youth inadvertently fails to address racial and ethnic disparities in HPV health outcomes by underrepresenting those with the highest risk; this is especially critical given the role of governmental public health agencies in disseminating relevant health information.

In health communication, there is frequently an emphasis on the power of culturally sensitive communication, defined by Betsch et al as “the deliberate and evidence-informed adaptation of health communication to the recipients’ cultural background to increase knowledge and improve preparation for medical decision making and to enhance the persuasiveness of messages in health promotion” [50,51]. The aim is to create greater congruency between the health promotion messages and the recipient’s existing cultural context with an overall goal of increasing the effectiveness of messaging [52]; this can be accomplished through either targeting (aimed at the general cultural group) or tailoring (aimed at specific individuals within a cultural group). Visual representation of racial and ethnic minorities would represent a minimal effort at targeting high-risk groups and although it is not sufficient to guarantee the vaccine uptake, the exclusion of racial and ethnic minorities and males from images used in communication about the HPV vaccination is likely to perpetuate disparities in uptake and disease.

Our findings demonstrate discordance between HPV images on Twitter and those at the greatest risk of HPV-associated cancer. Twitter can be harnessed to disseminate HPV messages aimed at racial minorities who are more likely to be Twitter users [27]. Public health agencies would benefit from formative research with minority youth and their parents to improve Web-based health communication strategies to reduce the burden of HPV-associated cancer for all racial and ethnic groups.

There are limitations to this observational study. In this study, we also collected data for specific time points that yielded a relatively small sample of Twitter images, which speaks to the generalizability of findings. However, we found that characterizing the race, ethnicity, and age of faces was particularly challenging with both automated image analysis algorithms and manual annotation. Nearly half of the errors produced by Face++ were related to the misclassification of black female faces. Consistent with these results, Buolamwini and Gebru evaluated commercial gender classification algorithms and found that darker females have highest misclassification error rates (20.8%-34.7%) than darker males (0.7%-12.0%), lighter females (1.7%-7.1%), and lighter males (0.0%-0.8%) [53]. These findings highlight the limitations of automated image analysis for women of color, which have marked implications for real-world applications of this technology.

Self-identification is considered the gold standard for all race and ethnicity variables. In this instance, self-identification was not possible, and the race was ascribed by researchers engaged in manual annotation and through algorithms designed for this purpose. Unfortunately, the automated image analysis and, to a lesser extent, the manual annotation are dependent on the stereotypical phenotypic expressions of racialized features. The increasing population of mixed-race individuals, the complexity of the race and ethnicity with Latino populations, and indeed, the recognition that the race is only a social construct make an accurate determination of racial categories difficult. Furthermore, similar constraints made it difficult to consider any nonbinary expressions of gender. Thus, the potential for error of categorizing faces by specific age and race and gender needs to be considered in this study and future research.

This study provides insights into racial and gender differences in HPV images on Twitter. Findings can inform imagery-driven health communication strategies to increase the vaccine uptake to mitigate negative health outcomes, particularly within the context of social media. Culturally sensitive communication, which would include increased representation of minorities in images, may enhance the salience of HPV messaging to populations disproportionately affected by HPV-related health outcomes.