Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at, first published .
Development of the Emoji Faces Pain Scale and Its Validation on Mobile Devices in Adult Surgery Patients: Longitudinal Observational Study

Development of the Emoji Faces Pain Scale and Its Validation on Mobile Devices in Adult Surgery Patients: Longitudinal Observational Study

Development of the Emoji Faces Pain Scale and Its Validation on Mobile Devices in Adult Surgery Patients: Longitudinal Observational Study

Original Paper

1Anesthesiology Department, Shuguang Hospital, Shanghai University of Traditional Chinese Medicine, Shanghai, China

2Faculty of Dentistry, The University of Hong Kong, Hong Kong SAR, China (Hong Kong)

3Acupuncture and Anesthesia Research Institute, Shuguang Hospital, Shanghai University of Traditional Chinese Medicine, Shanghai, China

*these authors contributed equally

Corresponding Author:

Jiangang Song, PhD

Anesthesiology Department

Shuguang Hospital

Shanghai University of Traditional Chinese Medicine

528 Zhangheng Road, Pudong New Area

Shanghai, 20011


Phone: 86 021 20256301

Fax:86 021 20256301


Background: Measuring pain on digital devices using classic unidimensional pain scales such as the visual analog scale (VAS), numerical rating scale (NRS), and faces pain scale (FPS) has been proven to be reliable and valid. Emoji are pictographs designed in colorful form following the Unicode standard. It could be more beneficial to use emoji as faces of FPS on digital devices because emoji can easily fit on most devices and emoji are open-source so no approval would be needed before use. With a concise and user-friendly design, the emoji faces pain scale (Emoji-FPS) might be more generalizable to a wider population and more preferred by digital device users.

Objective: This study was designed to develop an Emoji-FPS as well as to evaluate its reliability, validity, and preference on mobile devices in adult patients who underwent surgery.

Methods: A modified Delphi technique with 2 rounds of web-based surveys was applied to obtain panelists’ consensus on the sequence of emoji that can best represent 6 levels of pain. The initial candidate sequences of emoji for the Delphi process were constructed referring to 2 well-validated FPSs (Wong-Baker FACES pain rating scale [Wong-Baker FACES] and faces pain scale-revised [FPS-R]). Then, a prospective cohort of patients scheduled to receive perianal surgery was recruited and asked to complete a web-based questionnaire on a mobile device at 5 time points (before surgery [T1], wake up after surgery [T2], 4 hours after surgery [T3], the second day after surgery [T4], and 15 minutes after T4 [T5]). The 4 well-validated pain scales (NRS, VAS, Wong-Baker FACES, and FPS-R) were used as reference scales.

Results: After 2 rounds of surveys on 40 Delphi panelists, an Emoji-FPS was finally determined to represent 6 pain levels (0, 2, 4, 6, 8, and 10) from “no hurt” to “hurts worst.” For validation, 300 patients were recruited and 299 were analyzed, the mean age of whom was 38.5 (SD 10.5) years, and 106 (35.5%) were women. For concurrent validity, the Emoji-FPS was highly correlated with 4 reference scales with Spearman correlation coefficient ρ ranging from 0.91 to 0.95. Excellent agreements were observed between 4 versions of Emoji-FPS (iOS, Android, Microsoft, and OpenMoji), with weighted κ coefficients ranging from 0.96 to 0.97. For discriminant validity, patients’ mean preoperative Emoji-FPS score (T1) was significantly higher than their postoperative Emoji-FPS score (T4) with a difference of 1.4 (95% CI 1.3-1.6; P<.001). For test-retest reliability, Emoji-FPS scores measured at T4 and T5 were highly correlated with a ρ of 0.91. The Emoji-FPS was mostly preferred, followed by the Wong-Baker FACES, FPS-R, NRS, and VAS.

Conclusions: The Emoji-FPS is reliable and valid compared with traditional pain scales in adult surgery patients.

J Med Internet Res 2023;25:e41189



Pain is a complex and subjective experience that is usually measured by self-reported scales [1]. Commonly used unidimensional scales are the visual analog scale (VAS), numerical rating scale (NRS), verbal rating scale, and face pain scale (FPS) [1]. Each of these scales has its own features and application scenarios [2-5].

With the steady development of telemedicine and mobile health in the past two decades, the medical community as well as pain research are adapting themselves to the increasingly digital world [6,7]. In pain monitoring, electronic data capturing methods have robust reliability and validity compared with traditional paper-based data collection methods [8-10]. Electronic pain measurement was more favorable in terms of compliance and patient satisfaction [8]. In the meantime, emerging pain-related smart device apps were developed to collect and track users’ pain information and even provide interventions [10,11].

Although means of collecting data have been greatly developed, fewer improvements were seen for the tools used for measuring pain. The upgrade from pencil-paper to finger-touchscreen interaction provides us with more possibilities including more accurate and timely data capture, and there is a need for an upgrade of the traditional pain scale so that a new pain scale can optimize the potential of smart devices without losing validity and reliability. Evidence revealed that electronically measured VAS, NRS, and FPS have the same performance as those measured by paper and pen [8]. However, VAS and NRS are less appealing to patients compared with FPS [4]. FPS, compared with other types of pain scales, is preferred by people from all age groups and is especially suitable for children and those with cognitive impairment [12,13]. Widely applied FPSs are the Wong-Baker FACES pain rating scale (Wong-Baker FACES) and the faces pain scale-revised (FPS-R), but both of them were developed years ago, and they were designed in monotonic color, which might not be as appealing as colorful faces on digital devices [14,15]. Additionally, traditional FPS might require approval before use and even potential royal fees for commercial applications. As images, it is difficult to guarantee that these traditional scales can fit properly on web pages or mobile apps without being distorted due to a variation in screen sizes and devices. One advantage of using faces to measure pain is that it is more self-explanatory than other types of scales. Using face images that are already popular with users might be more efficient to apply. As pictographs that are typically presented in a colorful form and used inline in text, emoji conform to the Unicode standard and are characterized by standardization and universality. Compared with images, it is flexible to adapt Emoji-FPS to mobile devices without additional image download as emoji have become a preloaded digital set of images that can work across platforms [6]. In addition, emoji are open-source, and they can be easily obtained without approval from anyone.

Digital device users are not unfamiliar with emoji faces such as , , and [6,16]. Until September 2021, a total of 3633 emoji had been created. Among all those emoji, at least 126 face-related emoji can be regarded as potential candidate faces to construct a pain scale. Face pain scales made of emoji faces might be more generalizable to more populations because emoji are not specific to a certain group of people. A certain sequence of emoji faces that can best measure levels of pain needs to be determined before they are applied.

An emoji faces pain scale (Emoji-FPS) should possess consistent reliability and validity compared with classical pain scales. Validation of its usefulness in adults without cognitive impairment first can provide evidence for its further use in other special populations. Patients who underwent perianal surgery are suitable for initial validation of the Emoji-FPS because those patients are usually characterized by mild to moderate preoperative pain and intense postoperative pain, which provides a good opportunity to test the scale’s discriminative validity [17]. In addition, they are also capable of using mobile devices before and after surgery themselves to answer questions.

This study aimed to (1) develop a unidimensional pain scale Emoji-FPS from the emoji faces using a modified Delphi technique; (2) test reliability and validity of the Emoji-FPS in patients scheduled to receive perianal surgery; and (3) evaluate patients’ reference over the VAS, NRS, Wong-Baker FACES, FPS-R, and Emoji-FPS.

Development of Emoji-FPS

The Modified Delphi Technique

A modified Delphi technique was used to obtain experts’ consensus on which emoji sequence can best represent different pain levels. Delphi is a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem [18,19]. The Delphi technique is an iterative process characterized by collecting controlled feedback from experts and requiring some degree of anonymity for the individual responses [20]. The expert panel evaluates and re-evaluates potential items until consensus is met after rounds of sequential surveys [18]. A “modified” Delphi technique was used in this study since the initial emoji sequences for the first round were determined based on literature review and expert interviews [21]. Instead of generating multiple items, only 1 item (1 sequence of emoji) was determined in this study. Reporting of the results of the Delphi survey followed the CHERRIES (Checklist for Reporting Results of Internet E-Surveys) [22].

Participants (Delphi Panel)

The expert panel consists of health workers with at least 1 year of working experience in the field of anesthesiology or pain medicine. Both clinicians and nurses were included. Panelists were mainly recruited from the anesthesiology department of Shuguang Hospital affiliated with Shanghai University of Traditional Chinese Medicine. The snowballing technique was applied when panelists’ colleagues from other institutions had interest in participating in this study. According to the rule of thumb, a sample size of 15-30 participants was generally recommended for a homogeneous survey (eg, overlapping specialties exist) [21]. To accommodate possible attrition of panelists from round 1 to round 2, a sample size of 40 was finally chosen.

Construction of Initial Emoji Sequences

Since it usually takes months before the latest version of emoji is adapted to major platforms, Emoji version 13.1 (Unicode Consortium) was used in this study [16]. As of August 2021, support for Emoji 13.1 is available on iOS 14.5 (Apple Inc) and various Google platforms. The search for potential emoji faces was confined to the smileys and emotion emoji category. A total of 91 emoji in the following types were selected as candidates: face-smiling, face-affection, face-tongue, face-hand, face-neutral-skeptical, face-sleepy, face-unwell, face-hat, face-glasses, and face-negative. After a literature review, it was found that both widely used FPS—Wong-Baker FACES and FPS-R—choose 6 faces as anchors. It is more practical to transform a scale with 6 levels to either 0-1-2-3-4-5 or 0-2-4-6-8-10; therefore, the 0-5 or the 0-10 metric can be applied. Therefore, our proposed Emoji-FPS also used the 6-level scale scheme.

Both Wong-Baker FACES and FPS-R are well-developed and validated scales over the years, so we aimed to find a sequence of emoji that also possess some properties (curve of eyebrow, shape of eye, curve of mouth, and tears) of these existing face scales so that this sequence of emoji could be more likely to be valid for measuring pain. After discussions of 3 experts (2 experienced in pain and anesthesiology and 1 experienced in public health), a total of 10 sequences were initially constructed for the first round of the survey.

Delphi Survey Procedure

The surveys were conducted on the internet from July 2021 to September 2021. Each Delphi survey questionnaire consisted of mandatory instructions for panelists. The panelists were instructed to rate each candidate emoji sequence on a Likert scale from 1 (unsuitable) to 2 (slightly suitable) to 3 (moderately suitable) to 4 (suitable) to 5 (very suitable) based on their feelings about to what extent these emoji sequences can represent the pain levels from “no hurt” to “hurts most.” All panelists’ ratings were equally weighted. Since the appearance of emoji may be different on different devices and platforms, candidate sequences of Emoji-FPS were demonstrated using emoji images from iOS, Android, Microsoft, and OpenMoji versions separately in both rounds of Delphi surveys (see Multimedia Appendix 1). In both rounds, the panelists were instructed to rate each emoji sequence 4 times, once with each version of emoji. OpenMoji was included since it is an open-source project and it is free to share and adapt without restrictive usage rights [23]. Responses of panelists were kept anonymous.

Round 1

The link to the web-based survey was officially distributed to 40 potential participants. The panelists were allowed to select emoji beyond those used in the initial 10 sequences if they considered none of the emoji to represent certain levels of pain, and a total of 91 emoji faces are provided in Multimedia Appendix 1. This round was designed to find possible emoji suitable for each level of pain. Based on the results of this round, emoji sequences may be reformed in the next round. The panelists were asked to complete the survey within 3 days, and messages were sent to remind those who had no response after 5 days.

Round 2

In this round, invitations of surveys were sent to those who responded in round 1. The survey was in the same form as the previous round. However, fewer emoji sequences were provided, and the number of sequences was subject to the variation in ratings from the previous round. After this round, the final sequence of emoji was determined.

Validation of Emoji-FPS


Patients scheduled to receive perianal surgery were identified and recruited from the anorectal surgery department of Shuguang Hospital affiliated to Shanghai University of Traditional Chinese Medicine. Inclusion criteria were 18-65 years old and preoperative NRS ≤3. Those with a preoperative mini-mental state examination score <26 or with perianal abscess were excluded.

Sample Size

To test consistency between types of Emoji-FPS, pairwise comparisons were made between groups. Assuming the Pearson correlation coefficient was 0.8 (0.6 under the null hypothesis), a sample size of 51 was required to achieve 0.8 power with a significance level of 0.05. Thus, a total of 255 individuals would be needed for the 5 groups. This sample size would also provide enough power for other tests in this study. Considering the dropping out of patients, a final sample size of 300 was determined.


In addition to the Emoji-FPS, a total of 4 well-validated pain scales (NRS, VAS, Wong-Baker FACES, and FPS-R) were selected as references. The 4 versions of Emoji-FPS (iOS, Android, Microsoft platforms, and OpenMoji) were provided (Table S5 in Multimedia Appendix 2).

Procedure and Data Collection

Participants were asked to complete the 5 types of pain scales (NRS, VAS, Wong-Baker FACES, FPS-R, and Emoji-FPS) at 5 time points: before surgery (T1), wake up after surgery (T2), 4 hours after surgery (T3), the second day after surgery (T4), and 15 minutes after T4 (T5). Participants’ demographics and surgery information were also collected. To attenuate the influence of the display sequence of the 5 pain scales on the rating result, a total of 5 groups (A, B, C, D, and E) with different sequences of pain scales were designed using a Latin square matrix to ensure that an equal number of participants would rate the 5 scales with a different display sequence (S1, S2, S3, S4, and S5) at each time (see Supplemental Methods section in Multimedia Appendix 2 for details). In addition, 4 versions of the Emoji-FPS were evenly allocated into 4 groups (A, B, C, and D) so that an equal number of participants would rate different versions of the Emoji-FPS at the same time and each participant would rate different versions of the Emoji-FPS at different time except for T4. Participants rated all 4 versions of the Emoji-FPS simultaneously at T4 to evaluate their agreement. After informed consent was obtained, participants were randomly allocated into 1 of the 5 groups using block randomization with a block size of 5. Each participant was provided with a mobile device (eg, iPad) to complete our web-based questionnaire during the study period. At T1, patients completed questionnaires under instruction. Patients can select faces best representing their feeling of pain by taping the radio button with the corresponding scores (0, 2, 4, 6, 8, and 10) displayed below them on the screen, as Wong-Baker FACES and FPS-R did. For NRS, an NRS scale was displayed on screen, and patients could select the corresponding number from a dropdown list. To ensure that different versions of emoji can display well, images of emoji were used. For the remaining follow-ups, patients were reminded of completing questionnaires themselves.

Ethics Approval

This study was approved by the Chinese Ethics Committee of Registering Clinical Trials (reference number ChiECRCT20210462). Informed consent was obtained from Delphi panelists and patients in the validation step. All study data were deidentified and encrypted when transferred or preserved. Compensation was not provided for participants.

Statistical Analysis

Descriptive analyses were performed. Spearman rank correlation coefficients (ρ) were calculated to measure the correlation between scales. A paired sample t test was used to compare patients’ Emoji-FPS results before and after surgery. The weighted κ coefficient was applied to measure pairwise consistency between versions of the Emoji-FPS. The significance level was .05. SPSS 27.0 (IBM Corp) and SAS 9.4 (SAS Institute Inc) were used for data analysis.

Development of Emoji-FPS (Delphi)

Of the 40 invited panel members, all consented and responded to round 1. A total of 36 panelists responded to round 2 with a response rate of 90% (see Multimedia Appendix 2 for a detailed description of the characteristics of panelists and survey results). After 2 rounds of surveys, out of 91 candidate emoji faces, an emoji sequence of , , , , , and was finally determined to represent 6 pain levels from “no hurt” to “hurts worst” (Table 1).

Table 1. Developed emoji faces pain scale.


DescriptionNo hurtHurts little bitHurts little moreHurts even moreHurts whole lotHurts worst
CLDRa short nameSlightly smiling faceNeutral faceSlightly frowning faceFrowning faceWeary faceLoudly crying face
Code pointsU+1F642U+1F610U+1F641U+2639U+1F629U+1F62D

aCLDR: The Common Locale Data Repository Project of the Unicode Consortium.

bOpenMoji is an open-source project and independent emoji system.

Validation of Emoji-FPS

Patient Characteristics

During November 2021 and January 2022, a total of 352 patients were assessed for eligibility, of whom 300 were randomly allocated to 1 of the 5 groups, and 1 was excluded due to a preoperative NRS score greater than 3. Finally, 299 patients were included in the data analysis (Figure S1 in Multimedia Appendix 2 for flowchart). Study participants had a mean age of 38.5 (SD 10.5) years, and 106 (35.5%) were women (see Table S6 in Multimedia Appendix 2 for details). A total of 68.6% (n=205) of patients had a college or above educational level. Overall, 201 (67.2%) patients received hemorrhoidectomy and 98 (32.8%) received anal fistula resection. The mean surgery duration was 25.9 (SD 11.1) minutes. The distribution of Emoji-FPS scores covered 0 to 10 over T1 to T5 (see Figure S2 in Multimedia Appendix 2).

Concurrent Validity

Generally, Emoji-FPS was highly correlated with 4 reference scales with ρ ranging from 0.91 to 0.95 (Table 2). The associations between the Emoji-FPS and 4 reference scales at each time point were satisfactory, with ρ values ranging from 0.89 to 0.96. Each version of the Emoji-FPS also revealed a sound correlation with 4 reference scales (range of ρ=0.67-0.98). The agreements between different types of Emoji-FPS were high (range of weighted κ coefficients=0.96-0.97; see Table S7 in Multimedia Appendix 2).

Table 2. Results of concurrent validity test.
Reference scale and time pointTypes of Emoji-FPSa, ρ






















aEmoji-FPS: emoji faces pain scale.

bNRS: numerical rating scale.

cVAS: visual analog scale.

dWB FACES: Wong-Baker FACES.

eFPS-R: faces pain scale-revised.

Discriminant Validity

For all participants, their postsurgery mean Emoji-FPS score was 1.4 points (95% CI 1.3-1.6; P<.001) higher than the presurgery results, indicating a sound discriminant validity (Table 3).

Table 3. Results of the discriminant validity test.
Time pointTypes of Emoji-FPSa

T1, mean (SD)1.4 (0.7)1.3 (0.5)1.5 (0.7)1.4 (0.6)1.4 (0.7)
T4, mean (SD)2.9 (1.2)2.5 (1.0)2.9 (1.3)2.6 (1.2)2.8 (1.2)
Difference (95% CI)1.6 (1.3-1.9)1.2 (0.9-1.1)1.4 (1.1-1.8)1.3 (1.0-1.6)1.4 (1.3-1.6)
P valueb<.001<.001<.001<.001<.001

aEmoji-FPS: emoji faces pain scale.

bCalculated using paired t test.


Good test-retest reliability was supported by the high correlation between the Emoji-FPS results at T4 and T5 (range of ρ=0.88-0.98; Table 4).

Table 4. Results of the test-retest reliability test.
Time pointTypes of Emoji-FPSa

T4, mean (SD)2.9 (1.2)2.5 (1.0)2.9 (1.3)2.6 (1.2)2.8 (1.2)
T5, mean (SD)2.9 (1.2)2.5 (1.1)2.9 (1.2)2.7 (1.2)2.7 (1.1)

aEmoji-FPS: emoji faces pain scale.

bSpearman correlation coefficient for results between T4 and T5.

Scale Preference

Emoji-FPS was mostly welcomed by participants (mean 2.0, SD 1.2), followed by Wong-Baker FACES (mean 2.8, SD 1.2), FPS-R (mean 3.2, SD 1.3), NRS (mean 3.2, SD 1.4), and VAS (mean 3.8, SD 1.4). See Figure 1 for a detailed distribution.

Figure 1. Distribution of patients’ rank of preference of 5 pain scales. Emoji-FPS: emoji faces pain scale; FPS-R: faces pain scale-revised; NRS: numerical rating scale; VAS: visual analog scale; WB FACES: Wong-Baker FACES.

Principal Findings

In this study, a 6-level Emoji-FPS (, , , , , and ) was developed. Satisfactory validity and reliability of the Emoji-FPS were confirmed in patients who underwent perianal surgery.

The potential use of emoji in medical research has been discussed previously, although its application is still limited [6,7]. Emoji were considered to be used as an indicator to evaluate patient-reported outcomes in breast cancer treatment due to its friendliness on social networking services and electronic devices [24]. Kiliç et al [25] used emoji as anchors to replace common numerical anchors in the Likert-type Psychological Well-Being Scale. They used , , and to represent a 3-point scale (disagree, neutral, and agree); , , , , and to represent a 5-point scale (from strongly disagree to strongly agree), and , , , , , , and to represent a 7-point scale (from strongly disagree to absolutely agree). However, as O’Reilly-Shah et al [7] commented, “two interrelated problems are the durability of an Emoji’s meaning over time and the potential variability in its meaning in different cultural and linguistic contexts.” For example, “face with tears of joy” () can be used to convey the feeling of “laughing so hard I’m crying,” and it is significantly different from its named intent. This is one of the reasons why we chose to use a modified Delphi technique to determine the Emoji-FPS. Our developed Emoji-FPS was reached upon collated opinions from a group of people to avoid the subjective choice of emoji. He et al [26] also developed an Emoji-FPS with a sequence of , , , , , and . A high level of agreement was observed between their Emoji-FPS and VAS after validation in 109 patients. Compared with their proposed Emoji-FPS, our proposed Emoji-FPS does not contain the smiling face and includes a painful face . Since smiles can be reminiscent of happiness, starting with a more neutral face might be more advantageous [27]. Before wide application, more validation studies are needed to reach a consensus.

It is not surprising that the Emoji-FPS was preferred by patients, followed by the Wong-Baker FACES, FPS-R, NRS, and VAS. Colorful and intuitive images could be more attractive to users than monotonic and abstract images, especially on mobile devices. In addition, emoji have fewer details than Wong-Baker FACES and FPS-R. While these details can make faces of different pain levels more distinguishable, excessive details might make users think that these faces no longer look like them, for example, if patients have a different gender, age, or race. Emoji are more concisely designed and are not limited to these specific characteristics, so they have the potential to be applied regardless of age, gender, and race.

Unlike typefaces and fonts of characters appearing exactly the same across platforms, emoji could have different appearances between devices and platforms, although the meanings they convey should be unified. As the Unicode Consortium states, “while the shape of the character can vary significantly, designers should maintain the same core’ shape, based on the shapes used mostly commonly in industry practice” [28]. In this study, excellent agreement was observed between different versions of the Emoji-FPS. It is flexible to adapt Emoji-FPS to mobile devices as emoji has become a preloaded digital set of images that can work across platforms [6]. Moreover, as users of digital devices have become more familiar with them, the Emoji-FPS can be applied across language and communication barriers (eg, patients with dysphasia or patients intubated and unable to communicate verbally). However, as the versions or variations of emoji emerge, the additional evaluation of the reliability and validity of other versions of emoji before application would be useful.


This study has limitations. Only health workers were invited to decide the Emoji-FPS from thousands of emoji, and opinions from patients were not considered. However, the relatively large panel size may accommodate this shortcoming. Our Emoji-FPS was only validated among adult surgery patients in a single center, and generalization of the Emoji-FPS to other populations, such as children, might be limited. Qualitative interviews with children or patients will be planned before further validation. Due to a limitation of the design of the web-based questionnaire, patients needed to select answers to the NRS from a dropdown list, so patients’ experience of using the NRS might be compromised. Their preference for NRS might change if a better interface is provided.


Our developed Emoji-FPS has been proven to be reliable and valid compared with traditional pain scales on mobile devices among perioperative patients. Further studies are needed to confirm the reliability and validity of the Emoji-FPS in other settings.


The authors acknowledge support from the National Natural Science Foundation of China (82204868, 81904283, and 81774108) and research project of Shanghai Municipal Health Commission (201940062) fundings.

Data Availability

Research data are available upon reasonable request.

Authors' Contributions

LL and SW drafted the manuscript and contributed greatly to study conception and design. JW participated in the study design. CW and WZ participated in data collection and collation. JS and LY are co-corresponding authors. They contributed greatly to study conception and design.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Questionnaires of the Delphi survey.

DOCX File , 1995 KB

Multimedia Appendix 2

Supplemental results and methods.

DOCX File , 1050 KB

  1. Younger J, McCue R, Mackey S. Pain outcomes: a brief review of instruments and techniques. Curr Pain Headache Rep 2009;13(1):39-43 [FREE Full text] [CrossRef] [Medline]
  2. Gries K, Berry P, Harrington M, Crescioni M, Patel M, Rudell K, et al. Literature review to assemble the evidence for response scales used in patient-reported outcome measures. J Patient Rep Outcomes 2018;2:41 [FREE Full text] [CrossRef] [Medline]
  3. Karcioglu O, Topacoglu H, Dikme O, Dikme O. A systematic review of the pain scales in adults: which to use? Am J Emerg Med 2018;36(4):707-714. [CrossRef] [Medline]
  4. Li L, Liu X, Herr K. Postoperative pain intensity assessment: a comparison of four scales in Chinese adults. Pain Med 2007;8(3):223-234. [CrossRef] [Medline]
  5. Safikhani S, Gries KS, Trudeau JJ, Reasner D, Rüdell K, Coons SJ, et al. Response scale selection in adult pain measures: results from a literature review. J Patient Rep Outcomes 2018;2(1):40 [FREE Full text] [CrossRef] [Medline]
  6. Lai D, Lee J, He S. Emoji for the medical community—challenges and opportunities. JAMA 2021;326(9):795-796. [CrossRef] [Medline]
  7. O’Reilly-Shah VN, Lynde GC, Jabaley CS. Is it time to start using the emoji in biomedical literature? BMJ 2018 Dec 12;363:k5033. [CrossRef]
  8. Jibb LA, Khan JS, Seth P, Lalloo C, Mulrooney L, Nicholson K, et al. Electronic data capture versus conventional data collection methods in clinical pain studies: systematic review and meta-analysis. J Med Internet Res 2020;22(6):e16480 [FREE Full text] [CrossRef] [Medline]
  9. Turnbull A, Sculley D, Escalona-Marfil C, Riu-Gispert L, Ruiz-Moreno J, Gironès X, et al. Comparison of a mobile health electronic visual analog scale app with a traditional paper visual analog scale for pain evaluation: cross-sectional observational study. J Med Internet Res 2020;22(9):e18284 [FREE Full text] [CrossRef] [Medline]
  10. Sánchez-Rodríguez E, de la Vega R, Castarlenas E, Roset R, Miró J. An app for the assessment of pain intensity: validity properties and agreement of pain reports when used with young people. Pain Med 2015;16(10):1982-1992. [CrossRef] [Medline]
  11. Sundararaman LV, Edwards RR, Ross EL, Jamison RN. Integration of mobile health technology in the treatment of chronic pain: a critical review. Reg Anesth Pain Med 2017;42(4):488-498. [CrossRef] [Medline]
  12. Lor M, Kim KS, Brown RL, Rabago D, Backonja M. Comparison of four pain scales among Hmong patients with limited English proficiency. Pain Manag Nurs 2021;22(2):205-213. [CrossRef] [Medline]
  13. Wong DL, Baker CM. Smiling faces as anchor for pain intensity scales. Pain 2001;89(2-3):295-300. [CrossRef] [Medline]
  14. Wong DL, Baker CM. Pain in children: comparison of assessment scales. Pediatr Nurs 1988;14(1):9-17. [Medline]
  15. Hicks CL, von Baeyer CL, Spafford PA, van Korlaar I, Goodenough B. The Faces Pain Scale-revised: toward a common metric in pediatric pain measurement. Pain 2001;93(2):173-183. [CrossRef] [Medline]
  16. Emoji version 13.1. Emojipedia. 2021.   URL: [accessed 2023-03-22]
  17. Joshi GP, Neugebauer EA, PROSPECT Collaboration. Evidence-based management of pain after haemorrhoidectomy surgery. Br J Surg 2010;97(8):1155-1168. [CrossRef] [Medline]
  18. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000;32(4):1008-1015. [Medline]
  19. Linstone HA, Turoff M. The Delphi method: Techniques and Applications. Reading: Addison-Wesley Publishing Company; 1975.
  20. Mullen PM. Delphi: myths and reality. J Health Organ Manag 2003;17(1):37-52. [CrossRef] [Medline]
  21. de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach 2005;27(7):639-643. [CrossRef] [Medline]
  22. Eysenbach G. Improving the quality of web surveys: the checklist for reporting results of Internet e-surveys(CHERRIES). J Med Internet Res 2004;6(3):e34 [FREE Full text] [CrossRef] [Medline]
  23. OpenMoji 13.1—open source emojis for designers, developers and everyone else!. OpenMoji. 2021.   URL: [accessed 2023-03-22]
  24. Suzuki Y, Taira N, Kajiwara Y, Kochi M, Iwamoto T, Ikeda H, et al. Abstract OT3-19-01: validation study of the emoji scale in evaluating patient reported outcomes for breast cancer patients. Cancer Res 2020;80(4 Suppl):3-19. [CrossRef]
  25. Kiliç AF, Uysal I, Kalkan B. An alternative to Likert scale: emoji. J Meas Eval Educ 2021;12(2):182-191. [CrossRef]
  26. He S, Renne A, Argandykov D, Convissar D, Lee J. Comparison of an emoji-based visual analog scale with a numeric rating scale for pain assessment. JAMA 2022;328(2):208-209 [FREE Full text] [CrossRef] [Medline]
  27. Moisset X, Attal N, Ciampi de Andrade D. An emoji-based visual analog scale compared with a numeric rating scale for pain assessment. JAMA 2022;328(19):1980. [CrossRef] [Medline]
  28. Davis M, Edberg P. Unicode® technical report #51. Unicode emoji. Unicode. 2015.   URL: [accessed 2023-03-22]

CHERRIES: Checklist for Reporting Results of Internet E-Surveys
Emoji-FPS: emoji faces pain scale
FPS: face pain scale
FPS-R: faces pain scale-revised
NRS: numerical rating scale
VAS: visual analog scale
Wong-Baker FACES: Wong-Baker FACES pain rating scale

Edited by A Mavragani; submitted 19.07.22; peer-reviewed by D Dunsmuir, B Davies; comments to author 19.12.22; revised version received 05.02.23; accepted 15.03.23; published 17.04.23


©Lili Li, Sicheng Wu, Jian Wang, Chunchun Wang, Weixin Zuo, Liping Yu, Jiangang Song. Originally published in the Journal of Medical Internet Research (, 17.04.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.