Best-worst scaling (BWS) is a method used to measure individual preferences, and it has been categorized into three types: cases 1, 2, and 3. In case 1, researchers explore the preferences of a population of interest regarding each item of a particular list of objects. Individuals under observation are presented with a set of objects and asked to choose separately the best and the worst object. In case 2, researchers focus on the importance ranking of items presented in a single profile structure developed by combining attributes at various levels. Case 3 presents multiple profiles to the individuals and asks them to choose the best and worst one from each choice set. In this study, we used the BWS case 1 method.

The development of the BWS questionnaire was a robust and meticulous process that involved several steps. The first step involved drafting 8 possible attributes based on 3 literature reviews [1,6,7]. This developed the foundation of this study and helped the team form a preliminary understanding of the current landscape of mobile health solutions in the pandemic. In the second step, the team further reviewed existing evidence on barriers and facilitators to mobile app use in reducing COVID-19 risks. This helped to identify additional attributes that could be added to the attribute pool. The team sought to ensure that the attributes were comprehensive and covered possible barriers and facilitators to mobile app use [2-5,8-16]. After step 2, a list of 12 attributes was identified. In the third step, the team randomly recruited 10 medical university students and conducted semistructured one-on-one interviews to discuss the feasibility of the attributes. The aim was to identify any potential issues with the attributes and to revise them as necessary. The team combined and revised 6 of the attributes and additionally created a set of 11 attributes. The inclusion of young people’s perspective was important, as they are a key demographic group that could benefit from mobile health solutions in the pandemic. In the fourth step, an expert group consisting of public health researchers, health policymakers, and a mobile app developer reviewed the list of attributes. The panel revised some wording, resulting in a draft of 10 attributes. Their input helped to ensure that the attributes were clear and concise. In the fifth step, a convenience sample of 10 members of the general public was recruited to provide feedback on the 10 attributes from a user perspective. The aim was to ensure that the attributes were relevant to the general public and that they could be easily understood. Although no further revisions were made, the team took their feedback into consideration. Last, the team reviewed all previous information and evidence and included one additional attribute in the list. Finally, a set of 11 attributes (ie, functions) was confirmed to be used to develop the BWS questionnaire. A description of each attribute is provided in Multimedia Appendix 1.

To ensure the effectiveness of the research design, a balanced incomplete block design (BIBD) was used to construct the choice task. BIBD is the optimal way to ensure that each attribute level is equally replicated and appears within a block with every other attribute level an equal number of times [17]. This method guarantees that there is an equal probability of selecting each attribute in the questionnaire. The experiment design is displayed in Table 1. In this study, 11 choice task profiles, each with 5 attributes, were created. Participants were asked to select 2 attributes, with one being the most important and the other being the least important function that the app should provide. In this study, each participant had to make a total of 22 choices (11 best and 11 worst).

Data used in this study were obtained from a web-based cross-sectional survey in January 2023. All participants were recruited from the web-based panel of Wenjuanxing, a survey company. The inclusion criteria were as follows: (1) age ≥18 years; (2) ability to understand and read Chinese; and (3) agreement to provide informed consent. All eligible participants were invited to join to complete a web-based questionnaire. There is no gold standard for determining the sample size for a BWS case 1 study. A recent systematic review of health-related studies found that the sample size for BWS case 1 studies ranged between 15 and 803, with a median of 175 [18]. In addition, according to Louviere et al [19], a sample of 510 is sufficient to meet the required assumption (P<.05) based on Thompson’s formula for estimating the sample size for multinomial proportions data. However, a larger sample provides greater power. Therefore, a convenience sample size of 2000 was chosen for this study. The first page of the questionnaire contained an informed consent section that participants needed to read and agree to before proceeding. The questionnaire consisted of 2 sections: the first was the BWS questionnaire and the second was a series of questions asking for respondents’ personal information, such as gender, age, and COVID-19–related information. We ensured data quality by working with the survey company on a series of indicators, including completion time, IP address, and an artificial intelligence formula developed by the company to analyze the participants’ response patterns to avoid random error. The survey was anonymous and did not collect any personally identifiable information. All the BWS questions were predefined as compulsory to ensure no missing data needed to be imputed in our analysis.

A count analysis was performed to examine the frequency of choice for each BWS attribute. The BWS score was calculated as the difference between the number of times an attribute was selected as the most important and the number of times that attribute was selected as the least important for each of the 11 attributes. A positive BWS score indicated that the attribute was chosen more often as the most important, while a negative score indicated the opposite. Another outcome measure was mean BWS score, which was calculated by dividing the BWS score by the number of participants who responded to each attribute.

The data were further analyzed using 2 regression methods: a conditional logit model (CLM) and a mixed logit (MXL) model. Both methods incorporate the logit procedure, where each attribute is dual-coded. The “best” variable is assigned a value of 1 if the attribute is chosen as the most important, and 0 otherwise. Meanwhile, the “worst” variable is assigned a value of –1 if the attribute is chosen as the least important, and 0 otherwise. These variables are used to estimate propensity scores for the presence of an attribute in a specific combination of attributes.

Unlike CLM estimation, MXL accommodates unobserved taste heterogeneity by specifying preference parameters as random variables with means and SDs rather than fixed parameters. It can derive an SD to indicate unexplained variation around each attribute’s mean. An SD value significantly departing from zero indicates significant heterogeneity. The Akaike information criterion (AIC) and log likelihood (LL) values were used to assess model fit. Smaller AIC values indicate a better fit, while higher LL values indicate a better model. Stratified analysis was used to detect the heterogeneity of mean BWS score across different subgroups (divided by COVID-19–related and socioeconomic characteristics) with the robust Yuen method for trimmed means (bootstrap=1000). R (R Core Team) was used to design the BWS and perform all the statistical analyses.

The study protocol and informed consent procedure were approved by the institutional review board of the Hong Kong Polytechnic University (HSEARS20221029001). All study data were anonymous. Respondents were provided with a cash incentive (RMB 2; US $0.30) by the survey company to encourage their participation.

A total of 4383 individuals were invited to complete the web-based survey, of whom 2606 agreed to participate. Among them, 66 (2.5%) declined to consent and were screened out, and 300 (11.5%) who began the survey were excluded from analysis because they did not complete it. A further 87 (3.3%) completed the survey but were excluded because either their completion time was shorter than 120 seconds or they did not pass the platform’s AI concentration test. The responses of the remaining 2153 participants were eligible for analysis. Feedback on the BWS survey is shown in Multimedia Appendix 2.

Table 2 presents the characteristics of the study participants. Nearly 55% (1192/2153) were female, and the mean age was 31.4 years. Most participants (1765/2153, 81.9%) had completed tertiary or higher education, and approximately 70% (1523/2153) were urban residents. Additionally, 63.5% (1368/2153) of participants had been infected with COVID-19 in the past year. Almost 90% (1921/2153) reported experiencing COVID-19–related depression to some extent, and 68.6% (1478/2153) usually used smartphone apps to manage their health.

Table 3 presents the distribution of BWS scores for each attribute. The BWS score range for each attribute was between –5 and 5, with the highest proportion of responses centered around 0.

Count analysis demonstrated that participants rated “surveillance and monitoring of infected cases” as the most important app function (mean BWS score 0.98, 95% CI 0.88-1.07). “Mental health therapy” was selected as the least important function, with a mean BWS score of −1.12 (95% CI −1.2 to −1.04; Table 4).

Both the CLM and MXL models exhibited similar estimations (Table 4). Each attribute’s importance was measured relative to “mental health therapy,” which was consistently rated as the least important attribute (ie, this was used as the reference). Both models showed that “surveillance and monitoring of infected cases” (odds ratio [OR] 2.73, 95% CI 2.620-2.850 for CLM, mean coefficient 1.52 for MXL) was the function most favored by the participants, followed by “quick self-screening” (OR 2.47, 95% CI 2.620-2.850 for CLM, mean coefficient 1.39 for MXL). The remaining attributes were of intermediate importance, except for “platform for experience sharing,” which was close to the reference attribute (“mental health therapy”) in the CLM. Additionally, the MXL model provided the SD estimation for each attribute. The significant results indicated that there was substantial heterogeneity regarding the relative importance for all items.

Stratified analyses showed significant differences across subgroups (Table 5). Compared to individuals who had never been infected with COVID-19, those who had been infected indicated that “mobile-based consultation for treatment” and “mobile-based consultation for rehabilitation” were more important. On the other hand, “offering education” was seen as more attractive by those who had not been infected with COVID-19. Rural residents preferred functions such as “offering education,” “supporting medical research,” and “mental health therapy” more than their urban counterparts.

Mean BWS scores for the participants’ willingness to use this smartphone app (yes, no, or maybe) are shown in Figure 1. All functions can be regrouped into 3 domains. Participants who preferred to use the app were more likely to assign a high weight to the preventive functions (eg, contact tracing) than those who did not prefer to use it. Conversely, participants who showed lower willingness to use the app tended to indicate a higher preference for supportive functions than those who preferred to use it.

This is the first study to use the BWS method to inform policymakers about the importance of various smartphone app functions based on the public’s preferences for dealing with health risks and maintaining a good quality of life during the pandemic. We found surveillance- and prevention-related functions were identified as more crucial than others to uncover the meaning of the relative importance respondents attach to smartphone apps; this information can be used to inform the development of such apps to prepare for the outbreak of future public health crises. Although the value of smartphone apps in generating population-wide, real-time, and highly informative data in fighting the COVID-19 pandemic has been widely recognized, it is important to acknowledge that there are limitations to the use of smartphone apps in managing a pandemic. Due to limited resources and technical constraints, all necessary functions cannot be provided by a single app. A trade-off must be made to ensure that the most demanding functions can be included. Our findings provide quantitative evidence on which functions should be given higher priority than others based on the perspective of the general population in China.

Public health surveillance measures were listed as the most important functions of the app in our study, which is not a surprise. This is in line with previous findings that governments, private firms, and universities worldwide have used smartphone apps to surveil infected cases as a potential tool to manage the COVID-19 pandemic [20]. In this study, we identified that surveillance and monitoring of infected cases was the most preferred app function of respondents and was approximately 10.5% and 14.6% more important than the second-ranked “quick self-screening” and third-ranked “early detection of infected cases” functions, respectively. This finding is interesting because it shows that people in China may prioritize containing infected individuals over rapid detection during the pandemic. This preference reflects people’s belief that this approach can result in a strong and effective response to the outbreak of COVID-19 and could be useful in handling future public health crises as a top priority in promoting public health and safety. This aligns with the Chinese government’s COVID-19 policies and strategies, which focus on controlling the spread of the virus through measures such as mass and instant lockdowns, compulsory travel restrictions, and strict contact tracing [21,22]. However, some critics argue that this approach may come at the cost of individual liberties and privacy. Future studies about a balance to be found between public health and personal freedoms should be encouraged.

It was unexpected that providing mental health therapy and services via smartphone apps was consistently selected as the least important function in this study. It is in contrast to previous studies, which reported that mental health services are useful to some extent. For example, Liu et al [23] indicated that web-based mental health services for the COVID-19 epidemic facilitated the development of emergency public interventions and improved the quality and effectiveness of emergency interventions [24]. There may be several reasons for this finding. First, problematic use of smartphone apps can result in issues and a decrease in the desire to continue using them. For instance, a recent systematic review indicated that there are numerous digital mental health tools available in China, but a lack of education about their proper use has a negative impact on individuals’ mental health and emotional well-being [25]. Another reason might be concerns about safety and security of personal information using smartphone apps. People may not seek web-based mental health therapy due to worry that their data could be compromised or that their privacy could be violated [26]. Additionally, some people may have difficulty navigating web-based platforms or may lack access to the necessary technology to participate in web-based mental health services due to low eHealth literacy [27]. With the pandemic, it is more important than ever to take care of one’s mental health. Given the government has invested a significant amount of resources into web-based mental health services, it is essential to encourage people to use these services.

Although the Chinese government has encouraged and invested in telemedicine for years, our study found that the treatment-related functions of mobile apps are not considered very important. This indicates that previous concerns, such as lack of reliable access to the internet [28], privacy and security concerns [29], and difficulty building rapport with health care providers via web or mobile-based communication [30], have not been adequately addressed. Additionally, we found respondents showed a lack of interest in sharing their experiences in fighting the pandemic or supporting medical research. However, sharing experiences and uploading data can provide valuable insights into the ways that the virus can affect individuals, which can help researchers and health care professionals better understand and manage the disease. We suggest conducting further research to encourage the sharing of experiences. This can help reduce stigma and the spread of misinformation, as well as promote a more supportive and informed community response to the pandemic.

Further stratified analysis confirmed that there was heterogeneity of preference in different population groups. For example, individuals who had not been infected by COVID-19 were more likely to rate “offering medical education” as more important than those who had been infected. However, individuals who had been infected rated mobile-based treatment or rehabilitation services as more important. These findings reflect changes in the public’s needs and preferences during the pandemic and provide suggestions for the development of different app functions for managing health, whether the users are patients or healthy individuals. Additionally, we found that preferences for mobile app features in urban and rural residents were quite different. For instance, the rural population rated providing mental health services as being more important than did urban residents. This is reasonable, as in China, for various reasons related to culture, socioeconomics, and health care, people with mental health needs have long been underserved [31]. Although treatment for severe mental illness was incorporated into the national public health service in 2009 [32], we found that during the pandemic, the provision of services may not have been sufficient, particularly for rural populations. Therefore, it is important to further study the potential of telemedicine to improve mental health services.

The BWS case 1 method is a conventional ranking method that provides a simple rank ordering of scores to potential factors. It offers several benefits, such as reduced cognitive burden, easier choice tasks, smaller sample sizes, full rankings instead of partial rankings, and reducing personal response–style bias. However, BWS case 2 and case 3 approaches are increasingly identified as promising new ways to generate richer preference information than the case 1 method. Therefore, the use of these alternative designs should be encouraged in future studies.

Personal preference has been identified as a key mechanism for understanding users’ app adoption behavior. This emphasizes the value that users can gain from using apps and proposes that when users’ personal needs are met, they are more likely to continue using apps [33]. Our findings can facilitate the creation of effective strategies to develop mobile apps that fully consider users’ preferences and willingness to manage their health, not only during a pandemic, but also during other public health crises.

Several limitations should be addressed. First, all data used in this study were obtained from a web-based survey. This method of data collection may have introduced selection bias because individuals who are not familiar with web-based surveys may have been excluded, even though respondents in this study came from various regions of China. Second, to reduce the cognitive burden on respondents, we included only 11 smartphone app functions in the BWS questionnaire and left out other possible functions and services. This limitation may affect the generalizability of our findings. Last, since this study used a cross-sectional design, individuals’ long-term state of health and the associated changes were not taken into account in the model estimation. This may affect the validity of our findings.

This study used a BWS method to rank the importance of features of smartphone apps that provide health care services during pandemics based on general population preferences in China. The 3 most vital functions according to the respondents were “surveillance and monitoring of infected cases,” “quick self-screening,” and “early detection of infected cases.” In contrast, “mental health therapy” was listed as the least vital function for managing health during the pandemic. Our findings provide empirical evidence for decision-makers to develop eHealth policies and strategies that address future public health crises from a person-centered care perspective. Continued use of apps and smart investment in digital health can help improve health outcomes and reduce the burden of disease on individuals and communities.