Approximately a quarter of the global population has sleep problems [1-4], such as sleep deprivation, excessive sleep, and reduced sleep quality, resulting in health, safety, and economic issues. These sleep problems not only reduce the quality of life but also increase the risk of various diseases, such as high blood pressure [5], diabetes [6], and digestive diseases [7]. Sleep problems can also cause economic losses [8] and lead to traffic accidents [9]. However, in the context of the COVID-19 epidemic, traditional offline sleep interventions are difficult to perform. Therefore, a convenient, remote, and effective tool is necessary to solve or alleviate the global challenge of sleep problems.

Mobile health (mHealth) apps have the potential to provide this tool; however, their use in the real world is poor. The popularity of mobile phones and the immediacy and convenience of the mobile internet have led to the unique advantage of mHealth apps in providing sleep management services, and their effectiveness has been recognized in research. For example, Hasan et al [10] collected data from 54 randomized controlled trials, including 11,815 participants. They found that compared with conventional sleep interventions, cognitive behavioral therapy for insomnia–based apps significantly increased total sleep time, reduced sleep latency time, and improved sleep efficiency and quality. However, the real-world use of mHealth sleep apps is poor; nearly 25% of users stop using them after one use [11]. The existing literature indicates that user satisfaction is a major factor influencing continuous intention [12,13]. Therefore, exploring the factors that influence user satisfaction is vital for improving the real-world use of mHealth sleep apps.

Existing research on user satisfaction with mHealth sleep apps is limited by the small sample size, high costs, and difficulty in generalizing the findings. For example, Buman et al [14] used structured interviews to explore the acceptability and satisfaction of an mHealth sleep app (BeWell24) among 26 veterans. Similarly, Huberty et al [15] used interviews to explore the satisfaction and acceptability of an mHealth sleep app (Calm) among 88 college students. Philip et al [16] developed an mHealth app (KANOPEE) that supports the diagnosis of sleep disorders and interventions. Then, they explored the acceptability of the app by distributing questionnaires to 2000 users via the internet.

In the field of mHealth sleep apps, research based on real-world big data deeply exploring the factors influencing user satisfaction is lacking. Existing studies have shown that user satisfaction factors are asymmetrical. For example, the two-factor theory suggests that there is an asymmetrical effect between the factors influencing user satisfaction and dissatisfaction [17]. Influencing factors can be divided into motivation and hygiene factors. Motivation factors refer to the value-added attributes that users usually do not expect (ie, when some features of the app meet user expectations, users are satisfied, and when they do not meet expectations, users are not disappointed). Hygiene factors are basic attributes (ie, users will not feel satisfied when the feature of the app meets their expectations and are disappointed when the feature does not meet their expectations) [17]. Failure to consider the asymmetry of factors influencing user satisfaction may lead to poor construction and predictive power of the explanatory models [18].

Guided by the two-factor theory and using the internet as the data source to explore the factors influencing user satisfaction and dissatisfaction with mHealth sleep apps can, we can alleviate the above problems and fill the gaps in the research. Moreover, the number of user reviews is large and can reflect how the apps are used in the real world. Thus, the conclusions obtained may be more objective with less bias. Studies have analyzed user reviews of weight loss and precision nutrition apps [19,20]; however, research on mHealth sleep apps is lacking.

Therefore, in this study, we used machine learning–based topic modeling to analyze user reviews of 10 mainstream mHealth sleep apps in Chinese and American mobile app stores. This study had the following research objectives:

In the data preprocessing section, we applied the Python Natural Language Processing Toolkit and Sentiment Knowledge Enhanced Pretraining (SKEP) algorithm to preprocess the user review data of the 2 countries in the following seven steps:

Finally, 372,730 user reviews were included after data preprocessing, including 202,963 (54.45%) American user reviews and 169,767 (45.55%) Chinese user reviews.

We used latent Dirichlet allocation (LDA), a 3-level hierarchical Bayesian model in which each item of a collection is modeled as a finite mixture over an underlying set of topics, for collections of discrete data such as text corpora, which provides easy operation and high performance [24,25]. We used Python Gensim library to implement the LDA modeling. According to the perplexity curves, coherence scores, classification results, and realistic considerations (Figures S1-S6 in Multimedia Appendix 1), we identified 14 topics in Chinese and American user reviews.

The two-factor theory is the most widely replicated study on user satisfaction [17]. We set 2 dependent variables, namely, positive deviation (PD) and negative deviation (ND) [26], for data analysis. The 2 variables were defined by the difference between the user rating and the overall rating of the app in the app store. The range of values for PD was (0,4) and that for ND was (−4,0). The independent variable in this study was the probability distribution of each topic returned by the LDA model for each user review. In our study, the Tobit model, which is designed to estimate linear relationships between variables when there is either left-censoring or right-censoring in the dependent variable, was chosen to explore the factors influencing user satisfaction and dissatisfaction as follows:

Where β_k represents the correlation coefficient between the k^th topic X_ki for comment I and user satisfaction (PD_i and ND_i), k represents the number of topics included in the model, and δ_i is the error term.

Finally, we applied the Wald test [26] to verify the variance of the parameters in the PD and ND models to analyze the asymmetrical effects of the factors influencing user satisfaction and dissatisfaction.

A total of 488,071 user reviews of 10 mainstream mHealth sleep apps were obtained, and after data preprocessing, 372,730 (76.4%) user reviews (202,963, 54.5% from the United States and 169,767, 45.5% from China) were included. After LDA topic modeling, we identified 14 topics in Chinese and American user reviews (the topics identified in both sets were not identical). The topic modeling results are shown in Table 2 and Table 3.

Tables 4 and 5 show the results of the Tobit analysis. Topic 8 in the Chinese data was user attitude (positive), and it was excluded from the Tobit model because it did not include user opinions on app features and effectiveness.

In Table 4, model 1 shows the results of the PD for the Chinese user data. We found that 13 user-discussed topics had a significant effect on the PD of user satisfaction. The variable with the largest positive effect was the app’s sound recording function (β=1.026; P<.001), and the variable with the largest negative effect was the value of money (β=−1.232; P<.001). Model 2 provides the results of ND for Chinese user data, and all 13 topics were statistically significant. The variable with the largest positive effect was the sleep improvement effect of the app (β=1.185; P<.001) and that with the largest negative effect was the reliability of the app (β=−3.268; P<.001).

In Table 5, model 3 shows the PD results for American user data. All 14 topics had a significant effect on the PD of user satisfaction; the variable with the largest positive effect was the app’s sleep improvement effect (β=1.168; P<.001) and that with the largest negative effect was the app’s reliability (β=−0.826; P<.001). Model 4 presents the results of the ND for American user data. All 14 topics included were statistically significant; the variable with the largest positive effect was the app’s sleep promotion effect (β=1.389; P<.001) and that with the largest negative effect was the sleep tracking function (β=−1.446; P<.001).

To explore the effects of asymmetry of the influencing factors, we used the Wald test to verify the differences in the parameters between models 1 and 2 and between models 3 and 4. The results of the Wald test are presented in Tables 6 and 7. We found that between models 1 and 2 and between models 3 and 4, all effect parameters were significantly different.

In Table 6, for Chinese users, 6 topics, namely, sleep improvement effect, sleep advice, alarm, sleep evaluation, sound record, and reminder function of the app, had a significant positive impact on the 2 models, with a significant difference in impact. In contrast, 7 topics, namely, the value of money, reliability, usability, sleep tracking, other device support, user interface, and advertisement, had a significant negative impact on both models.

In Table 7, for American users, 6 topics, namely, sleep improvement effect, user interface, ease of use, meditation, sharing function, and sleep advice, had significant positive effects on the 2 models, and the effects were significantly different. In contrast, 8 topics, namely, sleep tracking, sound recording, alarm, sleep evaluation, activity tracking, value of money, compatibility, and reliability, had a significant negative impact on both models, and the impact was significantly different.

This study had several limitations. First, because not all users provide reviews, we cannot evaluate the satisfaction with mHealth sleep apps of such users, which could lead to selection bias. Nevertheless, 372,730 user reviews were analyzed, which provided sufficient data to explore user satisfaction with mHealth sleep apps and their influencing factors. Second, our analysis did not consider the characteristics of app users and the metadata of user reviews, which may have improved the accuracy of the model and significantly affected the results. However, the user review data in mobile app stores usually contain the user’s name, release date, user score, and user text reviews. In addition, we cannot obtain the user characteristics and metadata of user reviews without the assistance of app developers. This is a common limitation in user-generated content studies. Moreover, only user reviews of 10 mainstream mHealth sleep apps from mobile app stores in China and the United States were included, leading to possible bias. However, China and the United States have the highest number of internet users in the world, and the number of active users of mainstream apps far exceeds that of other apps. Thus, we believe that the results of this study are representative and adequate to reflect the overall satisfaction with mHealth sleep apps.

To improve users’ intention to continue using mHealth sleep apps, we identified motivation and hygiene factors for such app use by processing and comparing >480,000 user reviews of 10 mainstream sleep apps in China and the United States. We found that the user satisfaction rates of mHealth sleep apps were poor, resulting in users’ low continued use intentions. Moreover, the effects of the influencing factors of Chinese and American user satisfaction and dissatisfaction with mHealth sleep apps were asymmetrical. Overall, the motivation factors for mHealth sleep apps were the sleep suggestion function and sleep promotion effects of the apps. The hygiene factors included the reliability, compatibility, value of money, and sleep monitoring function of the apps. To promote the use of mHealth sleep apps in the real world, we should first improve the hygiene factors and then achieve the motivation factors. In future research, we should focus on the characteristics of the user and the metadata of user reviews and then introduce them into the equation as covariates to further improve the accuracy of the model.