Assessing the Quality of Mobile Exercise Apps Based on the American College of Sports Medicine Guidelines: A Reliable and Valid Scoring Instrument

Background Regular physical activity can not only help with weight management, but also lower cardiovascular risks, cancer rates, and chronic disease burden. Yet, only approximately 20% of Americans currently meet the physical activity guidelines recommended by the US Department of Health and Human Services. With the rapid development of mobile technologies, mobile apps have the potential to improve participation rates in exercise programs, particularly if they are evidence-based and are of sufficient content quality. Objective The goal of this study was to develop and test an instrument, which was designed to score the content quality of exercise program apps with respect to the exercise guidelines set forth by the American College of Sports Medicine (ACSM). Methods We conducted two focus groups (N=14) to elicit input for developing a preliminary 27-item scoring instruments based on the ACSM exercise prescription guidelines. Three reviewers who were no sports medicine experts independently scored 28 exercise program apps using the instrument. Inter- and intra-rater reliability was assessed among the 3 reviewers. An expert reviewer, a Fellow of the ACSM, also scored the 28 apps to create criterion scores. Criterion validity was assessed by comparing nonexpert reviewers’ scores to the criterion scores. Results Overall, inter- and intra-rater reliability was high with most coefficients being greater than .7. Inter-rater reliability coefficients ranged from .59 to .99, and intra-rater reliability coefficients ranged from .47 to 1.00. All reliability coefficients were statistically significant. Criterion validity was found to be excellent, with the weighted kappa statistics ranging from .67 to .99, indicating a substantial agreement between the scores of expert and nonexpert reviewers. Finally, all apps scored poorly against the ACSM exercise prescription guidelines. None of the apps received a score greater than 35, out of a possible maximal score of 70. Conclusions We have developed and presented valid and reliable scoring instruments for exercise program apps. Our instrument may be useful for consumers and health care providers who are looking for apps that provide safe, progressive general exercise programs for health and fitness.


No
Partially Yes 1 2 3 4 5 Note: Safety warnings in disclaimers do not count -some apps may have safety warnings at the beginning of workout videos.

Does the app advise you to perform aerobic exercise 3-5 times per week?
No Partially Yes 1 2 3 4 5 Note: Give a partial score if the app does not explicitly instruct you to perform aerobic exercise 3-5 times per week but allows choosing exercise frequency in the program.

Does the app advise you to monitor the intensity of aerobic exercises by monitoring heart rate?
No Partially Yes 1 2 3 4 5

Note:
1. Other methods for measuring aerobic exercise intensity include the perceived exertion methods in which you may be asked "How hard are you working on a scale of 1-10 or 6-20 points?" 2. Give a partial score if the app does not explicitly instruct you but allows you to monitor the intensity.

No
Partially Yes 1 2 3 4 5 Note: ACSM recommends adults get ≥150 minutes of moderate (or ≥75 minutes of vigorous) intensity exercise per week. This can be met through 30-60 minutes of moderate (or 20-60 minutes of vigorous) intensity exercise per day.

Does the app advise you to engage in different types of aerobic exercises?
No Partially Yes 1 2 3 4 5

Note:
1. Give a partial score if the app does not explicitly instruct you to engage in different types of aerobic exercises but provides/demonstrates different types of aerobic exercises in the workout routines. 2. Examples of aerobic exercises include but are not limited to walking, running, swimming, cycling, and rowing.

Does the app advise you to progress appropriately for aerobic exercises?
No Partially Yes 1 2 3 4 5

Note:
1. Give a partial score if the app does not explicitly instruct you how to progress appropriately, but there are signs of appropriate progression in the workout routines. Note: 1. Give a partial score if the app does not explicitly instruct you to warm up or how to warm up but has warm up activities in the workout routines. 2. ACSM recommends warming up for 5-10 minutes with light or moderate cardiovascular exercise.

No
Partially Yes 1 2 3 4 5 Note: ACSM recommends 20-60 minutes of aerobic, resistance, neuromotor, or sport activities; or accumulative bouts of 10 minutes of activity during the day.

Does the app advise you to cool down after aerobic exercise?
No Partially Yes 1 2 3 4 5

Note:
1. Give a partial score if the app does not explicitly instruct you to cool down or how to cool down but has cool down in the workout routines. 2. ACSM recommends cooling down for 5-10 minutes with light or moderate cardiovascular exercise.

Note:
1. Give a partial score if the app does not explicitly instruct you to stretch after aerobic exercise but includes stretching activities after aerobic exercise in the workout routines. 2. ACSM recommends stretching for ≥10 minutes after exercise.

Part II. Strength and Resistance Does the app have a strength/resistance training component? □ Yes □ No (Skip to Part III if the answer is No)
A. Safety

No
Partially Yes 1 2 3 4 5 Note: Example safety warnings: do not keep exercising if pain develops; perform with proper technique (full range of motion in joint when possible); perform with controlled inhaling during the eccentric phase and exhaling during the concentric phase.

Does the app advise you to train each major muscle group 2-3 times per week?
No Partially Yes 1 2 3 4 5

Note:
1. Give a partial score if the app does not explicitly instruct you to train 2-3 times per week but allows choosing exercise frequency in the program. 2. Give a partial score if the app does not cover all the major muscle groups: quadriceps, hamstrings, calves, chest, back, shoulders, triceps, biceps, forearms, trapezius, and abs. Note: Single-joint exercises can be included but are not required. Using a variety of equipment is not required, but each routine should include all major joints such as the knee, hip, shoulder.

Does the app clearly state or support the appropriate repetitions for strength training?
No Partially Yes 1 2 3 4 5 Note: ACSM recommends 8-12 repetitions to improve strength for most adults and 10-15 repetitions to improve strength in middle aged and older persons starting exercise.

Does the app clearly state or support the appropriate sets for strength training?
No Partially Yes 1 2 3 4 5 Note: ACSM recommends 2-3 sets for most adults and 1 set for older and novice trainees.

Does the app clearly advise you to rest for 1-3 minutes between sets and 48 hours between training days?
No Partially Yes 1 2 3 4 5

Note:
1. Give a partial score if either the between-sets or the between-days rest recommendation is missing. 2. Give a partial score if the app does not explicitly instruct you to rest but has appropriate rests in workout routines.

Does the app clearly advise you to progress gradually each week for greater resistance, repetitions, or frequency?
No Partially Yes 1 2 3 4 5 Note: Give a partial score if the app does not explicitly instruct you to progress appropriately, but there are signs of appropriate progression in the workout programs.