This is an openaccess article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Due to a resurgent debate on the misuse of
The objective of this study is to reperform the primary and sensitivity analysis of the AMADEUS2 trial using a Bayesian framework and to compare the results with those of the original analysis.
The same regression models used in the original analysis were employed in this reanalysis (negative binomial regression). Model parameters were given uniform priors. Markov chain Monte Carlo was used for Bayesian inference, and posterior probabilities were calculated for prespecified thresholds of interest.
Null hypothesis tests did not identify a statistically significant difference between the intervention and control groups, potentially due to a few extreme data points. The Bayesian analysis indicated a 93.6% probability that there was a difference in grams of alcohol consumed at followup between the intervention and control groups and a 71.5% probability that the incidence rate ratio was <0.96. Posterior probabilities increased when excluding three potential outliers, yet such post hoc analyses were not necessary to show the preference toward offering an eSBI to harmful and hazardous drinkers among university students.
The null hypothesis framework relies on point estimates of parameters.
International Standard Randomized Controlled Trial Number (ISRCTN) 02335307; http://www.isrctn.com/ISRCTN02335307
During the past decade, student health care centers across Sweden have routinely invited all students they serve to complete an electronic screening and brief intervention (eSBI) targeting harmful and hazardous alcohol consumption. Students are, on a yearly basis, invited via email to complete a 10item questionnaire, after which they are given personal feedback alongside some advice on behavior change. The evidence for eSBIs generally indicates that they may have a small, yet positive effect on the amount of alcohol consumed in the short term (Cohen d=−0.17, 95% CI −0.27 to −0.18 [
In 2011, the first
The unconventional trial design employed in the AMADEUS1 trial necessitated inclusion of many individuals at followup who had decided not to complete the baseline assessment, as well as of nonharmful drinkers and abstainers. This prompted the AMADEUS2 trial [
The AMADEUS2 trial [
Eligible students who gave consent to take part in the trial were randomized into two groups: intervention and control. The intervention group was offered an eSBI immediately after randomization. They were asked to complete a 10item questionnaire, which assessed their current consumption, after which they received feedback on their responses, including graphical representations of their current risk level, normative comparison with other students, and personal advice on how to reduce one’s consumption. The control group was told that they would receive the intervention in 2 months.
At followup, 2 months after the initial invitation, both groups were sent identical emails with an invitation to participate in the followup survey. The survey consisted of the same questionnaire and feedback that was offered to the intervention group at baseline.
In 2017, Benjamin et al [
This recommendation met critique, as others believed that trichotomization of evidence does not solve the issue of Phacking, selective reporting, and publication bias [
One approach that could potentially replace the
The primary outcome in the AMADEUS2 trial was selfreported weekly alcohol consumption at the 2month followup. The main hypothesis was that the intervention group would report a lower weekly alcohol consumption than the control group at followup. An unplanned sensitivity analysis was also conducted, which excluded three data points considered outliers post hoc. The objective of this study is to redo the primary and sensitivity analysis using a Bayesian framework and contrast the results with those of the original analysis.
In the original analysis of the AMADEUS2 trial, negative binomial regression was used to contrast grams of alcohol consumed per week between the intervention and control groups. The primary model was adjusted for baseline variables. The same model was used in the enclosed Bayesian analysis, with uniform priors for all model parameters. Negative binomial regression with uniform priors used to contrast grams of alcohol per week is expressed by Equation 1:
Equation 1 presents the full specification of the model, where HED represents the number of heavy episodes of drinking per week at baseline, that is, the initial screening question.
The primary interest was the regression coefficient θ_{1} for the GROUP variable, that is, the expected difference in log count of grams of alcohol consumed between the intervention and control groups. By exponentiating this coefficient, we get the incidence rate ratio (IRR), which indicates by how much we should multiply the control group’s consumption to get the intervention group’s consumption. Thus, a value of exp (θ_{1}) lower than 1 would suggest that the grams per week consumed for the intervention group was lower than that for the control group at the time of followup. Informed by the original analysis, thresholds for which the marginal posterior distribution of exp (θ_{1}) should be inspected were chosen at 1, 0.96, and 0.92. The threshold of 1 was chosen to communicate whether offering the intervention was preferable to not doing so, and the thresholds 0.96 and 0.92 were chosen to indicate the magnitude of the difference between the two groups.
Hamiltonian Monte Carlo, a type of Markov chain Monte Carlo (MCMC) technique, was used for Bayesian inference. The model was coded using Stan (
When using MCMC for inference, we aim to draw samples from the posterior distribution of all model parameters. These samples can then be used to calculate how probable different values of these parameters are. For each model in the enclosed analysis, 50,000 iterations were run with 25,000 warmup iterations in four chains.
data {
int<lower=1> N; // Number of data items
int<lower=1> K; // Number of predictors
matrix[N,K] X;
int<lower=0> y[N]; // Response
}
parameters {
real<lower=0> phi; // Dispersion parameter
vector[K] beta;
}
model {
y ~ neg_binomial_2_log(X * beta, phi);
}
This study was approved by the Regional Ethical Committee in Linköping, Sweden (No. 2013/4631).
In total, 1605 eligible students agreed to take part in the trial, of which 825 were randomized to the intervention group and 780, to the control group. Two months after the initial invitation, 58% (931/1605) of trial participants completed the followup questionnaire.
Part of the original analysis of the AMADEUS2 trial is presented in
In an unplanned sensitivity analysis, data were graphically assessed for skewness (using QQ plots), and three potential outliers were identified (
Original analysis of grams of alcohol consumer per week at followup compared between the intervention and control groups. When removing three potential outliers, the difference was marginally statistically significant.

Intervention group (n=402), mean (SD)^{a}  Control group (n=529), mean (SD)^{a}  Incidence rate ratio^{b} (95% CI)  
Weekly alcohol consumption (g/wk)^{b}  113.4 (81.1)  120.8 (86.4)  0.937 (0.8611.019)  .13 
Sensitivity analysis excluding three outliers  107.4 (73.4)  119.1 (81.3)  0.921 (0.8481.000)  .049 
^{a}Mean and SD given by negative binomial regression.
^{b}Incidence rate ratio given by negative binomial regression (adjusted for sex, age, university, and frequency of heavy episodic drinking at baseline).
Unplanned sensitivity analysis identifying three potential outliers with respect to weekly alcohol consumption.
We recall from our discussion in the Methods section that Equation 1 represents the coefficient for the GROUP variable, that is, the difference between the intervention and control group in terms of log count of grams of alcohol consumed per week. We get the IRR by exponentiating this coefficient. The control group’s consumption is multiplied with the IRR to get the intervention groups consumption, thus an IRR less than 1 implies that the intervention group consumed less than the control.
Histograms of the samples drawn from the posterior distribution of θ_{1} during MCMC are shown in
For different IRR thresholds of interest, we can calculate the marginal posterior probability by simply counting the rate of samples that fall below or above a given threshold. In
No sampling issues during MCMC were found when inspecting trace plots (
Samples from the posterior distribution of θ_{1} (exponentiated).
Samples from the posterior distribution of θ_{1} (excluding three potential outliers, exponentiated).
Bayesian analysis of incidence rate ratios comparing the intervention and control groups at followup.

Intervention (n=402), mean (SD)  Control (n=529), mean (SD)  Probability^{a} (%)  

Incidence rate ratio<1  Incidence rate ratio <0.96  Incidence rate ratio <0.92  
Weekly alcohol consumption (g/wk)  113.4 (81.1)  120.8 (86.4)  93.6  71.5  33.9 
Sensitivity analysis excluding three outliers  107.4 (73.4)  119.1 (81.3)  97.5  83.8  49.1 
^{a}Marginal posterior probabilities for incidence rate ratios comparing intervention and control groups, given by negative binomial regression (adjusted for sex, age university, and frequency of heavy episodic drinking at baseline, see Equation1).
The original analysis of the AMADEUS2 trial did not find a statistically significant difference between the intervention and control groups at followup (
The study found no strong evidence of shortterm effectiveness of the Swedish national system of proactive online alcohol intervention for university and college students. However, inspection of the confidence intervals for the primary outcome reveals that this study does not rule out an intervention effect of up to 13% reduction in total weekly alcohol consumption.
Thus, dichotomization leads us into a state of uncertainty: We cannot rule out that the intervention had no effect, yet we cannot conclude that the intervention had an effect.
The unplanned sensitivity analysis excluding outliers identified a marginally statistically significant difference; however, such unspecified analyses should be viewed with skepticism. It is generally impossible to know which data points should be considered correct, which are data entry errors, and which are malicious entries.
Although not included in the original analysis, we calculated the
The Bayesian analysis of the AMADEUS2 trial (
When excluding the three entries with extreme levels of consumption, the probability of a difference increases. However, the difference is not extreme, partially because we are not relying on dichotomization, but mainly because in a Bayesian framework we look at the entire posterior distribution of parameters, rather than point estimates. The major benefit here is that we do not feel obligated to remove the potential outliers at all. Since analyses where outliers have been removed should be viewed with high skepticism, we can keep them in our data analysis while still obtaining similar results.
The posterior probabilities in
The AMADEUS2 trial was not sufficiently powered to obtain the prespecified effect size considered worth investigation. Approximately one quarter of the target sample size was recruited, creating a limitation on the possibility of detecting significant effect sizes. This also creates a limit for the Bayesian analysis, as the width of the posterior distribution, in general, decreases as the number of samples increase, allowing for narrower posterior distributions.
All analyses were performed under the intentiontotreat principle with complete cases, which assumes that data are missing at random. Although attrition analyses in the original publication did not find evidence against data missing at random, there was a difference in followup rates between the intervention group (404/825, 49.0%) and the control group (529/780, 67.8%), which should temper any strong conclusions from the original analysis and this reanalysis.
Finally, subjective measures were used to collect data at baseline and followup, which requires participants to recall their alcohol consumption in a typical week. Although such measurements may be subject to several sources of bias, such as recall and social desirability bias, it is the norm in brief interventions to use subjective measures, as in most cases, it is infeasible to collect biomarker data.
The use of null hypothesis testing with
In the original publication of the AMADEUS2 trial, it was acknowledged that it is challenging to reliably detect small effects and that the process may be subject to chance. Digital lifestyle interventions targeting large and sometimes non–treatmentseeking populations are generally expected to have a smalltomodest effect. Basing policy decisions on
Trace plots.
Alcohol Email Assessment and Feedback Study Dismantling Effectiveness for University Students
electronic screening and brief intervention
incidence rate ratio
Markov chain Monte Carlo
MB owns a private company that develops and distributes evidencebased lifestyle interventions to be used in health care settings, including student health care centers. No other disclosures were reported.