This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Website evaluation is a key issue for researchers, organizations, and others responsible for designing, maintaining, endorsing, approving, and/or assessing the use and impact of interventions designed to influence health and health services. Traditionally, these evaluations have included elements such as content credibility, interface usability, and overall design aesthetics. With the emergence of collaborative, adaptive, and interactive ("Web 2.0") technologies such as wikis and other forms of social networking applications, these metrics may no longer be sufficient to adequately assess the quality, use or impact of a health website. Collaborative, adaptive, interactive applications support different ways for people to interact with health information on the Web, including the potential for increased user participation in the design, creation, and maintenance of such sites.
We propose a framework that addresses how to evaluate collaborative, adaptive, and interactive applications.
In this paper, we conducted a comprehensive review of a variety of databases using terminology related to this area.
We present a review of evaluation frameworks and also propose a framework that incorporates collaborative, adaptive, and interactive technologies, grounded in evaluation theory.
This framework can be applied by researchers who wish to compare Web-based interventions, non-profit organizations, and clinical groups who aim to provide health information and support about a particular health concern via the Web, and decisions about funding grants by agencies interested in the role of social networks and collaborative, adaptive, and interactive technologies technologies to improve health and the health system.
Most of the publications on the evaluation of Web-based health applications focus on sites designed to provide health information to patients, their caregivers, or health professionals [
Eligible articles were identified through a search of (1) MEDLINE (1990 - Nov 2007), CINAHL (1990 - December 2007), Cochrane, PsycINFO (1990 - Nov 2007), Social Science Abstracts and Citation Index (1990 - 2008), and ERIC (1990 - Nov 2007); (2) personal collections of the authors; and (3) reference lists of relevant publications. The search strategy, developed in consultation with a medical librarian, included a string of Internet-related terms cross-matched with an evaluation framework string using Boolean operators. For example, the MEDLINE search used terms related to technology (Internet, World Wide Web, informatics, online), Web 2.0 terms (blog, wiki, podcast, tag), terms related to patients (some include consumer participation, education/non professional, consumer participation), and evaluation concepts (these include outcome, process, quantitative, for example). Please see
Articles published in English in a peer-reviewed journal were deemed potentially eligible for inclusion in the review:
If they described a generic evaluation framework applicable to a wide range of Web-based health applications for lay members of the public; or
For health-specific websites, if they provided a full description of the process followed for the evaluation of such a framework that met criteria point 1.
By “generic” we meant frameworks that are applicable to sites that provide information or tools to promote decision support, social support, self-management, or self-care support.
Articles concerning evaluation of websites that provided some form of therapy or treatment such as cognitive behavioral therapy or communication with health care professionals were excluded because these sites serve a purpose that is distinctly different from the sites described above. Such sites are, in effect, a form of treatment or an extension of the clinical encounter per se, rather than a means to access information and support. As such, they have important evaluation criteria that extend beyond the website itself. We also considered out of scope those articles describing evaluation of a single formal decision aid, of diagnostic aids, or of sites designed for education of or use by medical professionals.
Three investigators (LOG, HW, JB) independently reviewed a random selection of 100 titles and abstracts from the articles that were identified through the literature search. A Single Measures Intraclass Correlation Coefficient (ICC) of 0.89 (95% CI 0.85 - 0.95) was calculated using SPSS v15.0 with a two-way mixed effects model. In the repeated search one year later, the same procedure was followed. Two of the raters had zero variance in their assessments, rendering ICC calculation not applicable. In this case, of the 100 randomly selected articles, there was agreement for 98. In both instances, the level of agreement was deemed sufficient to support independent evaluation of one-third of the total yield of the search. During independent evaluation, if the eligibility of a particular citation was judged to be questionable, the investigator included it in this initial filtering step in order to allow the other investigators to make an assessment as to whether or not it satisfied the inclusion criteria. Investigators met to review and confirm each other’s findings. Two authors (HW, JB) then reviewed the full text of all of the potentially eligible articles.
Articles were selected for inclusion in the final analysis if they described an evaluation framework applicable to Web-based, consumer-oriented health applications that could be categorized under at least two of the three core evaluation phases: (1) formative evaluation, (2) summative evaluation, and (3) outcome assessment. Within each category, parameters were organized according to these temporal phases. In order to clarify the practical differences between the phases, we describe formative evaluation as a stage of development and laboratory testing, summative evaluation as a stage of field-testing, and outcome evaluation as a stage of overall impact assessment.
For each of the 12 articles, we generated a complete list of evaluation parameters. The parameters were pooled and organized via a multidimensional card sort [
The initial literature search yielded a total of 3304 citations, of which 41 were deemed to be potentially eligible for inclusion in the review. After reviewing the full reports, 13 articles that described evaluation frameworks met all of the inclusion criteria. Seven articles were identified from the literature search; and six were identified in personal libraries and reference lists. Two articles [
We identified five themes that cut across the three core evaluation phases. These included an emphasis on: (1) the People affected by the website, (2) the Content of the website, (3) the Technology of the website, (4) Human-Computer Interaction between the person and the website, and (5) effects on the greater health care community, or Health Systems Integration. These themes reflect the core attributes, user-centric, context-centric, and functionality-centric, that Currie [
In constructing this framework, we observed and filled in gaps relevant to collaborative, adaptive, interactive applications. For example, when evaluating applications that promote collaboration among users, we must consider interactions not only between humans and computers, but also between humans, mediated by computers. Accordingly, we refer to this theme as “Computer-Mediated Interaction” to encompass this larger scope. Articles within the review contained few to no elements corresponding to the “Content”, “Technology” and “Computer-Mediated Interaction” categories within the outcome assessment column, reflecting perhaps that general information websites, whose static content was governed mainly by webmasters, did not need to address these parameters during the outcome phases of the evaluation. However, the nature of collaborative, adaptive, interactive applications necessitates that evaluators consider and assess these parameters during the outcome phase of a project.
It has been suggested that applications that are “...interactive, user-centred, dynamic and evolving...” should have measures appropriate to these aspects [
None of the identified frameworks matched the evaluation needs of collaborative, adaptive, and interactive technologies; therefore, we propose a new, dynamic framework in
Evaluation schema: collaborative, adaptive and interactive technology. Elements which were not identified in the authors' review of the literature are printed in
Formative |
Summative |
Outcome |
|
People |
Identification of Stakeholder Characteristics and Needs [
|
User Traits [ Computer Proficiency [
User Perspectives [ Intentions to Use [ Satisfaction [
|
Patient Outcomes [ Impact on Interpersonal Relationships [ Patient-physician [
|
Content |
Quality and Credibility [ Utility [ Completeness [ Understandability [ Relevance [ |
Subjective Utility [
|
|
Technology |
System Robustness [ Performance [ Functionality and Features [
|
Usage Statistics: Hits; Visitors; Browsers; Errors [ System Reliability [ Speed [
|
|
Computer-Mediated Interaction |
Usability [ Accessibility [
Information Architecture [ |
User Perspectives on Usability[ User Perspectives on Accessibility[ Demonstrated Sociability[
Findability [ |
|
Health Systems Integration |
Definition of Evaluation Metrics and Process [ Ethics/Liability [ |
Administration [ Service Utilization [ Care Coordination [ Patient Safety [ |
Public Impact (may include community-defined outcomes) [ Cost-Effectiveness [ Intended Effect [ Appropriateness [ Effectiveness [ |
The category “People” contains parameters related to the individuals who are involved in using or developing the site, or who may be affected by the implementation of the site. Within this category, evaluation parameters in the formative phase consist of
At the formative stage, the category “People” must assess not only the informational needs of the stakeholders, but also the broader interests that will transform them from users of the site to contributors and collaborators of a dynamic enterprise transferring or generating new knowledge. We refer to this as “
The category “Content” describes parameters related to all content on a website, including text, images, and multimedia components. In the formative phase, evaluation of content may include appraisal of content
In collaborative, adaptive, interactive applications, the potential fluidity of content presents new challenges to evaluation. The shift towards dynamic, user-generated content necessitates a change in how credibility is depicted and its subsequent assessment [
Outcome evaluations of collaborative, adaptive, interactive applications create entirely new requirements and avenues for evaluation. For sites that support user-generated content,
The category 'Technology' refers to the underlying technology used to create and run the site. The primary formative evaluation parameter discussed in the reviewed articles was
The incorporation of new technologies into health websites serves to shift the focus of several evaluation categories. Extensive formative evaluations of
In the summative phase, the dynamic nature of collaborative, adaptive, and interactive technologies prompts evaluation of the application’s
The category “Computer-Mediated Interaction” refers to assessments of user interactions with and via the interface. In the formative phase, evaluation parameters include
At the formative stage, a shift to collaborative, adaptive, and interactive technologies expands the scope of interaction study. In the context of collaborative websites, this category is not only about interacting with the technology, but also focused on computer-assisted interactions with others. Within this category, therefore, we suggest additional focus on parameters in the formative phase such as
In the summative phase, the shift to “Computer-Mediated Interaction” expands the scope of the evaluation and assessment.
“Computer-Mediated Interaction” should also be assessed during the outcome evaluation phase and on an ongoing basis for monitoring and quality-improvement purposes.
The category “Health Systems Integration” refers to the larger system, health processes, and society in which a health website for laypeople might be implemented. Formative evaluation parameters within this category include
Evaluation of the integration of information technologies and their processes into the larger health system has been well-covered in previous frameworks. It is worth noting, however, that due to their expanded capabilities, collaborative, adaptive, and interactive technologies may offer both greater benefit and greater unintended consequences in this area [
We have presented an evaluation framework that proposes formative, summative, and outcome evaluation indicators for five themes of Collaborative, Adaptive, and Interactive applications: People, Content, Technology, Computer-Mediated Interaction, and Health Systems Integration. The increased use of collaborative, adaptive, and interactive technologies in health care and other fields underscores the importance of their evaluations. We need to determine whether something is effective before it can be of value. Collaborative, adaptive, interactive technologies are becoming pervasive and rapidly becoming an integral part of society. In record time, resources promoting collaboration such as Wikipedia, Facebook, MySpace, YouTube, and Orkut have joined the ranks of the most widely used online services in the world. With their vertiginous ascent, they have heralded an era in which the public can wield enormous power to create and share knowledge, to communicate with people and machines, and to find and evaluate services with unprecedented levels of freedom.
Although at a slightly slower pace, health-specific collaborative, adaptive, and interactive technologies are emerging, promising to transform the roles, workflows, rights, and responsibilities of all stakeholders within the system [
We offer this framework as a means to structure evaluations across a wide range of applications and purposes. In some cases there may not be sufficient resources to conduct an evaluation that addresses all of the components listed in this framework. In these situations, we suggest that the evaluation focus on the particular aspect(s) (People, Content, Technology, Computer-Mediated Interaction, Health Systems Integration) that are the most relevant to the objective(s) of the evaluation. Tailoring an outcome evaluation to the specific requirements of the funding organization rather than attempting to address all of the various components in
Evaluation plays a critical role in high-quality design, efficient development, and effective implementation of Collaborative, Adaptive, and Interactive applications. In an era of constrained resource allocation, the adoption of robust and appropriate evaluation frameworks will help to ensure that collaborative, adaptive, and interactive technologies live up to the expectations and that they contribute to the improvement of health for all.
Laura O’Grady is supported by a CIHR post doctoral fellowship sponsored by the Ontario Women's Health Council. Jackie Bender is supported by a CIHR doctoral fellowship in Cell Signaling in Mucosal Inflammation and Pain.
Funding was also provided for Laura O’Grady, Holly Witteman, and Jackie Bender by CIHR, the Supporting Institutes (Knowledge Translation Branch, Institution of Health Services and Policy Research, and Institute of Gender and Health), and the Health Care, Technology, and Place Research Training Program.
We would like to thank Marina Englesakis, Information Specialist, University Health Network, Toronto General Hospital for her assistance in developing search strategies and implementing the searches.
ARJ owns Foresight Links Corporation, a consultancy company which works in the area of systematic reviews and evaluation, which may indirectly benefit from publication of this article.
Search strings for 2007 and prior
Search string for 2008 - 2009
QUOROM statement flow diagram for Literature Search 2008 and prior
QUOROM statement flow diagram for Literature Search 2008 and 2009
application programming interface
health level 7
Intraclass Correlation Coefficient
personal digital assistant