This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Universities have begun deploying public Internet systems that allow for easy search of their experts, expertise, and intellectual networks. Deployed first in biomedical schools but now being implemented more broadly, the initial motivator of these research networking systems was to enable easier identification of collaborators and enable the development of teams for research.
The intent of the study was to provide the first description of the usage of an institutional research “social networking” system or research networking system (RNS).
Number of visits, visitor location and type, referral source, depth of visit, search terms, and click paths were derived from 2.5 years of Web analytics data. Feedback from a pop-up survey presented to users over 15 months was summarized.
RNSs automatically generate and display profiles and networks of researchers. Within 2.5 years, the RNS at the University of California, San Francisco (UCSF) achieved one-seventh of the monthly visit rate of the main longstanding university website, with an increasing trend. Visitors came from diverse locations beyond the institution. Close to 75% (74.78%, 208,304/278,570) came via a public search engine and 84.0% (210 out of a sample of 250) of these queried an individual’s name that took them directly to the relevant profile page. In addition, 20.90% (214 of 1024) visits went beyond the page related to a person of interest to explore related researchers and topics through the novel and networked information provided by the tool. At the end of the period analyzed, more than 2000 visits per month traversed 5 or more links into related people and topics. One-third of visits came from returning visitors who were significantly more likely to continue to explore networked people and topics (
Returning visitors in an ever-increasing pool of visitors to an RNS are among those that display behavior consistent with using the tool to identify new collaborators or research topics. Through direct user feedback we know that some visits do result in research-enhancing outcomes, although we cannot address the scale of impact. With the rapid pace of acquiring visitors searching for individual names, the RNS is evolving into a new kind of gateway for the university.
New tools are enabling the search and discovery of researchers, their expertise, intellectual output, and professional networks. These Web-based applications mine a variety of data sources to automatically generate searchable profiles and expose existing networks of collaborators. In addition to large-scale commercial services [
RNSs deployed at medical schools can be contextualized against industry deployments of “expertise location” and “social networking” systems [
With an increasing number of RNSs deployed in academic biomedicine, there is now a focus on understanding how these tools are used. In industry, studies show that such tools are used predominantly to search for people [
In 2010, the University of California, San Francisco (UCSF) became the first external adopter of an open source RNS called “Profiles Research Networking Software” [
The study plan was submitted to the UCSF Institutional Review Board (Human Research Protection Program) and was determined not to be human subjects research.
A screenshot of the UCSF Profiles RNS is shown in
Screenshot of profile in UCSF Profiles. Subareas A, B, C denote 3 sets of links that when clicked through count as an additional page on site. Subareas A1 and A2 denote links connecting visitors to related people and topics.
We abbreviate the RNS “UCSF Profiles” to “Profiles”. We also use the term “profile” without capitalization to denote individual pages describing researchers. Although we describe the users and usage of an RNS, we refer to those who use the site as “visitors” to minimize confusion between our description of those who visit the site versus those who own profiles on the site and can log in to access additional owner-related functions.
To understand visitors to Profiles and how they evolved over time, we analyzed data from December 2009 (the beta launch of Profiles) through December 2012. We primarily present time course data for the 28 months from the public launch in September 2010 through December 2012 and summary data from the last 6 months analyzed (Months 23-28), from July 2012 through December 2012.
Individuals were added mostly en masse and offered the option to opt-out of the system (7 did), or invited to opt-in if their relationship with the university was less formal or if otherwise deemed appropriate (for example, faculty whose primary affiliation is with another institution but have “without-salary” appointments at UCSF). A total of 7297 profiles were created from the launch date to December 31, 2012, at which point there were 5928 active profiles. The difference represents people no longer at UCSF and the 7 who chose to opt out. Individuals added to Profiles by December 2012 included 2691 full time faculty, 356 without-salary faculty, 386 instructors, 2079 trainees (postdoctoral scholars, residents, and fellows), and 1785 other university leaders, researchers, and staff.
Website usage was measured and analyzed primarily using Google Analytics, which is an industry standard Web analytics measurement framework [
We used terminology consistent with that used by Google. We filtered out IP addresses associated with our internal office to remove the effect of internal testing and usage. “Visit” indicated a series of one or more interactions with the website, separated by gaps of no more than 30 minutes between subsequent interactions. “Pages/Visit” represented the number of individual Web pages accessed over the course of a visit. “Time on Site” meant the length of time between the first and last interaction during a visit (time on site is ignored for visits of only one page). “Visitor Location” indicated a visitor’s organizational and/or geographic location as inferred by performing geolocation and network ownership analysis of a user’s IP address. We divided locations into five non-overlapping categories: UCSF, San Francisco, California, US, and World.
“UCSF” included visitors who originated from an IP address associated with the UCSF internal network; 99.73% (114,944/115,258) of these visits originated from facilities in the City of San Francisco. This underestimates UCSF visitors because it excludes some UCSF sites (eg, San Francisco Veterans Administration Hospital, the San Francisco General Hospital, and the Gladstone Institutes) and those using mobile devices or working off-site without a VPN. To help us understand the degree to which we underestimated UCSF visits, we analyzed IP addresses of those who opened emails targeted specifically to UCSF employees (using the tool MailChimp). Of 615 opened emails, 344 (55.9%) were opened from campus IP addresses and 271 (44.1%) from other external IP addresses. Though it may be more common for mobile visitors to read email than access websites, this nevertheless suggests that visits from the UCSF campus network reflect only a portion of visits from all UCSF personnel, and the actual percentages could be up to 79% higher than the numbers we see coming from the campus network. In this paper, however, when we refer to UCSF visitors, we mean only those unambiguously located with the campus network.
“San Francisco” included visitors who originated from the City of San Francisco, other than those from the UCSF internal network. We expect that this included almost all of the remaining UCSF sites that were not otherwise covered by the UCSF network, though their contributions to overall visits are small. San Francisco is also the location most likely to capture traffic from patients of the UCSF Medical Center.
“California” included visitors who originated from California other than those from either San Francisco or the UCSF internal network. This too included some UCSF-related traffic, since many researchers and staff live in the greater Bay Area outside of San Francisco. “US” included visitors who originated from the United States outside of California, and “World” included visitors who originated from sites outside the United States.
Under “Referring Source”, terms included (1) “Search”, visits from search engines (eg, Google, Bing, Yahoo, UCSF.edu search engine), (2) “Website Referrals”, visits from other websites, excluding traffic via search engines, and (3) “Direct or Unknown”, visits from sources that cannot be programmatically identified (eg, user typing in “profiles.ucsf.edu” on their Web browser, clicking a browser bookmark, clicking a link in a desktop email or Twitter client, or following a link from websites that use the secure HTTPS protocol to the non-secure Profiles website). For “New and Returning Visits”, returning visits are those sessions (visits) where the visitor is recognized as having visited the site before via the presence of cookies. Because of the reliance on cookies, this is generally an underestimate of the true value.
We report on “Depth of Visit”, which can be measured as time on the site or as pages per visit (pages/visit). We used pages/visit since that most directly addresses the next steps visitors take after viewing a profile page. A one-page visit might mean that the visitor arrived on a profile page and then stopped using Profiles or clicked a link to some other website (such as PubMed) without returning in the next 30 minutes. Any visit that lasts for 2 or more pages occurs because a visitor clicks on one of three types of links from a profile page. These three categories of links are depicted in
Between September 3, 2011 and January 31, 2013, site visitors were asked, “How has Profiles helped you?” The survey appeared on all Profiles pages to all visitors (unless the visitor chose to minimize the survey’s inline pop-up), and 475 comments were submitted. Of these, 5 comments were deleted for having no content or email address, and 1 was deleted as it came from a Profiles developer. The remaining 469 responses were binned into seven categories based on the content and submitter’s email address (if available): (1) Enables Research, (2) Provides Background Information for Clinical Care, (3) Provides Contact Information for Clinical Care, (4) Generally Positive, (5) Negative, (6) Other, and (7) Spam.
We assessed whether depth of usage of Profiles (measured as pages per visit) differed based on whether a visit was a first-time or returning visit. We also assessed whether UCSF visitors differed in their behavior from other visitors. Unfortunately, Google Analytics does not provide access to raw data describing depth of visit (pages/visit) for each individual visit and as a result statistics were performed on daily averages for each category measured. Daily averages were analyzed for 6 months from July 1 through December 31, 2012, resulting in n=184 days for which average daily depth of visit was calculated for new and returning visitors. A standardization approach was used to control for varying “n” contributing to daily averages. Two-factor analysis of variance (ANOVA) with multiple comparisons was performed on the data. The 2x5 ANOVA assessed whether there was an interaction effect of visitor status (new or returning) with location (UCSF, San Francisco, California, USA, World) in explaining depth of usage of Profiles. All tests were two-sided and a statistical comparison or model was considered significant if
Location of visitors to Profiles for the 28 months since launch.
We assessed how visitors arrived at Profiles by analyzing the referring sources of traffic. From July to December 2012 (Months 23-28), search traffic accounted for 208,304 (74.78%) of the 278,570 visits to Profiles, referrals from other websites accounted for 47,861 (17.18%), and direct or unknown sources for 21,926 (7.87%). Google dominated search traffic; 190,170 (91.29%) of the 208,304 visits from a search engine came via Google, while another 7992 (3.84%) came via the search engine on the UCSF website, which is powered by Google and uses the same underlying ranking algorithms. The remaining 10,142 (4.87%) of search-driven traffic (“Other search”) came via sources such as Bing and Yahoo. Visitors from all sources arrived on Profiles via search, but non-UCSF visitors had a greater likelihood of landing on Profiles via a search engine than UCSF visitors (80% for non-UCSF vs 63% for UCSF). The time course of visitor acquisition from search, website referrals, and direct or other sources is shown in
Time course of growth in traffic from various referral sources (peak in direct traffic in September 2010 coincides with launch activities at UCSF).
We analyzed 250 search queries that landed on Profiles randomly selected from the set of such queries during Months 23-28. In this sample, of all searches that sent visitors to Profiles, 210 (84.0%) included a name, with half of these only including a person’s name (42.0%, 105/250). Another large set used the name along with a reference to the university (33.2%, 83/250; eg, <name> ucsf ; <name> university of california san francisco). A smaller set searched for a name along with additional information (8.8%, 22/250; eg, <name> radiation oncology). Finally, 10.0% (25/250) used the name of a paper, and the remaining 6.0% (15/250) included a mix of the name of the university (without other information), URLs, and other data.
The percentage of visits from returning visitors doubled in the first year (from 1203 of 7039 visits, or 17.09%, in the month of launch, September 2010, to 7235 of 21,924 visits, or 33.0%, in August 2011) but remained constant after that. But as visits continued to increase, this means that the number of visitors who returned to the site continued to increase as well. A higher proportion of UCSF visits included returning visitors (26,758 of 40,140, or 66.66% in Months 23-28) compared to non-UCSF visits (66,105 of 238,430, or 27.73% in Months 23-28). But returning visits continued to increase in absolute numbers for both UCSF and non-UCSF visits. Returning visits are of particular interest not only because they represent a group of visitors who find the tool useful, but, as we show below, these visitors also tend to use the tool in more depth.
We analyzed the distribution of visits by depth of visit for the period from Months 23-28. As
To further understand the distribution of actions taken by visitors immediately after they landed on a profile page, we evaluated actions taken on the first page of 1024 visits to profile pages randomly selected from all visits during Months 23-28. The results are described in
Actions taken from a profile page (n=1024).a
Action category | n (%) | Descriptive data |
Leave (Exit) | 706 (68.95) | Profile page is last page user looked at before leaving the website |
|
||
|
20 (1.95) | Clicks on list of all Co-Authors |
|
12 (1.17) | Clicks on list of all Similar People |
|
5 (0.49) | Clicks on list of all Keywords for that person |
|
||
|
74 (7.23) | Clicks to another user’s profile (eg, by clicking on a Co-Author, Similar Person, etc, link) |
|
24 (2.34) | Clicks back to list of search results, then clicks on another person |
New Searchb | 79 (7.71) | Runs a new search (eg, by entering text in mini search box) |
|
||
|
13 (1.27) | Clicks to homepage |
|
6 (0.59) | Clicks on "How Profiles Works" page |
Edit Profile | 10 (0.98) | Clicks on edit profile link |
Unknown | 75 (7.32) | Unknown or unrecorded |
aAnalysis of user behavior on the initial landing page for 1024 visits to profile pages randomly selected from all visits during Months 23-28.
bPaths to “networked links” that connect the initial profile page to related people or topics.
Distribution of visitors by depth of visit (pages/visit) during Months 23-28 from launch.
Depth of visit (pages/visit) for new and returning visitors. The figure describes visitors originating from different locations during Months 23-28 from launch. Error bars show 1 Standard Deviation.
In September 2011, we implemented an online user feedback form that asked visitors, “How has Profiles helped you?” During the 16 months after implementation (through Jan 2013), 469 submissions were received from a total of 670,000 visits, with 0.07% of all visits converted to a submission. Overall, 284 (60.5%, 284/469) comments describing positive outcomes were submitted. We found 58 (12.4%, 58/469) described how Profiles enabled research, 91 (19.4%, 91/469) described how Profiles enabled clinical care by providing contact information, 18 (3.8%, 18/469) described how Profiles enabled clinical care by providing research-related information on clinicians, and 117 (24.9%, 117/469) were positive without specific details. In addition, 118 (25.2%, 118/469) categorized as “Other” primarily requested help with the tool, 3 (0.6%, 3/469) were spam, and 64 (13.6%, 64/469) were “Negative”. Of the “Negative” comments, 29 (45%) out of 64 left negative comments reflecting their unmet expectations of a traditional directory listing (such as organization by department or sub-specialty), while 21 (33%) simply said “no”, and the remainder noted a negative comment related to technical difficulties. We show a sample of the comments, categorized as “Enabling Research”, in
I am hoping that it just helped me find a mentor...
Found speakers for grand rounds at SFGH pediatrics.
Great resource for finding potential research collaborators and for PhD dissertation committees.
It is wonderful to find the current publications. My faculty rarely tell me. I need it for our annual report and for some grant applications.
This profile allowed me to review the background and experience of an author from a journal article I was reading.
Research information about scientists for my biomedical foundation.
Highlights/generates awareness about science accomplishments beyond my immediate sphere of research but very stimulating and interesting.
Gives me a profile of a potential contributor to a book that is under consideration, helps to locate people in a specific field. Important service!
Making your remarkable research available to me and my translational research projects is invaluable. Thank you so very much.
Trying to find interesting colleagues to meet with during my stay at UCSF.
Quite useful in introducing UCSF colleagues to others outside the university. Also a quick way to get a feel for the interests of people I don’t know.
To find researchers with common interests.
Contact a faculty member recommended by an associate. Initiate a sponsored project.
Research networking systems automatically generate and publically display profiles and networks of researchers. Within 2.5 years, our system, UCSF Profiles, achieved one-seventh of the monthly visit rate of the longstanding main university website, with a continually increasing trend. Visitors came from diverse locations beyond the institution. Responses to an online survey suggest a broad range of benefits supporting the research and clinical mission. Returning visitors are among those that appear to use the system to identify new collaborators or research topics. And, with the continually increasing visits arising from public searches for individual names, UCSF Profiles is evolving into a new kind of online gateway for the university.
Schleyer et al propose a comprehensive framework to advance the study of research networking systems [
Although the initial goal of Profiles was to enable collaboration by making it easier for researchers to find partners and build teams, the most striking aspect of its deployment was how rapidly visitors were drawn from both within and outside the university. In January 2013, visits to UCSF Profiles were one-seventh of the overall visits to the established UCSF campus website and, at the time of publication, visits had caught up to almost one-fifth (18%) of that traffic. We attribute several factors to this increase in visits. As
In efforts to understand the rapid pace of acquiring visitors, we also found that most visits to Profiles came from visitors searching for a name. As visits continue to increase, Profiles continues to capture more of this type of visitor. Where might these Internet users have landed had they not found Profiles? Perhaps not the primary university website—compared to more than 70% of visits arriving via search to Profiles, only 25% of visits come via search to the university website. But, most researchers have multiple affiliated websites, some official such as departmental, laboratory, or clinical, and some not, such as news articles, and some related to other organizations such as a journal or corporation. The visits being captured by Profiles were otherwise likely to be dispersed among these various possibilities. Instead, they are now being aggregated and captured by the university RNS. And, with its search-optimized interface and increasing number of personally-curated elements, the site is also becoming the link of choice for external media. When Gurpreet Dhaliwal was profiled in the
An assessment of the searches that bring visitors to Profiles does not describe the universe of searches. Still, of those brought to Profiles via search engines, we found that 84% searched for individuals’ names with or without accompanying search terms including the institution’s name. But regardless of whether a visitor to Profiles is aware of the institutional affiliation of the person for whom they search, by the time they land on a profile page they are in effect being exposed to the institution. And when they continue on the site, they are introduced to the university through the unique lens of expertise and intellectual networks, not schools, departments, or administrative organization. That Internet users search for people at a university independent of their interest in the university itself makes sense even if we may not have predicted the volume of this trend. Public affairs offices have long understood this—promoting the university through news stories about individual faculty. But an RNS such as Profiles (among others [
analysis of variance
research networking system
University of California, San Francisco
We thank Clay Johnston and Dan Lowenstein for providing inspiration and indispensable advice, David Arrington for facilitating the assessment of Profiles in the context of university websites, Isabel Allen for help with statistics, and Sarah Nelson for superb image production contributions. This publication was supported by NSF Award 1160906 and the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), through UCSF-CTSI Grant Number UL1 TR000004. The Profiles Research Networking Software is supported by NCATS NIH Award 1UL1 TR001102-01. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.
None declared.