This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Seeking medical information can be an issue for physicians. In the specific context of medical practice, chatbots are hypothesized to present additional value for providing information quickly, particularly as far as drug risk minimization measures are concerned.
This qualitative study aimed to elicit physicians’ perceptions of a pilot version of a chatbot used in the context of drug information and risk minimization measures.
General practitioners and specialists were recruited across France to participate in individual semistructured interviews. Interviews were recorded, transcribed, and analyzed using a horizontal thematic analysis approach.
Eight general practitioners and 2 specialists participated. The tone and ergonomics of the pilot version were appreciated by physicians. However, all participants emphasized the importance of getting exhaustive, trustworthy answers when interacting with a chatbot.
The chatbot was perceived as a useful and innovative tool that could easily be integrated into routine medical practice and could help health professionals when seeking information on drug and risk minimization measures.
A conversational agent, also known as a “chatbot,” is an artificial system that can converse with a human user through automated message exchange [
To improve drug information access and awareness of drug risk minimization measures among physicians, a pilot version of a chatbot was developed and tested. The chatbot was expected to improve drug information access and awareness of drug risk minimization measures. This qualitative study aimed to improve the chatbot to meet physicians’ needs and expectations. User participation [
The study focused on the pilot version of a text-based chatbot developed by an international pharmaceutical company. This was qualitative research that employed individual, in-depth, semistructured interviews to explore physicians’ opinions and perceptions of a chatbot. The aim of the chatbot was to provide quick, 24-hour access to information on drugs for physicians to improve their medical practice and the application of recommendations regarding drug risk minimization measures among patients. Thus, all information provided by the chatbot was sourced from official regulatory documents issued by French health authorities to ensure patient security. The chatbot was accessible online through a web browser. When logged in, it displayed a dialog box on the right side of the screen and a female figure in the central area. Answers provided by the chatbot followed pre-established topics related to drugs (eg, treatment initiation, discontinuing treatment, treatment renewal, documentation or information that can be downloaded and obtained by the user). When asked a question, the chatbot provided a direct answer based on its database. When a question was not sufficiently focused, the chatbot proposed between 2 and 4 categories related to the question. The user then had to choose between those categories by typing the corresponding number. When the chatbot was asked a question that was not available in its database or not understood because of a typing error, the user was alerted. In this latter case, the user had to reformulate the question or ask a new question related to pre-established topics. As a pilot version, the chatbot initially focused on one drug. Further information can be added and delivered when the final version is deployed; the chatbot is meant to be used by physicians who seek quick access to information in the presence of a patient or when preparing a consultation.
General practitioners were approached initially as the conversational agent was developed for this specific population. Specialists were also approached because some of their patients used drugs subject to risk minimization measures.
Participants were recruited on a voluntary basis after being asked by email whether they were interested in “taking part in a project that will test an innovative digital health tool.” Their email addresses were available from a database of health practitioners who had already participated in previous digital epidemiological studies. The interviewer had not met the participants before the interview. General practitioners were recruited to represent various situations such as urban and rural general practice. Specialists were recruited from either public or private practice. Thematic saturation was not taken into account since the study aimed to improve the chatbot through physicians’ needs and expectations in an explorative way without being representative.
Semistructured interviews were conducted during June 2018 and July 2018 and took place primarily in participants’ medical offices. At the beginning of the interview, each participant received a brief summary of the study from the interviewer, and sociodemographic data (gender, age, profession, and workplace) were collected. Interviews were scheduled for approximatively 1 hour.
To standardize the interviews, they were divided into 3 sections: introduction, introduction of the chatbot, and conclusion.
The introduction included knowledge about the use of drugs and recommendations about risk minimization measures and acceptability of a chatbot within their daily practice.
The chatbot was presented through the example of 3 preselected input phrases based on various situations: “How to accompany a pregnant woman taking this medicine?”, “I want to renew the prescription of a patient.”, “Does this drug decrease the impact of contraception?” Test sessions were conducted as follows. After the introduction, the interviewer logged in to the chatbot with a confidential password. In the dialog box, introductory text was displayed and read aloud by the interviewer to explain the purpose of the chatbot. Then, preselected input phrases were first inputted by the interviewer to show how the chatbot worked. Participants were then asked to try the chatbot by searching for information on drugs as they would do during a real consultation, using their own terminology. Questions were collected by the interviewer after a physician concluded the test, but participants were left free to use the conversational agent as much as they wanted during the rest of the interview. Since it was a pilot version, participants did not have access to the conversational agent before or after the interview. Moreover, the test was different from a real-case scenario since participants were not totally autonomous in using the chatbot and had to share their experience with the interviewer.
During the conclusion, feedback was collected on the user experience with the chatbot. Physicians were asked about the relevance and appreciation of the tested tool as well as their needs for and expectations of the chatbot.
The interviewer followed a semistructured interview template composed of the following themes: (1) knowledge about drug risk minimization measures and acceptability of digital health within their daily practice (Section 1), (2) relevance and appreciation of the chatbot to facilitate information acquisition (Section 3), and (3) needs for and expectations of the tested chatbot (Section 3).
Interviews were recorded using a digital audio recorder after receiving participant agreement. They were fully transcribed and anonymized by the interviewer. Transcripts were organized, sorted, and coded using a systematic thematic analysis approach [
A total of 10 health professionals participated in individual, in-depth, semistructured interviews in France.
The sample size included 8 general practitioners and 2 specialists. The average age of the participants was 51.5 years. There were 7 men and 3 women; 6 participants worked in an urban area only, 3 worked in a rural area only, and 1 worked in both urban and rural areas. All participants were following patients who needed routine drug prescriptions including risk minimization measures.
All participants reported being vigilant about new recommendations for prescription including risk minimization measures before using the chatbot. Seven physicians reported that they sought information on drugs subject to risk minimization measures only when their patients were directly concerned, when they were confronted with a particular case, or when adverse events had been reported to them:
I have access to information obviously, but I get more interested when I am confronted with the problem.
They considered the information on drug risk minimization measures easy to find but fragmented. The main source of information they reported was the internet, followed by information from health authorities (letters, email), databases on drugs, medical journals, and communications from laboratories (press, visits). Exchanges with other practitioners had also been cited as sources of information:
The easiest and most accessible information is the summary of characteristic products, but it is in rather indigestible to read since it’s really big.
Generally, I do research spontaneously. I’m looking out on the internet when I have a question about a treatment or when I’m not sure about a question a patient asked me, we can look together.
I get information from National Agency for Medicines and Health Products Safety, from health authorities, and from laboratories as well. I receive either emails or letters.
Several themes and subthemes emerged from the interviews. The central themes were (1) achievement by man including ergonomics and format of asking; (2) achievement by tool including design, tone, and form of presentation; (3) content of output including the amount of information, clarity, and accuracy; and (4) user needs. These topics are discussed with illustrative quotes in the following paragraphs.
In total, 52 questions were asked to the chatbot by participants. Of the 52 questions formulated by the physicians (
Some physicians judged the ergonomics favorably due to simplicity and ease of use:
This is correct. The style is sober, quiet; it is not aggressive. It is simple and readable.
Yes, it is really easy to use. The ergonomics are really good.
Over half (6/10) of the participants found that the chatbot lacked intuitiveness and fluidity to access information:
The answer is clear, but I don’t know, there is something with the fluidity... If I compare with where I’m usually looking for information, there is one section for each topic, and I get instant access. Here, the answers are too robotic.
I would say this is not intuitive. Everyone knows how to use a computer. But the ergonomics... I don’t know, there is just a login page, then the home screen. If it is just that, there is no ergonomics. It would be nice to have a website plan.
The size of the dialog box was also cited as improvable:
There is a part of the screen that is not used; the dialog box looks really small compared to the rest. It could occupy a larger space, which would allow a bigger typeface.
The window to type our questions is too small. If you enter a long sentence, you can’t see the beginning of your sentence, which is bothering. Indeed, the window is too small and the typeface as well.
Most of the participants reported that the colors and icon of the chatbot were too simple, but the global design was judged serious and professional, which was appreciated:
You could give us the possibility to customize the design by choosing between 3 or 4 different colors. Maybe it should be a bit more cheerful. It looks seriously made, but a little too dark.
I like the font; it is quite nice. It looks as if everything was made to be relatively neutral. This is good because the tool can’t be visually aggressive during a consultation. This is not really attractive; it is quite neutral, but this is adapted because visual elements catching your eyes all the time can be tiring. This is better.
Seven participants reported that the tone of the answers was positive. However, the display of the answers provided by the conversational agent needed to be more concise according to half (5/10) of the physicians:
It speaks from a medical point of view so there is no problem with the tone.
It needs a display with bold characters so we can reach key points by reading through it diagonally
The response delay from the chatbot was acceptable for 9 of 10 participants, but 5 physicians were not satisfied with the length of the information, which was too broad to read or to access:
It is a little bit long because every time it asked me for a clarification, it seems that it doesn’t understand.
This is accurate but it is too much talking. Two lines should be enough; this is too long in my opinion.
Even though all participants found the information provided by the chatbot very clear, the main issue pointed out by physicians was the accuracy of the answers. Obtained answers did not provide a sufficient amount of information. Thus, even if they trusted the provided information, it was considered as too generic and not relevant enough:
We need clear and concise scientific information. We need to know globally what to do; then, we translate the information but at least it needs to be precise. Because for now, it is a little bit general; this is too elementary.
The answers are understandable; it comes out quickly; it comes out clear. The only inconvenience is that it is not the answer I was looking for.
It is centered on the developers’ insight, not from a physician’s insight. This is almost a tool for a patient actually. Physicians have much more questions than the chatbot can answer for now.
Participants identified pros and cons of the chatbot after they tested it (
Pros and cons of the chatbot.
| Observation and examples | Example excerpts | |
|
|
|
|
|
|
Relevant concept: give fast and precise information | “In my opinion, the concept is really interesting. Why? Because this is provisioning a tool in which we can get precise and fast information. This is a concept that seems very interesting and very relevant to me.” [Man, in practice for 35 years] |
|
|
Ergonomic tool: can be utilized as a research tool | “The ergonomics, the conciseness…Yes, the shape. You call this a conversational agent; to me, it is a good research engine.” [Man, in practice for 11 years] |
|
|
Diagnosis helping: |
“The fact that it can give answers when we have a question is interesting. That doesn’t mean we’ll necessarily follow the answer but at least it gives a direction. Tools that can help medical care or to diagnose, I find great.” [Woman, in practice for 24 years] |
|
|
Time-saving | “There are only benefits anyway. This is instantaneous, verified information. It saves us time. A consultation is time-consuming; we need to get immediate answers. It has already happened to me that I had to call a laboratory to get an answer, and I was told that someone would call me back. Here, I get my answer almost directly” [Man, in practice for 30 years] |
|
|
|
|
|
|
Natural language comprehension issues: can be difficult to access the right information | “The tree structure to get to the information. That is to say… If I get some information, in what degree is it intuitive? Does it need to be questioned in a precise way, or can I ask my questions without thinking about the formulation? Can it answer my everyday needs?” [Man, in practice for 18 years] |
|
|
Medical data issue: can be dangerous for medical data security | “Of course, there is hacking. I can’t take control of a doctor, but I can get control of a conversational agent without problems. All the security part, as powerful it can be, can always be breached if someone intends to do it. That’s the limit.” [Man, in practice for 23 years] |
|
|
Threatens health care professionals: a chatbot could replace doctors | “Let’s say that, in 10 years, we are told that our job will be replaced; it is a nightmare. We studied for 10 years, not 2 years, but 10 years. If we are told that we will be replaced by artificial intelligence, it does not make anyone dream.” [Man, in practice for 23 years] |
Nine participants estimated that this tool would be easily integrated in their routine practice when fully developed, especially if it could provide fast, trustful, easily accessible information on a range of medical topics:
It is simple; it is practical. When it has acquired more vocabulary, I think it will be an efficient tool.
This is a tool in which we can get precise and fast information. This is a concept that seems very interesting and very relevant to me”
However, half (5/10) of the participants considered that the chatbot did not meet their needs. The main reason was the natural language comprehension issues:
It needs to be accurate and well-edocumented because, during a consultation and in our daily routine, we don’t have much time. Not having enough time means we need to get to the point; so, we need fast, specific information”
If I get information, what is the degree of intuitiveness? The machine has to adapt, because when I’m doing my job with a patient in front of me, I can’t be focused on how I should formulate my question to the chatbot.
During a consultation, everything has to be done quickly, I need information quickly. If the chatbot doesn’t understand me, I put it aside to get the information elsewhere.
Participants reported areas to improve the chatbot prototype (
Display and appearance
Highlight important information in the answers
Possibility to choose interface colors
Content
Provide [printed] information to the patients
Regular updates
Extensions of use
Ask questions orally
Integrate more themes (eg, drug interactions, pathologies)
Declare side effects
The study was conducted to elicit physicians’ perceptions of a chatbot that is meant to improve drug information access and awareness of drug risk minimization measures. A qualitative approach was chosen to collect detailed data on how the chatbot was used and perceived by physicians on a small scale and needs and expectations for the chatbot. As a pilot version, the chatbot did not meet physicians’ expectations.
Overall, the findings demonstrated that physicians’ needs towards information delivered by the chatbot were the reliability of sources, precise information, and speed of access. This is consistent with previous research demonstrating that all these features are critical to physicians when seeking information [
The chatbot format was also appreciated, as it could easily be integrated into physicians’ routine practice, either during or before a consultation with a patient. This was judged as innovative since most chatbots in health care are developed to provide information for patients [
It was pointed out that the conversational agent did not always respond to the questions they formulated, either because it did not recognize the everyday language employed by the users or because the answers, while based on official regulatory documents, were too broad without highlighting the most important parts of information. However, they were aware that the chatbot was still in an experimental stage. Regarding this fact, most practitioners were willing to use the conversational agent in its fully developed version if this tool remains easy to use, secure, and easily accessible.
Areas of improvement for the conversational agent proposed by medical practitioners included a better understanding of the questions formulated, highlighting the most important information, and better ergonomics.
This market research is not without limitations. As an exploratory study with a sample size of 10 medical practitioners, findings are not generalizable to the entire population of medical practitioners. This was a qualitative study conducted with semistructured interviews, which allowed us to explore tendencies and opinions on the usage of a conversational agent. The average age of our sample (51.5 years) was slightly above the average age of the medical practitioner population in France, which was 50.8 years in 2018 [
In addition, medical practitioners who were recruited had already participated in other digital epidemiological studies carried out by the same research team, and it is possible that they were more receptive to new health technology. In other words, while our sampling method was not meant to be representative, this specific market research may be prone to researcher bias.
Finally, because of the confidentiality policy, the drug associated with the chatbot could not be cited in the paper. However, we believe that it did not interfere with the results, which were meant to elicit physicians’ perceptions of the chatbot.
According to the results of this study, it appears that chatbots could be a solution for quick and easily accessible information. By reinforcing the knowledge on drugs based on official and institutional recommendations, a chatbot could be used to enhance compliance with the drug risk minimization measure within the physician population. In particular, this chatbot prototype was perceived by medical practitioners as a useful, acceptable, innovative tool that could easily be integrated in their daily medical practice. Finally, even though the chatbot prototype could not be used as it was because of insufficient information in the database, findings suggest that physicians are willing to use a chatbot not only for prescription but also to get information on drug interactions or to obtain assistance within medical care for complex pathology or disease management. A future challenge for the chatbot should be to accommodate physicians’ needs for accurate, concise information based on official regulatory documents that ensure patients’ security.
Questions tested by physicians.
Sanofi is the study funder.
SS, JK, NT, and AM (Kap-Code) received financial support from SANOFI SA for data collection and interpretation and for writing the manuscript. Kap-Code is a start-up in digital health, which provides solutions in chronic diseases and detection of pharmacovigilance signals on social networks. KF is employed by SANOFI SA.