To train staff to use MyGenAssist specifically for this activity, a training session was planned before initiating its use. The earlier-described operating mode was given to the pharmacovigilance officers. A training phase was also planned to verify their good understanding of the use of the tool. The operating procedure was also formalized for the use of potential new arrivals. To allow pharmacovigilance officers to report any difficulties or suggestions for improvement regarding the operating procedure, meetings were scheduled every other week until the termination of the study.

The study was conducted between January 2 and May 3, 2024. The time spent on each case was measured from the moment the pharmacovigilance case was acknowledged until the letter was sent and the Argus database was filled out.

Over an initial period of 9 weeks, the time spent on drafting each documentation letter, without the use of MyGenAssist, was measured. Over a second equivalent period, the time spent on the same task, but using the LLM as a drafting aid, was also measured. During the 2 periods, apart from the addition of MyGenAssist, no significant changes in the management of the activity took place: 3 workers, including 2 experimented participants and a newcomer were present throughout the 4 months of the study, and the process did not undergo any notable modifications.

This study was not considered as being related to human research in health, as it clearly focused on evaluating work performance with or without an AI-based tool. Regulations in France require such studies to be in compliance with the General Data Protection Regulation (ie, European Regulation 2016/679) [25]. The data used in this study were retrieved first during the normal course of the pharmacovigilance activities and reused for the study as authorized by General Data Protection Regulation [25] and after having received oral consent from pharmacovigilance workers who were informed of the objective of the study and how the data will be used.

The study included 122 cases (48 without the use of MyGenAssist and 74 with its use). The average time spent on each case was 19.05 (SD 6.05; 95% CI 17.97-20.12) minutes. The average time for cases handled was 22.25 (SD 7.62; 95% CI 20.09-24.41) minutes without the use of MyGenAssist and 16.97 (SD 3.50; 95% CI 16.17-17.77) minutes when the LLM was used (Figure 3). When all factors that potentially impact the time spent on the activity are taken into account in a multiple linear regression model, an average of 23.3% (95% CI 13.8%-32.8%; P<.001) of time savings were realized for each case thanks to MyGenAssist (adjusted R²=0.286; df=115; Figure 4). All explanatory variables of the linear regression model got a P value inferior to .05 apart from the number of questions added or removed by the pharmacovigilance officer (P=.05).

The hierarchical regression performed found that all the parameters used as explanatory variables were useful to get the best model (the lowest Akaike information criterion is obtained with all the parameters included in the model; Table 1).

According to the number of spontaneous pharmacovigilance reports for which a documentation request was necessary during the last 3 years, it can be estimated that the mean time saved on the activity with the use of the LLM could be between 66.27 (SD 22.79) and 108.70 (SD 37.23) hours per year, on the basis of an average 23.3% time saved on the average 22.25 (SD 7.62) minutes spent on a case without the use of MyGenAssist. Based on a working day of 8 hours, the use of MyGenAssist for this activity in the 3 previous years could have saved an average of 10.7 (SD 3.6) days of work per year (Table 2).

No statistical difference has been found regarding the answers’ rates after 1 attempt of contact, whether MyGenAssist was used or not: 31% (15/48) of letters produced without MyGenAssist were provided with an answer and 24% (18/74) when the generative AI was used for the writing (P=.40; χ²₁=0.7). The same situation occurred after 2 attempts: 42% (20/48) without the tool vs 36% (27/74; P=.57; χ²₁=0.3). In a subgroup analysis in function of the type of recipient, these results were consistent (Table 3).

In cases for which an answer is provided, the average time between the answer and the last attempt of contact does not differ significantly (Figure 5): 2.20 (SD 3.27; 95% CI 1.27-3.13) days when the letter was written without MyGenAssist and 2.65 (SD 3.30; 95% CI 1.90-3.41) days with its use (P=.64; t₄₄=–0.46).

Before the study, a 2-hour training was planned to present the use of MyGenAssist in the pharmacovigilance case documentation process to the pharmacovigilance officers. During the study, meetings were planned in the team on a 2-week basis. No difficulties in the use of MyGenAssist were identified, and users only suggested slight improvements in the prompt used to consider specific cases for which additional elements are required in the letter. An operating mode, detailing all the steps required to use MyGenAssist for the activity, was written to offer a reminder for the users if necessary.

In this study, we demonstrated that the implementation of MyGenAssist to the pharmacovigilance case documentation process provided an average of 23.3% time saving on the task without any modification in the rate of answers. Whatever the experience of the workers, the use of MyGenAssist induces time savings, although this effect is higher in nonexperimented ones. Moreover, while using MyGenAssist, it seems that the time spent on the different pharmacovigilance cases becomes less variable: when the use of the tool is added to the process, the range of time is decreased from 9-38 minutes to 10-25 minutes. MyGenAssist may have a more beneficial impact in cases for which more time is required. The easiest cases needing a slight amount of time to be handled would benefit less (or not) from its use, but the ones that are the more time-consuming would be realized significantly quicker than without it.

Therefore, the use of Bayer’s LLM improved the efficiency of this activity. That must save dozens of hours yearly for our local pharmacovigilance team, which could be dedicated to other activities with higher added value. A unique, short, and reachable training was sufficient to implement the use of MyGenAssist. This consisted of introducing the changes in the tool’s operating mode to the users. No additional training content was required to enable them to use the LLM in the activity, though they did not have any specific qualifications or skills in informatics. Hence, we showed in this study that making workers use an LLM could be easy, as far as they are assisted at the beginning, while other AI tools require important training to be used daily. However, the daily use of an LLM can make the user continually improve their abilities with this tool. Therefore, LLMs seem easy to integrate into several work environments and a way for the worker to get new skills without needing a large amount of time. Thanks to its ease, all users kept using the generative AI for their activity after the end of the study.

Although we included in the linear regression model all parameters that could have an influence on the time spent on the activity, in our point of view, the adjusted R² is low (0.286). Therefore, the most part of the time spent on each case could not be fully explained by the considered parameters. It seems that some aspects that are specific to each and difficult to transcribe in the model are important factors too. Moreover, a “human factor” could play a role here, as the user could spend a variable duration of time for the same task, in function of a lot of exterior parameters. Whatever explanatory variable was not included here, the average difference created by MyGenAssist (more than 5 minutes on a task requiring about 22 minutes without its use) seems too high to be questioned by these potential “hidden” parameters.

Nonetheless, 3 factors could not be included in the model. First, the experience of the user in the activity may directly influence the time spent on each case. It could also have an effect on the number of questions added or removed by the pharmacovigilance officer: while a newcomer could have difficulties selecting the required information to request from the recipient, an experienced one would more accurately determine which question is important enough to assess an adverse event. Second, electronic literacy, which is the capacity to understand or create messages using electronic means, may have an influence on the time spent on the activity. This impact exists for all cases because they are handled by using informatics, but must be stronger in the cases for which MyGenAssist would be used, as the ability of the user to handle technology could also impact their ease of familiarizing themselves with this new tool. By the time we started this experiment, we were not able to measure either user experience or user electronic literacy [26]. Even if we could not include them in the model, the variable “user” integrates both their experience and their electronic literacy: so, without deciphering which parameter has more impact, we still take them into account. Third, the complexity of the case, which may have an influence on the time spent on the case, is a subjective parameter that could not be measured accurately. The parameters we chose to include in this study were selected during a meeting with experienced pharmacovigilance officers, who were asked to give all factors that could influence the time spent on each case. It seems that no study so far, to the best of our knowledge, has been published reporting the parameters influencing the time spent to document a pharmacovigilance case. Moreover, no comparison of our internal process and the ones of other pharmaceutical companies could be done, as none of them seems to be available publicly. The European GVPs [17] do not define a strict process for the documentation of pharmacovigilance cases and let the market authorization holders organize the activity. Pharmaceutical companies have to create a Pharmacovigilance System Master File that describes the handling of all pharmacovigilance activities. It has to be sent to the national and European health authorities but is not publicly disclosed [27]. Our process could then have specificities (eg, the implication of both a case evaluator and a pharmacovigilance officer) that are not possible to detect and could create a lack of reproducibility of our study.

Other ways could have been explored to determine the effectiveness of generative AI. In this study, we chose to use the fulfillment of the objective of our task as the main criteria of effectiveness: to get the necessary information to analyze the pharmacovigilance case. From our perspective, other criteria did not seem relevant in this context. Moreover, we made the choice not to analyze the quality of the drafts written by MyGenAssist because it was clear in our operating mode and in the training that a human assessment of the letter was mandatory before sending it out. To get a relevant analysis of the effectiveness, recipients were not informed that the letter they received was first written with the help of an LLM: on the one hand, the recipient could have had the impression that its report was “automatically” handled, which could have encouraged this latter not to answer. On the other hand, the potential curiosity created by an eventual mention of generative AI, in a current era in which all eyes are turned to such tools, could have created a bias by making the recipient more willing to answer.

While our study was realized over a 4-month period, this latter only includes 122 cases. Even if it was enough to notice a difference in the time spent in function of the use of the tool, this has to be taken into account, particularly while analyzing the answers’ rate. However, this study is a first step before the extension of the use of the LLM in the pharmacovigilance case documentation process to all Bayer local pharmacovigilance teams worldwide. The efficiency improvement noticed by the French pharmacovigilance team could give fresh impetus to make other teams adopt the same principle. This will give us the opportunity to retrieve more data concerning our question.

Other publications regard the use of LLMs in pharmacovigilance: the use of ChatGPT without any optimization or fine-tuning of the public does not seem to fulfill the objective fixed, whatever this latter is providing answers to pharmacovigilance-related questions [28,29] or assess the risk of mortality of patients with toxic epidermal necrolysis [30]. On the other hand, another LLM, Bidirectional Encoder Representations from Transformers, was used with success for the extraction of adverse events from notes [31,32] or social media [33], but it always implied a step of fine-tuning that could not be handled by a simple user. Therefore, we could not find in the literature any study describing the successful use of ChatGPT or a strict equivalent in pharmacovigilance without the necessity to fine-tune the LLM after its use.

Moreover, the need to explore the possibilities that AI could offer to pharmacovigilance was highlighted by several publications before, particularly with the idea that the implementation of such tools could make both time and cost savings and provide an efficiency improvement, especially as the number of declarations of adverse events has been sharply increasing in the last few years [34]. Gholap et al [35] and Danysz et al [36] evoke the possibility for pharmacovigilance workers to focus on high-level tasks thanks to these time savings on administrative or repetitive tasks. However, in all these papers, these statements remain at the step of hypothesis without any experimental assessment.

Regarding the potential benefits that AI could represent for pharmacovigilance, the TransCelerate initiative, which includes 19 pharmaceutical companies, aimed to ask pharmacovigilance representatives of these companies which steps of the pharmacovigilance case processing might be the best suited for a pharmacovigilance implementation. In 2 surveys set between 2018 [37] and 2021 [38], the pharmacovigilance workers considered the pharmacovigilance case documentation process as one of the more appropriate steps for an AI implementation. The automation opportunity score, which combines the effort realized by companies to include AI in the task and the benefit or risk assessment regarding this inclusion, was 4.5/5 in 2021. The percentage of companies that set an AI-based process in this task rose from 19% to 35% between 2020 and 2021. Nonetheless, no information was given in these publications about a possible efficiency increase after an AI implementation.

Hence, our study may fulfill what seems to be, to the best of our knowledge, a lack in the medical literature. Descriptions of LLMs’ use cases for pharmacovigilance exist, but none of them include a measure of time saving realized thanks to this tool. A study showed that the use of ChatGPT for writing tasks enhanced the productivity of the workers by decreasing the time required by 40%, while the quality rose by 18% [39], but did not concern health. Hence, our objective was to determine if the integration of the LLM in the adverse event case documentation process can improve efficiency by reducing the time required for this GVP-related task without decreasing the response rate from the reporters.

These first encouraging results could be an incentive to implement MyGenAssist in other processes. In the pharmacovigilance field, some tasks fit well with the integration of MyGenAssist. For example, the comparison of different versions of the same procedure with this tool can provide a rapid insight into the modifications in order in a second step to assess their impact on other quality documents. Other use cases could be found in other departments, as the task given to the LLM, which is writing letters based on a template, seems pretty reproducible in other contexts for different objectives. The results of this study show that using LLMs in pharmaceutical activities, whatever the field, is relevant and can create improvements without losing quality concerning the regulations of good practice activities.

In this study, we showed the first example of a use case for a ChatGPT-based tool, MyGenAssist, in a pharmacovigilance industry department and assessed its efficiency over a 4-month period. An average of 23.3% of time savings was achieved thanks to this LLM, while its implementation did not modify the effectiveness of the task. It only required a short training to be set up. These results could be the first step to the largest use of LLMs in pharmaceutical activities.