The proposed augmented reality system is used as an extension to a standard Rectal Examination Trainer (Mk2, Limbs & Things Inc) benchtop model (Figure 1, [10]). The model is a semirealistic representation of the buttocks, anus, and rectum, allowing for the practice of diagnostic skills associated with rectal examination. It includes additional rectal examination perineum, which contains two rectal pathologies (polyp and carcinoma). For prostate examination, five interchangeable prostates are provided: normal, bilateral benign, unilateral benign, bilateral carcinoma, and unilateral carcinoma. The model can be used for both digital examination of prostate and rectum, as well as for the insertion and use of anoscope and proctoscope.

We used the HoloLens (version 1, Microsoft Inc) as an augmented reality head-mounted display (Figure 2). The HoloLens is immersive and see-through to help the user perceive the environment as realistic. It also enables interactions with holographic content. The HoloLens features an inertial measurement unit (accelerometer, gyroscope, and magnetometer), 4 environment understanding sensors (2 on each side), an energy-efficient depth camera with a 120°×120° angle of view, a 2.4-megapixel photographic video camera, a 4-microphone array, and an ambient light sensor [11].

HoloLens has been used in various medical simulations; VimedixAR (CAE Inc) was the first ultrasound simulator to integrate a HoloLens [12]; with this system, health care professionals manipulate representations of realistic anatomical parts and view the ultrasound beam in real time as it passes through human anatomy. With CAE LucinaAR, clinical learners can view 3D holograms of a fetus, as it descends the birth canal, and learn to manage complex deliveries [13]. HoloPatient (Microsoft Inc) is a mixed-reality learning tool for nursing, allied health, and medical schools that delivers simulated patient experiences [14]. Learning Heart (Spheregen) is a HoloLens application that assists students in understanding the physiology of the heart [15].

The 3D software in our system was developed using the Unity games engine (version 2018.3.7f1; Unity Technologies) [16] cross-platform authoring tool for creating 3D content.

To track and show the movement of the examining finger inside the benchtop model during the examination, we used a Trakstar magnetic tracking system (Northern Digital Inc) to obtain the position and orientation (pose) of the examining finger in real time, due to its ability to operate without line-of-sight [17]. It consists of a midrange magnetic field transmitter and a 6 degrees-of-freedom receiver (model 180), and it was connected to an electronics unit for amplification and digitization. The 6 degrees-of-freedom sensor has a position accuracy of 1.40 mm RMS and orientation accuracy of 0.50° RMS. The combination of the transmitter and the receiver allows tracking within a 30×40×30 cm³ zone, which is large enough for tracking the examining finger inside the benchtop model. The sensor was attached to the finger using thin tape (Figure 3, [17]). Sensor data were read at 40 Hz by the computer via an API using a previously developed C++ plug-in [18]. Once read, sensor data was transferred to the HoloLens via Wi-Fi. Position and orientation were transformed into the real-world coordinate system with the help of Vuforia Engine (version 8.1.7) [19]. The transformation was achieved by using the HoloLens to track the pose of the image target in world coordinates (Figure 4). This transformation enabled synchronization and overlay of the virtual finger on the tracked real finger, and for the result to be seen through the HoloLens (Figure 5).

To visualize the internal components of the benchtop model and to overlay the relevant virtual anatomy, it was necessary to first align the virtual model (Figure 6) with the physical benchtop model. For this purpose, the iterative closest point algorithm was used [20]; it takes the position of 7 anatomical landmarks as an input (Figure 7) and yields a transformation matrix. This matrix is used to rotate and translate the virtual benchtop model and virtual internal organs to align with the physical benchtop model (Figure 8). Once aligned, the user can visualize the movement of the examining finger and the model by directly looking at the model with the HoloLens (Figure 9).

A FingerTPS force sensor [21] was also attached to the examining finger to estimate the pressure applied to the prostate during palpation. The force sensor is flat and thin and can be worn under a surgical glove (Figure 10, [21]). The force sensor data were transferred to the HoloLens using the same process as for the pose information from the tracking sensor. A force visualizer, represented as a color bar with 3 regions showing different levels of pressure applied to the prostate gland, was displayed to provide real-time feedback (Figure 11).

Labels were used for displaying relevant information regarding internal anatomy inside the benchtop model; however, since there were several anatomical structures inside a small area, this made it difficult to label the anatomy in an effective way (in a way that avoids overlap with other labels or with the anatomy). Also, since our system allowed users to look at the anatomy from different perspectives, labels needed to be dynamically positioned. To address these issues, we implemented a view management system, capable of resolving occlusion among labels. Our labeling system used the labeling technique suggested by Tatzgern [22] (overcoming occlusion by limiting update rates, ie, not continuously moving the labels to separate them, but only when they occlude each other) combined with the theory by Hartzmann [23] to ensure that the labels were always near the object and that the lines from labels to objects did not cross each other. To achieve these criteria, the following steps are iteratively applied until all occlusions are resolved: (1) Place the label near the referenced object on the surface of the sphere, centered at the center of mass of the benchtop model. (2) Iterate through all labels to find all those that may be occluded by another label, establishing the side of the occlusion. (3) Move the occluded label to the opposite side to resolve the occlusion.

The labels were created using a ToolTip component of Unity, provided by the Microsoft Mixed Reality Toolkit (version 2.0.0 RC2) [24]. This component can rotate the label to always face the user, making it more readable (Figure 12).

Our augmented reality system incorporated step-by-step guidance to help trainees follow the correct steps and trajectory during the examination. Steps were extracted from the cognitive task analysis study performed by Low-Beer et al [25]. The system automatically tracked the position of the finger inside the rectal canal and checked whether the trainee had correctly followed the step. The user interface was designed to be readable and to require the least possible head movement to see all the contents (Figure 13), despite the narrow field of view of the HoloLens display.

In addition to step-by-step guidance, our augmented reality system also allowed recording and playback of the examination. The pose of the examining finger and the benchtop model were recorded so that, when played back, the virtual examining finger and the virtual benchtop model could be accurately displayed on the HoloLens. This feature is useful trainees as they can observe experts repeatedly, or they can analyze their own performance. It can also be used by an examiner to assess student performance from different angles.

The accuracy of the model alignment system was evaluated by performing an alignment task five times by an experienced user. After each alignment task was performed, the positions of each landmark on both the physical and virtual benchtop models were then measured using the magnetic tracking sensor. The error of the alignment system was then calculated as a distance between a point on the virtual model and a corresponding point on the physical model.

An initial validation study was conducted. Clinicians (n=9; 6 men, 3 women; mean 37.8, SD 5.4 years of age) who routinely perform DRE, and medical students (n=10; 7 men, 3 women; mean 21.8, SD 2.4 years of age) were recruited. The study was approved by the National Health Service Patient Safety Agency Research Ethics Committee (09/H0701/68). Before the study, a consent form was signed by each participant. During the study, participants were asked to perform DREs on the benchtop model with the augmented reality system, wearing both the position tracking sensor and the pressure sensor under a standard surgical glove. They were also asked to pay attention to the information displayed on the HoloLens, such as the force bar and the guidance panel. Once finished, they were asked to watch the recorded performance. Afterward, participants were asked to complete an online questionnaire using a 5-point Likert scale from 1 (definitely disagree) to 5 (definitely agree) regarding the usefulness of the system (Table 1).

The results showed that medical students and clinicians were interested in the system and recognized the value of visualization of DRE, performance recording, and step-by-step guidance in improving the learning and teaching of DRE skills. Medical students could visualize the movement of the finger and the pressure applied to the prostate. This visualization and sensor data, combined with the step-by-step guidance, allowed them to receive feedback in real time while performing DRE. Using the record-and-playback feature, students could not only rewatch examinations performed by an expert, but examiners could also review student performance from multiple viewing angles.

Most users reported that the HoloLens and the tracking sensors were comfortable to wear. Even with a narrow field of view, the user interface did not appear cluttered, facilitating navigation through the menu and visualization of the performance, while at the same time facilitating step-by-step guidance on the left-hand side of the display. The model alignment was also perceived as accurate which demonstrated the use of the iterative closest point algorithm, and the readings from the electromagnetic positioning sensor were valid.

User experience was improved by displaying all information on a head-mounted augmented reality display that also allowed the overlaying of the virtual finger on the real finger. This colocation allowed trainees to avoid having to change visual focus from the benchtop model to a separate display.

When comparing the feedback from medical students to the feedback from clinicians, it was observed that the mean ratings from medical students were higher than those from clinicians for all questions, with an average difference of 20%. A likely explanation is that experienced clinicians had higher expectations, and they were comparing the quality of the simulation with that of real cases because of their experience performing DREs. Having less experience, students would be more likely to require visualization and guidance, while clinicians would not because tactile feedback was adequate, given their level of experience. In addition, it may also reflect the increased acceptability and interest in this type of technology by younger generations.

While the proposed system was demonstrated to be beneficial, it has some limitations. Wearing and adjusting the HoloLens properly can take some time and require assistance. In addition, operating and interacting with the HoloLens by using gestures such as pinching, also requires practice. Regarding the sensors, properly wearing the tracking and pressure sensors is crucial, and at the moment, requires the presence of an assistant to be properly placed on the examining finger. In terms of implementation, the current alignment between the virtual and the physical benchtop model was done manually through 7 landmarks; however, automatic alignment would be faster, more accurate, and more convenient. With respect to features, a scoring system with real-time feedback would be valuable for teaching and assessing DRE skills. Apart from these, the benchtop model itself was reported to be much stiffer than real patients and generally unrealistic with limited anatomical landmarks.

This paper presents an augmented reality system for teaching and learning DRE that can be used with widely available benchtop models. It was designed to assist both trainees and teachers in learning, teaching, and practicing DRE, as well as to allow examiners to assess student technique. With colocation of the virtual and real models, students only need to focus on the benchtop model to visualize all important information. Even though the results from the user study are positive, further research is needed to evaluate the system. This would include more robust quantitative analysis with a larger number of participants and varying levels of experience. The augmented reality system could be improved by using the second version of the HoloLens which offers a larger field of view and resolution, and it could also be improved by using a haptic-based instead of standard benchtop model so that a wider variety of prostate glands and abnormalities may be generated and used during the examination. Such a model would also be able to directly track finger movement and estimate pressure applied without the need for external sensors. Finally, formative and summative assessment of DRE performance will be an important component of the next version of our system.