The tumor motion data (69 treatment fractions of 21 patients) used in this study were obtained from an open data set, which was recorded by the CyberKnife Synchrony (Accuray Incorporated) tracking system with a recorded sampling rate of 25 Hz [24]. To analyze the external movements of patients, charge-coupled device cameras were used to monitor the luminous diodes located on a patient's abdomen and chest. To analyze internal fiducial positions, orthogonal diagnostic x-ray systems were used to observe implanted markers periodically.

The general scheme for the prediction process of 2 models is outlined in Figure 1, and the arrangement of the respiratory signals that were used for network training and validation is shown in Table 1. Each recorded position (internal tumor and external marker positions) was stratified into 2 cohorts based on time t_s. The positions prior to time t_s (the training signals) were used to train the TCN and LSTM models. The positions after t_s (the testing signals) were used to evaluate the developed model.

For the training process, the training input data and prediction target data were first used to tune the hyperparameters, which was done by using a cross-validation model. Afterward, they were used to train the model. The external markers’ positions during the first input period of the training process (ie, the time between t=1 and t=t_delay) were used as the training input data for predicting the tumor positions (target positions) at a specific time frame (t=t_delay+t_ahead). This training process was repeated and continued to predict the next tumor position until either the threshold of the cost function or the maximum iteration number, which was set in advance, was reached. Each pair of data points (ie, the input data, M[t+1,…, t+t_delay], vs the output data, T[t+t_delay+t_ahead]) consisted of a training data set. “M” denoted 3 external markers’ positions (M1, M2, and M3), which were based on 3 directions (the superior-inferior, anterior-posterior, and left-right directions). t_ahead represented the ahead time we needed for making predictions.

For the evaluation process, the testing signals, which were represented as M(t_s+1, t_s+2,…, t_end) and T(t_s+1, t_s+2,…, t_s+ t_end), were used to evaluate the developed model. Similar to the process implemented in the training process, the external markers’ positions during the first input period of the evaluation process (ie, the time between t=1 and t=t_delay) were used to predict a tumor’s position (T’[t_s+t_delay+t_ahead]) at a specific time (t=t_s+t_delay+t_ahead). This process was also repeated to predict the next tumor position continuously. The external signals that were recorded during radiation therapy (ie, the time between t=t_end−t_delay−t_ahead+1 and t=t_end−t_ahead) were used to predict the final tumor position (T’[t_end]). Finally, the predicted signals (T’[t_s+t_delay+t_ahead],…, T’[t_end]) were compared to the real tumor positions (T[t_s+t_delay+t_ahead],…, T[t_end]).

The recurrent neural network (RNN) is a particular type of neural network that allows for self-cycle connections and transmits parameters across different time stamps. An RNN model can store the information of former time stamps. However, it is difficult for the RNN to memorize long-term memory information due to vanishing and exploding gradients [25-27].

The LSTM layer is a special RNN layer that overcomes the weakness that the RNN has with memorizing long-term memory information [26,28]. Figure 2 shows an LSTM unit. Unlike the simple RNN unit, the LSTM unit has a memory cell state c_t at time t. The information that passes through state c_t is controlled by the following three gates: the input gate (i_t), the forget gate (f_t), and the output gate (o_t). The input gate is used to control input data that flow into state c_t, the hidden state connection (h_t) is used to control the forgetting of state c_t, and the output gate is used to moderate the output data that flow from state c_t. A plurality of LSTM layers can be stacked in a deeper neural network, which can fit the data of the complicated functions that are required to analyze the inputs and the targets.

The TCN model was based on a transformation of a 1D fully convolutional network that was used for sequential prediction problems. The TCN model used a multilayer network to learn information over a long time span. Sequence information were transmitted layer by layer across the network until prediction results were obtained. The architecture of the TCN model is illustrated in Figure 3 [23], in which x₁, x₂,…, x_T are the original sequence signals (inputs), and are the prediction signals (outputs). The obvious characteristics of the TCN model, which were compared to those of the normal 1D fully convolutional network model, were as follows:

The input of the TCN model was interval sampled. The equation for the dilated convolution was as follows:

In equation 1, d is the dilation factor (sampling rate). A d value of 1 in the lowest layer meant that every signal was sampled, whereas a d value of 2 in the middle layer meant that every 2 respiratory signals were sampled.

Residual networks [29], which are shown in Figure 3, were imported in this study to accelerate convergence and stabilize training. A residual block that included a branch was used to make a series of transformations (F). Afterward, the outputs of the residual block (ie, F[X_residual]) were added to the input (ie, X_residual), as follows:

O_residual = Activation(X_residual + F[X_residual]) (2)

With regard to the TCN model, previous TCN studies [20-23] reported (in the Instruction section) using the same TCN architecture and only sometimes varying the number of layers (n) and the filter size. Hence, we tested these two hyperparameters and used a dilation factor (d) of 2ⁿ for layer n. Moreover, the number of neurons in the input layer and the learning rate of the TCN model were also investigated in this study. For the LSTM model, the number of LSTM layers, learning rate, number of hidden units per layer, and number of neurons in the input layer were investigated. Furthermore, the Adam algorithm was used as the optimization algorithm for both the TCN model and LSTM model. The Kingma and Ba [30] study demonstrated that the hyperparameters in the Adam model required little tuning. Goodfellow et al [31] also approved of the robustness of the Adam model for their hyperparameter of choice and provided advice on how to tune the learning rate from the default value. Hence, we used the good default settings that were tested by Kingma and Ba [30] as the hyperparameters of the Adam optimizer and tuned the learning rate. The default settings were exponential decay rates of 0.9 and 0.999 and a decay exponent of 10⁻⁸. In this study, all hyperparameters were tuned synthetically by using a grid search model. It should be noted that we tested the hyperparameters in a 4D hyperparameter space instead of a subspace (ie, while a parameter was investigated, others were fixed) to maintain the accuracy of hyperparameter tuning.

The respiratory signals from 69 treatment fractions of 21 patients with cancer who were treated with the CyberKnife Synchrony (Accuray Incorporated) device were used to evaluate the proposed model. Of the 69 treatment fractions, 5 were used to tune the hyperparameters. The rest of the patients were used to evaluate prediction performance. For each of the 69 treatment fractions, signals that were acquired around the first 3 minutes (4500 data points) were used as the training signals for training the prediction model, and signals from the following 30 seconds were used as the test signals for assessing the effectiveness of the proposed model. The ahead time (t_ahead) used in this study was 400 ms [1,5].

The root mean square errors (RMSEs) between real and predicted signals of respiratory motion in a 3D space were used for assessment [6,7]. The RMSEs for motion in each direction (RMSE_{SI, LR, AP}) and motion in a 3D space (RMSE_3D) were calculated by using equations 3 and 4, respectively, as follows:

In equation 5, is the average of the true values, and is the average of predicted values. Time point t in equation 3 ranged from t_start (t_s+t_delay+t_ahead) to t_end. The Wilcoxon signed-rank test was used as the statistical model for evaluating the differences between true values and predicted values.

A TCN model for predicting respiratory motion by using external markers’ prior signals was developed and tested in this study. The experiment demonstrated that the TCN model’s performance in predicting future respiratory signals with a 400-ms ahead time was better than that of the LSTM model.

As is well known, hyperparameter settings have a large influence on the prediction performance of machine learning models. This also holds true for our TCN and LSTM models. We tuned 4 major hyperparameters for both of the TCN and LSTM models. Among these hyperparameters, the number of neurons in the input layer and the learning rate were tested for both models. Having a large number of neurons in the input layer allows for the inclusion of more features in models. Obviously, useful features may increase prediction accuracy. However, redundancy features may also be brought in along with the useful features. Hence, if this hyperparameter is too large, prediction performance may degenerate. The best number of neurons in the input layer for the TCN and LSTM models in this study was 15 and 20, respectively. The learning rate was an important hyperparameter in the model optimization process. If the learning rate is too large, the model may oscillate around the global minimum value instead of achieving convergence. On the other hand, if this value is too small, the training time and the risk of overfitting increase. Learning rates of 0.001 and 0.01 were selected as the optimal hyperparameters of the TCN and LSTM models, respectively. In addition to the two abovementioned hyperparameters, the number of layers and filter sizes were also investigated for the TCN model, whereas the number of LSTM layers and number of hidden units per layer were tested for the LSTM model. With regard to the TCN model, the size of the effective window (receptive field) increased as the number of layers and filter size increased. Hence, these two hyperparameters should guarantee that the receptive field of TCN model covers enough context for respiratory signal prediction. The optimal values for these two hyperparameters in our experiments were 5 and 9, respectively. With regard to the LSTM model, on one hand, a deeper LSTM model (a large number of LSTM layers) may be representative of a more complicated relationship among respiratory signals and improve prediction performance. On the other hand, a deeper LSTM model also has an increased risk of overfitting and increased convergence speed. In this study, the prediction performance results of the LSTM model were comparable when the number of LSTM layers was over 2. Hence, we selected 2 as the optimal number of LSTM layers. Further, the number of hidden units per layer determined the width of each LSTM layer. We also found that having a large number of hidden units per layer was helpful for establishing a more complicated prediction model, but at the same time, this increased the risk of overfitting and convergence speed.

The effect that different numbers of external markers had on prediction performance was also investigated in this study. The TCN model had the best prediction performance when it used all three markers’ positions. As shown in Figure 6, the TCN model’s prediction performance when using 3 markers was more robust than when using 1 marker or 2 markers. For most treatment fractions, the RMSEs of the TCN model using 3 markers was slightly smaller than those obtained by using 1 marker or 2 markers. However, for some treatment fractions, such as treatment fractions 7 and 11, the RMSEs of predictions based on 1 or 2 external markers were quite larger than those of predictions based on 3 external markers. This was probably because having more external markers for different skin surface positions resulted in the inclusion of more useful features. Such useful features may alleviate the overfitting and underfitting problems.

Finally, we studied the influence of the different components (the filter size and residual blocks) in the TCN model. The size of the effective window (receptive field) increased with filter size. Hence, the model’s prediction performance initially became better as the filter size increased. However, the model’s prediction performance only slightly improved as the filter size increased continually. This may be because the receptive field that resulted from using a filter size of 3 provided enough context for the respiratory signal prediction task. On the other hand, we observed that the residual block architecture enhanced the model’s prediction performance immensely. We believe that this was because the residual blocks effectively allowed the TCN model to be modified based on identity mapping instead of a full transformation, which was crucial for the deep neural network architecture.

A deep learning approach based on the TCN architecture was developed to predict internal tumor positions with a 400-ms ahead time based on the external markers’ positions in this study. The results demonstrated that this model could predict tumor positions accurately. Further, the prediction performance of the TCN model using multiple external markers was more robust and positive than that of the TCN model using 1 or 2 external markers.