Finalize project

Code/python/notebooks/predict.py

@@ -45,8 +45,8 @@ file_list = load_file_list(file_list_path)
 
 base_ds = dataloader.EvenlySpacedDataset(
     filepaths=file_list,
-    n_input=warm,
+    n_input=warm //step,
-    n_output=pred,
+    n_output=pred //step,
     n_windows_per_file=5,
     step=step,
     feature_columns=("lat", "lon", "alt", "ias"),

report/main.tex

@@ -59,7 +59,13 @@
 \newpage
 
 \begin{abstract}
-    TODO
+Accurate prediction of aircraft trajectories is a key component of modern air traffic management, with direct implications for safety, efficiency, and airspace capacity. Traditional trajectory prediction methods are largely based on kinematic models and rule-based assumptions, which struggle to capture the complex, history-dependent dynamics observed in real flight operations. In this work, we investigate a data-driven approach to aircraft trajectory prediction using recurrent neural networks (RNNs) trained on large-scale Automatic Dependent Surveillance--Broadcast (ADS-B) data.
+
+We formulate trajectory forecasting as a sequential learning problem and evaluate multiple recurrent architectures, including standard RNNs, Long Short-Term Memory (LSTM) networks, and Gated Recurrent Units (GRUs), in an offset many-to-many configuration. To ensure physically meaningful optimization, we introduce a geometrically motivated loss function based on the Haversine distance, augmented by a separately weighted altitude error term. Aircraft type and registration information are incorporated through a compact latent encoding learned via an autoencoder and combined with kinematic and temporal features.
+
+Using a curated dataset of nearly \num{29000} commercial flight trajectories over major Scandinavian routes, we perform an extensive hyperparameter study to assess the impact of architectural choices, temporal window sizes, and loss weighting. The results show that gated recurrent models significantly outperform vanilla RNNs, with LSTMs providing the most stable performance across configurations. Optimal predictions are obtained for initialization horizons of approximately \qty{900}{\second} and short prediction horizons of up to \qty{150}{\second}, achieving a minimum validation loss of \qty{5.32}{\km}. Notably, prediction accuracy degrades gradually for longer horizons, indicating robust learned representations of aircraft motion.
+
+These findings demonstrate that recurrent neural networks, when combined with physically informed loss functions and appropriate preprocessing, offer a powerful and flexible framework for short- to medium-term aircraft trajectory prediction using publicly available surveillance data.
 \end{abstract}
 
 \maketitle
@@ -134,7 +140,7 @@ The LSTM augments the recurrent state with a persistent cell state $\mathbf{c}_t
 \mathbf{c}_t &= \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t, \\
 \mathbf{h}_t &= \mathbf{o}_t \odot \tanh\!\left( \mathbf{c}_t \right),
 \end{align}
-where $\sigma(\cdot)$ denotes the logistic sigmoid function and $\odot$ the element-wise product. The additive update of the cell state enables stable gradient propagation across long time horizons. In the offset many-to-many setting, the initialization sequence shapes $(\mathbf{h}_t,\mathbf{c}_t)$ before predictions are generated and evaluated. A conceptual illustration of the LSTM architecture is provided in Fig.~\ref{fig:lstm_architecture}.
+where $\sigma(\cdot)$ denotes the logistic sigmoid function and $\odot$ the element-wise product. The additive update of the cell state enables stable gradient propagation across long time horizons. In the offset many-to-many setting, the initialization sequence shapes $(\mathbf{h}_t,\mathbf{c}_t)$ before predictions are generated and evaluated. A conceptual illustration of the LSTM architecture is provided in \cref{fig:lstm_architecture}.
 
 \subsubsection{Gated Recurrent Units (GRU)}
 \begin{figure}
@@ -150,7 +156,7 @@ The GRU simplifies the LSTM architecture by merging the cell and hidden states a
 \tilde{\mathbf{h}}_t &= \tanh\!\left( \mathbf{W}_h \mathbf{x}_t + \mathbf{U}_h \left( \mathbf{r}_t \odot \mathbf{h}_{t-1} \right) + \mathbf{b}_h \right), \\
 \mathbf{h}_t &= (1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t .
 \end{align}
-By directly interpolating between the previous hidden state and a candidate update, the GRU achieves a balance between memory retention and adaptability with fewer parameters than the LSTM. As with the LSTM, the GRU is used here in an offset many-to-many configuration, with an initial conditioning phase followed by a prediction phase where outputs are produced and the loss is computed. An overview of the GRU architecture is shown in Fig.~\ref{fig:gru_architecture}.
+By directly interpolating between the previous hidden state and a candidate update, the GRU achieves a balance between memory retention and adaptability with fewer parameters than the LSTM. As with the LSTM, the GRU is used here in an offset many-to-many configuration, with an initial conditioning phase followed by a prediction phase where outputs are produced and the loss is computed. An overview of the GRU architecture is shown in \cref{fig:gru_architecture}.
 
 
 \subsection{Model Architecture}
@@ -222,22 +228,128 @@ For rapid prototyping and to overcome writer's block during the writing of this
 
 \section{Results} \label{sec:results}
 \subsection{Dataset Overview}
+In a first study of the dataset, we analyze the distribution of flight trajectories, aircraft types, and registration countries. The dataset comprises a total of \num{28796} flight trajectories, with a diverse range of aircraft types represented. In \cref{fig:trajectory_distribution}, we present a heatmap illustrating the density of flight trajectories across the selected Scandinavian routes.
+\begin{figure}
+  \centering
+  \includegraphics[width=\columnwidth]{include/air_traffic_density_over_norway.png}
+  \caption{Heatmap of flight trajectory density across selected Scandinavian routes.}
+  \label{fig:trajectory_distribution}  
+\end{figure}
+
+We observe clear patterns in the flight paths, with higher densities along the straight routes between major airports. On the other hand issues due to missing data points and signal loss in the ADS-B data also become apparent, particularly in regions with low population density or challenging terrain. This behavior is mainly attributed to the community-based nature of ADS-B data collection, where reception quality can vary significantly based on the distribution of ground stations and environmental factors. While urban areas benefit from a higher density of receivers, rural and remote regions often suffer from sparse coverage, leading to gaps in the recorded trajectories. These data quality issues highlight the importance of robust preprocessing and filtering techniques to ensure the reliability of the dataset for trajectory prediction tasks. On the other hand we need to note, that low signal strength does not lead to erroneous data points, but rather to missing data points, as the positioning information is collected by the sender aircraft itself via GPS. As the data is transmitted digitally, interference or low signal strength will either lead to a complete loss of the message or a correct reception, but not to corrupted data.
+
+Furthermore it becomes apparent that there is still a non-negligible spread in the actual flight paths taken by different aircraft on the same route. This variability can be attributed to factors such as weather conditions, air traffic control directives, and individual pilot preferences. Understanding and modeling this spread is crucial for developing accurate trajectory prediction models that can account for the inherent uncertainties in real-world flight operations.
+\Cref{fig:trajectory_distribution_oslo} shows a zoomed-in view of the flight trajectories around Oslo Gardermoen Airport (ENGM), highlighting the complexity of approach and departure patterns in the vicinity of a major airport.
+\begin{figure}
+  \centering
+  \includegraphics[width=\columnwidth]{include/air_traffic_density_over_oslo.png}
+  \caption{Zoomed-in heatmap of flight trajectory density around Oslo Gardermoen Airport (ENGM).}
+  \label{fig:trajectory_distribution_oslo}  
+\end{figure}
+
 
 \subsection{Hyperparameter Optimization}
+To evaluate the performance of different RNN architectures and hyperparameter configurations, we conduct a systematic hyperparameter optimization study. We explore various combinations of RNN types (LSTM and GRU), number of layers, hidden state sizes, RNN sizes and altitude loss weightings. The models are trained on the preprocessed dataset, and their performance is assessed using the Haversine distance-loss function described earlier. For each hyperparameter scan we define a grid of possible values and train models for each combination. The loss is then evaluated in pair wise comparisons with the defintion
+\begin{equation} \label{eq:pairwise-loss}
+  \mathcal{L}(A,B) = \min \mathcal{L}_\mathrm{validation}(A,B, \theta) \forall \theta \in \Theta ,
+\end{equation}
+where $A$ and $B$ are two different hyperparameter configurations, $\mathcal{L}_\mathrm{validation}$ is the validation loss after training, and $\Theta$ is the set of all other hyperparameter configurations. This approach allows us to identify the best-performing model configurations based on their relative performance across the validation set.
+
+All results obtained in this section are featured in \cref{app:hyperparameter_scan} in the appendix. In the following, we summarize the key findings from the hyperparameter optimization study.
+
+In a first run we compare different combinations of the following hyperparameters:
+\begin{enumerate}
+  \item RNN Type: RNN, LSTM, GRU
+  \item Number of Nodes per Layer in the MLP: 16, 32, 64
+  \item Number of Nodes in the RNN: 32, 64, 128
+  \item Step Size between consecutive Data Points: 1, 5, 10, 30
+\end{enumerate}
+The largest decline in accuracy is observed when using a simple RNN instead of the more advanced LSTM or GRU architectures. This behavior is expected, as standard RNNs are known to struggle with capturing long-term dependencies in sequential data due to vanishing gradients. Both LSTM and GRU models perform comparably well, with slight variations depending on the specific configuration of other hyperparameters. Increasing the number of nodes per layer in the RNN generally leads to improved performance, as it allows the model to learn more complex representations of the input data. However, this improvement comes at the cost of increased computational complexity and training time. When increasing the nodes in the MLPs, the converse effect is observed, as the increased complexity does not seem to add significant value to the model's ability to capture the relevant features from the input data. The step size between consecutive data points also plays a crucial role in model performance. Smaller step sizes (i.e., more frequent data points) provide the model with finer temporal resolution, which can enhance its ability to learn intricate trajectory patterns. However, this also increases the amount of data the model must process, potentially leading to vanishing influence of earlier time steps. The minimum validaton loss is observed for a step size of 10, we choose the LSTM architecture with 64 nodes in the RNN and 16 nodes in the MLPs for further experiments.
+
+In a second study, we investigate the impact of the number of hidden layers in the RNN and the I-MLPs, as well as a dropout probability to mitigate overfitting. The hidden layers in the I-MLPs are varied between 0 and 4 while the RNN hidden layers are varied between 1 and 5. The dropout rate is varied between \qty{0}{\percent} and \qty{15}{\percent} in steps of \qty{5}{\percent}. The results indicate that increasing the number of hidden layers does not significantly enhance model performance and may lead to vanishing gradients, particularly in deeper RNN configurations. The optimal configuration is found to be 0 hidden layers in the I-MLPs and 2 hidden layers in the RNN, with no dropout applied. We choose no dropout as the training and validation losses do not indicate overfitting in the current setup.
+
+Lastly, we manipulate the weighting factor $\alpha$ in the Haversine distance-loss function to assess its influence on model performance. Simulatenously, we vary the initialization horizon and prediction horizon, in ranges of \numrange{300}{1800} seconds and \numrange{150}{900} seconds, respectively. The results show that a higher weighting on altitude errors of \qty{1e-3}{\km \per \square \feet} leads to the best overall performance, as it encourages the model to prioritize accurate altitude predictions alongside horizontal positioning. The optimal initialization horizon is found to be \qty{900}{\second}, providing sufficient context for the model to learn from past trajectory data. The best prediction horizon is determined to be \qty{150}{\second}, which is to be expected as the uncertainty in trajectory predictions generally increases with longer forecast periods. Most importantly it becomes apparent, that longer initialization horizons do not necessarily lead to better performance, as the additional information may not be relevant for the immediate future trajectory.
+
 
 \subsection{Window-Size dependent Accuracy}
+Instead of training separate models for different prediction horizons, we investigate the performance of a single model across varying initialization and prediction horizons. This approach allows us to assess the model's generalization capabilities and its ability to adapt to different temporal contexts. We train a single LSTM model with the previously identified optimal hyperparameters and evaluate its performance across a range of initialization horizons (from \qty{300}{\second} to \qty{1800}{\second}) and prediction horizons (from \qty{50}{\second} to \qty{1800}{\second}).
+As expected the best performance is observed around the previously identified optimal horizons of \qty{900}{\second} for initialization and \qty{150}{\second} for prediction. The model's accuracy decreases as the prediction horizon increases, reflecting the inherent uncertainty in forecasting longer-term trajectories. This decrease in accuracy is on the other hand quite moderate, indicating that the model is capable of maintaining reasonable performance even for extended prediction periods. The influcence of the initialization horizon is more pronunced, with a sharp drop in accuracy for initializations for more than \qty{1200}{\second}. This behavior suggests that while additional historical context can be beneficial, there is a threshold beyond which the relevance of past data diminishes, potentially due to changes in flight dynamics or external factors not captured in the longer history. Overall, these results highlight the model's robustness and its potential applicability in real-world scenarios where varying temporal contexts are encountered. Especially the moderate decline in accuracy for longer prediction horizons is promising for practical applications in air traffic management and trajectory forecasting. The best obtainable validation loss for this study is \qty{5.32}{\km} for an initialization horizon of \qty{900}{\second} and a prediction horizon of \qty{50}{\second}.
+\begin{figure}
+  \centering
+  \includegraphics[width=\columnwidth]{include/contour_plot_mode4.pdf}
+  \caption{Contour plot of the validation loss as a function of initialization and prediction horizons for a single LSTM model.}
+\end{figure}
 
-\subsection{Error Analysis}
 
 \section{Conclusion} \label{sec:conclusion}
 \subsection{Conclusion}
+In this work, we investigated the use of recurrent neural networks for short- to medium-term aircraft trajectory prediction based on publicly available ADS-B data. By framing trajectory forecasting as a sequential learning problem and leveraging gated RNN architectures, we demonstrated that data-driven models can capture nontrivial spatio-temporal structure in real-world flight trajectories beyond what is achievable with simple kinematic extrapolation.
+
+A systematic comparison of architectures confirmed that gated models, specifically LSTMs and GRUs, substantially outperform vanilla RNNs, consistent with their ability to retain relevant temporal information over extended horizons. While both LSTM and GRU architectures exhibited comparable predictive performance, the LSTM showed slightly more stable behavior across hyperparameter settings and was therefore selected for further analysis. The hyperparameter studies further revealed that model capacity is primarily driven by the recurrent layers rather than the input MLPs, and that excessively deep recurrent stacks or long initialization windows can be detrimental due to diminishing relevance of historical context.
+
+A key contribution of this study is the use of a geometrically motivated loss function based on the Haversine distance, augmented by a separately weighted altitude error term. This formulation respects the spherical geometry of the Earth while allowing altitude deviations to be penalized in a physically meaningful way. The results indicate that an appropriate balance between horizontal and vertical errors is crucial for stable training and improved predictive accuracy, particularly in terminal flight phases where altitude dynamics are most pronounced.
+
+The trained model achieves its best performance for initialization horizons of approximately \qty{900}{\second} and short prediction horizons of \qty{50}{\second} to \qty{150}{\second}, with a minimum validation loss of \qty{5.32}{\km}. Importantly, the degradation in accuracy for longer prediction horizons is gradual rather than catastrophic, suggesting that the learned representations capture robust aspects of aircraft motion. This robustness, combined with the model’s ability to generalize across varying window sizes without retraining, makes the approach well suited for practical air traffic management applications such as conflict detection, short-term flow prediction, and trajectory-based operations.
+
+Several limitations remain. The study assumes clean and reliable ADS-B data and does not explicitly address missing data, adversarial manipulation, or uncertainty quantification in the predictions. Moreover, external factors such as weather, wind fields, and air traffic control interventions are only implicitly encoded through historical trajectories rather than being modeled explicitly. Incorporating such information, along with probabilistic or ensemble-based prediction methods, represents a natural direction for future work.
+
+Overall, this work demonstrates that recurrent neural networks, combined with physically informed loss functions and careful preprocessing, provide a viable and flexible framework for aircraft trajectory prediction. As richer datasets and contextual information become available, such models have the potential to play a central role in next-generation, data-driven air traffic management systems.
 
 
 \subsection{Potential Future Work}
+While the results presented in this study demonstrate the viability of recurrent neural networks for aircraft trajectory prediction, several directions remain open for further improvement and extension.
+
+A first and essential step is the expansion toward a more general and diverse dataset. The present analysis focuses on selected Scandinavian routes and a limited set of commercial aircraft types, which simplifies the learning problem but restricts generalization. Extending the dataset to include a wider range of airspaces, flight phases, aircraft categories, and traffic densities would allow for a more comprehensive assessment of model robustness and scalability. Such an extension would also enable the investigation of regional and procedural differences in air traffic operations.
+
+Further gains are expected from more systematic feature engineering. While the current model primarily relies on kinematic and basic contextual inputs, additional temporal and operational features could provide valuable information. Examples include explicit encoding of time-of-day, day-of-week, seasonal effects, and airline-specific identifiers, which may capture operational preferences, scheduling constraints, or standardized procedures. These features could help the model distinguish between structurally different trajectory patterns that are otherwise similar in purely geometric terms.
+
+Incorporating meteorological information represents another critical avenue for improvement. Weather effects such as wind speed and direction, temperature, pressure, and convective activity are among the dominant external drivers of trajectory deviations. Integrating gridded weather data or outputs from numerical weather prediction models would allow the network to explicitly account for environmental forcing, thereby improving prediction accuracy, particularly for longer horizons and in adverse conditions.
+
+The inclusion of information about nearby aircraft is also a promising direction. Aircraft trajectories are not independent, especially in dense airspace or during approach and departure phases, where separation constraints and air traffic control interventions play a major role. Modeling local traffic context—such as relative positions, velocities, and headings of surrounding aircraft—could enable the prediction model to capture interaction effects and collective traffic dynamics that are currently only indirectly reflected in the data.
+
+From an architectural standpoint, attention-based models offer a compelling alternative or complement to recurrent networks. Attention mechanisms can enable the model to selectively focus on the most relevant parts of the input history or contextual information, potentially alleviating the performance degradation observed for long initialization horizons. Hybrid architectures combining recurrent layers with temporal or spatial attention, or fully transformer-based approaches, may provide improved scalability and interpretability.
+
+Finally, a stronger focus on data quality and robustness is essential for operational relevance. Although ADS-B data are generally reliable, issues such as missing messages, uneven spatial coverage, and intentional or unintentional data corruption remain significant challenges. Future work should investigate methods to explicitly handle such imperfections, including anomaly detection, confidence-aware training, data imputation strategies, and robustness-enhancing loss functions. Addressing these issues is a prerequisite for deploying learning-based trajectory prediction models in safety-critical air traffic management environments.
 
 
 \onecolumngrid
 
 \bibliography{include/airtraffic}
 
+\appendix
+\section{Hyperparameter Scan Results}
+\label{app:hyperparameter_scan}
+
+This appendix presents the complete results of the hyperparameter optimization studies summarized in the main text. The figures referenced below provide a visual overview of validation performance across the explored configurations. All plots are based on validation losses obtained after training each model to convergence on the same preprocessed dataset, using the Haversine distance--loss function defined earlier.
+
+\Cref{fig:hyperparameter_scan_mode1} shows the results of the initial hyperparameter scan comparing different RNN architectures and core model sizes. The figure summarizes pairwise validation-loss comparisons for simple RNN, LSTM, and GRU architectures, as well as different RNN hidden-state sizes, MLP widths, and temporal step sizes. Each data point corresponds to the minimum validation loss achieved for a given configuration, marginalized over all remaining hyperparameters according to the definition in \cref{eq:pairwise-loss}. The plot was generated by evaluating the full hyperparameter grid and aggregating the best-performing validation losses for direct comparison.
+
+\begin{figure}[htbp]
+  \centering
+  \includegraphics[width=\linewidth]{include/hyperparameter_scan_results.pdf}
+  \caption{Initial hyperparameter scan comparing RNN type, RNN size, MLP width, and temporal step size.}
+  \label{fig:hyperparameter_scan_mode1}
+\end{figure}
+
+The second study, visualized in \cref{fig:hyperparameter_scan_mode2}, focuses on architectural depth and regularization effects. Validation losses are shown for varying numbers of hidden layers in the RNN and the I-MLPs, as well as different dropout probabilities. This plot was produced by fixing the best-performing configuration from the first scan and systematically varying only depth and dropout. The reported losses correspond to the minimum validation loss observed for each setting, enabling a direct assessment of stability and diminishing returns in deeper architectures.
+
+\begin{figure}[htbp]
+  \centering
+  \includegraphics[width=\linewidth]{include/hyperparameter_scan_results_mode2.pdf}
+  \caption{Hyperparameter scan over RNN and I-MLP depth and dropout probability.}
+  \label{fig:hyperparameter_scan_mode2}
+\end{figure}
+
+Finally, \cref{fig:hyperparameter_scan_mode3} presents the results of the scan investigating the influence of the altitude-loss weighting factor, initialization horizon, and prediction horizon. The figure displays the minimum validation loss obtained across all tested combinations for each parameter value. It was generated by jointly varying the loss weighting and temporal horizons and selecting the best-performing configuration for each parameter according to the pairwise comparison criterion.
+
+\begin{figure}[htbp]
+  \centering
+  \includegraphics[width=\linewidth]{include/hyperparameter_scan_results_mode3_min.pdf}
+  \caption{Validation-loss minima as a function of altitude-loss weighting, initialization horizon, and prediction horizon.}
+  \label{fig:hyperparameter_scan_mode3}
+\end{figure}
+
+Taken together, \cref{fig:hyperparameter_scan_mode1,fig:hyperparameter_scan_mode2,fig:hyperparameter_scan_mode3} provide the empirical basis for the hyperparameter choices adopted in the final model configuration used throughout this work.
+
+
 \end{document}