Comparative Analysis of Machine Learning Models for Predicting Cybersecurity Breaches

Thesis Abstract

The escalating frequency and sophistication of cybersecurity breaches pose a significant threat to organizational assets, data integrity, and stakeholder trust, necessitating the development of robust predictive models to enhance preemptive security measures. This study aims to conduct a comprehensive comparative analysis of machine learning models for predicting cybersecurity breaches, with a focus on identifying the most accurate and reliable algorithms to inform cybersecurity strategies. The specific objectives include evaluating the predictive performance of various machine learning algorithms—including Random Forest, Support Vector Machine (SVM), Gradient Boosting, Neural Networks, and Logistic Regression—under different data conditions, assessing their applicability across diverse organizational contexts, and determining the features that contribute most significantly to accurate breach prediction. Employing a quantitative research design, the study collected dataset samples from 15 organizations spanning??, healthcare, and public sector domains, encompassing a total of 25,000 recorded cybersecurity incidents over a five-year period. Stratified random sampling was used to select 10,000 breach-related entries, ensuring representation of different breach types and organizational sizes. Data were obtained through collaboration with the organizations’ cybersecurity departments, supplemented by publicly available cybersecurity datasets such as the CERT insider threat datasets and the CIC-IDS2017 intrusion detection dataset. The study utilized structured data collection instruments, including breach incident logs, network traffic data, and system vulnerability reports, ensuring data validity through expert validation and reliability through test-retest procedures. To analyze the data, the study employed extensive preprocessing techniques, including normalization, feature extraction, and handling of missing values. The performance of each machine learning model was evaluated using key metrics such as accuracy, precision, recall, F1-score, Area Under the Receiver Operating Characteristic Curve (AUC-ROC), and Matthews Correlation Coefficient (MCC). Comparative analyses were conducted using repeated k-fold cross-validation (with k=10) and statistical significance testing through ANOVA and post-hoc pairwise comparisons to determine the models' relative effectiveness. Model calibration was also assessed to evaluate the reliability of probabilistic predictions. Additionally, feature importance analysis was performed using SHAP (SHapley Additive exPlanations) values to interpret model outputs and identify the most influential variables contributing to breach predictions. Expected findings suggest that ensemble-based models like Random Forest and Gradient Boosting are expected to outperform simpler classifiers such as Logistic Regression and Neural Networks in terms of predictive accuracy and stability, especially in heterogeneous datasets. The study anticipates revealing critical features—such as network traffic anomalies, user account activities, and system vulnerability scores—that significantly influence breach prediction accuracy. Findings aim to demonstrate that model performance varies according to organizational context, breach type, and data quality, emphasizing the need for tailored cybersecurity models. The study’s contribution to knowledge lies in providing empirical evidence on the comparative effectiveness of multiple machine learning algorithms in cybersecurity breach prediction, filling existing gaps concerning contextual performance and feature impact analysis. It advances theoretical understanding by integrating the Theory of Information Security Threats and the Adaptive Security Framework to explain model behavior under different threat scenarios. Practically, the research offers actionable insights for cybersecurity practitioners and decision-makers by identifying optimal predictive models adaptable across sectors and providing guidelines for feature selection and model deployment. The main conclusion underscores that ensemble machine learning models, particularly Random Forest and Gradient Boosting, offer superior predictive capabilities, but their effectiveness depends on the quality and relevance of input features. It is recommended that organizations adopt a hybrid approach—integrating multiple models and continuously updating datasets—to enhance breach detection accuracy. Future research should explore the integration of real-time data streams, develop adaptive models capable of evolving with emerging threats, and investigate the application of explainable AI techniques to foster trust and transparency in predictive cybersecurity systems.