Design and Evaluation of Sparse Matrix Factorization Algorithms for Big Data

Thesis Abstract

The rapid growth of high-dimensional data arising from network analytics, recommender systems, and scientific simulations has intensified the need for scalable and accurate sparse matrix factorization (SMF) techniques that can operate effectively on Big Data without compromising interpretability. This study addresses the problem of developing, implementing, and evaluating robust SMF algorithms that deliver fast convergence, scalable memory usage, and high predictive accuracy for large, sparse datasets typical of industrial and scientific workloads. The aim is to design an integrated SMF framework that (i) adapts to heterogeneous sparsity patterns, (ii) leverages parallel and distributed computing architectures, and (iii) provides principled model selection and uncertainty quantification. Specific objectives include (a) devising novel regularization strategies that balance sparsity and reconstruction error; (b) integrating stochastic optimization and alternating minimization with boundary-aware update rules to enhance convergence on extremely sparse matrices; (c) implementing a modular software prototype capable of running on multi-core CPU and GPU environments; (d) benchmarking against state-of-the-art SMF methods on diverse data regimes; and (e) providing a reproducible evaluation protocol including synthetic and real-world datasets. A mixed-methods research design is adopted, combining algorithmic development with empirical evaluation. The population comprises publicly available sparse matrices from the SuiteSparse Matrix Collection and synthetic benchmarks designed to simulate real-world sparsity patterns observed in user-item interaction data and signal processing applications. A stratified sampling approach yields a training set of 2,500,000 nonzero entries across 100 matrices for empirical benchmarking, with a held-out test set consisting of 20% of entries to assess generalization. The data collection instruments include a modular SMF software stack implemented in Python and CUDA-enabled kernels, along with standardized datasets (e.g., MovieLens-20M, KONECT, and synthetic low-rank plus sparse-noise matrices). Validity and reliability are ensured through unit tests, cross-validation on multiple folds, and reproducibility via containerized environments (Docker) and fixed random seeds. Analytical methods comprise a combination of optimization theory and empirical performance evaluation. Convergence behavior is analyzed using empirical convergence curves and theoretical guarantees under assumptions of restricted isometry and incoherence where applicable. Regularization effects are examined via ablation studies, with performance metrics including root mean squared error (RMSE), mean absolute error (MAE), cosine similarity of factor matrices, and sparsity level. Statistical comparisons employ paired t-tests and Wilcoxon signed-rank tests to assess significance of improvements over baseline methods such as alternating least squares with L1/L2 penalties and nonnegative SMF variants. Model selection employs information-theoretic criteria (AIC, BIC) and cross-validated predictive likelihood. Dimensionality reduction implications are assessed through explained variance and recovery of latent factors. The framework integrates a Bayesian perspective for uncertainty quantification using variational Bayes on factor matrices to capture posterior uncertainty in sparse reconstructions. Expected findings indicate that the proposed boundary-aware stochastic alternating minimization with adaptive regularization achieves faster convergence by 30–45% relative to strong baselines, reduces memory footprint by up to 40% on graphs with power-law degree distributions, and improves test RMSE by 5–12% on real-world datasets while maintaining interpretable factor sparsity. The study anticipates demonstrating robustness across data regimes, including extremely sparse matrices with density below 0.01 and moderately dense matrices where low-rank structure persists. A comparative analysis will reveal regimes where the integrated SMF framework outperforms existing approaches in scalability, accuracy, and reproducibility. The study contributes to knowledge by presenting a scalable, theory-informed SMF framework with modular software support, comprehensive benchmarking protocols, and empirically validated guidelines for selecting sparsity-inducing regularizations and optimization strategies in Big Data contexts. It advances practical understanding of how sparsity, data heterogeneity, and computational architecture interact within SMF, and it informs the design of recommender systems, signal processing pipelines, and large-scale graph analytics. The main conclusion posits that a hybrid optimization approach, augmented with adaptive regularization and uncertainty quantification, yields superior performance for sparse matrix factorization in Big Data environments, with concrete recommendations for deployment on multi-core and GPU-backed systems. Recommendations include adopting boundary-aware updates, incorporating Bayesian uncertainty estimates in decision-critical applications, and adopting standardized, reproducible evaluation pipelines for future SMF research.