Comparing Latent Dirichlet Allocation and Non-Negative Matrix Factorization for Topic Modeling of User Complaints in the Access by KAI Application

Valen Tino Rosi; Ari Muzakir; A. Haidar Mirza; Fitri Purwaningtias

doi:10.63922/ijevss.v5i03.5952

Valen Tino Rosi Bina Darma University, Indonesia Ari Muzakir
Ari Muzakir Bina Darma University, Indonesia Ari Muzakir
A. Haidar Mirza Bina Darma University, Indonesia Ari Muzakir
Fitri Purwaningtias Bina Darma University, Indonesia Ari Muzakir

DOI: https://doi.org/10.63922/ijevss.v5i03.5952

Keywords: Topic, Modeling, Latent, Dirichlet, Allocation, Non-Negative, Factorization, Short, Text, Topic Coherence

Abstract

User reviews on the Google Play Store hold valuable information about technical complaints, yet they arrive as unstructured short text that is impractical to examine manually. This study compares Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF) for modeling complaint topics in 51,770 one- and two-star reviews of the Access by KAI application. Both algorithms were trained over the range of two to ten topics on an identical preprocessing pipeline and were evaluated using two coherence metrics and Topic Diversity. Robustness was examined through four test scenarios covering preprocessing quality, random initialization, and topic granularity, and the resulting topics were validated by two independent annotators. NMF outperformed LDA across all nine topic-number scenarios and on both coherence metrics, reaching a mean coherence of 0.5853 against 0.5273 for LDA, while LDA retained higher Topic Diversity. The advantage held after the normalization dictionary was expanded from 144 to 18,099 entries and across five random seeds. Human validation produced a mean readability score of 3.63 for NMF against 3.50 for LDA, with a Cohen's Kappa of 0.750. A document-level cross-mapping showed that each LDA topic corresponds to a dominant NMF topic at an average rate of 67.0%, indicating that both algorithms recover the same underlying themes. The four-topic NMF model is therefore recommended for complaint categorization on Indonesian short text.

References

Albalawi, R., Yeap, T. H., & Benyoucef, M. (2020). Using topic modeling methods for short-text data: A comparative analysis. Frontiers in Artificial Intelligence, 3, 42. https://doi.org/10.3389/frai.2020.00042
Altman, D. G. (1991). Practical statistics for medical research. Chapman and Hall.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3, 993–1022.
Bouma, G. (2009). Normalized (pointwise) mutual information in collocation extraction. Proceedings of the Biennial GSCL Conference 2009, 31–40.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256
Dieng, A. B., Ruiz, F. J. R., & Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8, 439–453. https://doi.org/10.1162/tacl_a_00325
Dwi Cahya, F. D., Satyareni, D. H., & Farhan, A. (2025). Analisis kinerja aplikasi Access by KAI menggunakan metode End-User Computing Satisfaction dan DeLone & McLean. Jurnal Ilmiah Sistem Informasi dan Teknik Informatika, 8(2), 153–161.
Egger, R., & Yu, J. (2022). A topic modeling comparison between LDA, NMF, Top2Vec, and BERTopic to demystify Twitter posts. Frontiers in Sociology, 7, 886498. https://doi.org/10.3389/fsoc.2022.886498
Heri Suroyo, & Pratama, E. J. (2025). Comparison of text representation methods for sentiment analysis using support vector machine. Journal of Advances in Information and Industrial Technology, 7(1), 21–30.
Indrawan, H., Irawan, B., & Suprapti, T. (2023). Klasifikasi ulasan pengguna aplikasi Access by KAI berbasis aspek dengan algoritma Naïve Bayes dan SVM, 7(6).
Jelodar, H., Wang, Y., Yuan, C., Feng, X., Jiang, X., Li, Y., & Zhao, L. (2018). Latent Dirichlet Allocation (LDA) and topic modeling: Models, applications, a survey (arXiv:1711.04305). arXiv. https://doi.org/10.48550/arXiv.1711.04305
Khairunnisa, S., Adiwijaya, A., & Faraby, S. A. (2021). Pengaruh text preprocessing terhadap analisis sentimen komentar masyarakat pada media sosial Twitter. Jurnal Media Informatika Budidarma, 5(2), 406. https://doi.org/10.30865/mib.v5i2.2835
Kusumaningtyas, K., Dwijayanti, I., Rizqi Lahitani, A., & Habibi, M. (2024). Analisis tren topik dalam ulasan negatif aplikasi M-Banking menggunakan Latent Dirichlet Allocation. Jurnal Fasilkom, 14(3), 549–555.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
Lau, J. H., Newman, D., & Baldwin, T. (2014). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, 530–539.
Lee, D. D., & Seung, H. S. (2000). Algorithms for non-negative matrix factorization. Advances in Neural Information Processing Systems, 13, 556–562.
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031
Mustaqim, I. Z., & Suryono, R. R. (2025). A systematic literature review of topic modeling techniques in user reviews. Journal of Information Systems Engineering and Business Intelligence, 11(2), 238–253.
Nanyonga, A., Wasswa, H., & Wild, G. (2023). Topic modeling analysis of aviation accident reports: A comparative study between LDA and NMF models. 2023 3rd International Conference on Smart Generation Computing, Communication and Networking (SMART GENCON), 1–2.
Nugraha, M. T., Sulistiyowati, N., & Enri, U. (2023). Analisis sentimen ulasan aplikasi Satu Sehat pada Google Play Store menggunakan Naïve Bayes classifier, 7(5).
Pratiwi, M. D., & Tania, K. D. (2025). Knowledge discovery through topic modeling on GoPartner user reviews using BERTopic, LDA, and NMF. Journal of Applied Informatics and Computing, 9(1), 1–7.
Rosalinda, G., Santoso, R., & Kartikasari, P. (2023). Pemodelan topik ulasan aplikasi Netflix pada Google Play Store menggunakan Latent Dirichlet Allocation. Jurnal Gaussian, 11(4), 554–561.
Röder, M., Both, A., & Hinneburg, A. (2015). Exploring the space of topic coherence measures. Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, 399–408. https://doi.org/10.1145/2684822.2685324
Zoya, Latif, S., Shafait, F., & Latif, R. (2021). Analyzing LDA and NMF topic models for Urdu tweets via automatic labeling. IEEE Access, 9, 127531–127547. https://doi.org/10.1109/ACCESS.2021.3112620