Discrepancies between Star Ratings and Textual Sentiment in MOOC Reviews and Their Pedagogical Implications

Wiputra Cendana; Nur Indah Eriyathi; Wiginy Kusliawan; David Habsara Hareva; Khoe Yao Tung; Ranny Fransiska

doi:10.63922/ijevss.v5i04.7125

Wiputra Cendana Universitas Pelita Harapan
Nur Indah Eriyathi Universitas Pelita Harapan
Wiginy Kusliawan Universitas Pelita Harapan
David Habsara Hareva Universitas Pelita Harapan
Khoe Yao Tung Universitas Pelita Harapan
Ranny Fransiska Universitas Pelita Harapan

DOI: https://doi.org/10.63922/ijevss.v5i04.7125

Keywords: MOOC reviews; star ratings; human annotation; coding agreement; pedagogical criticism; learning analytics

Abstract

Star ratings summarize course evaluations but may not represent textual sentiment or criticism relevant to instructional improvement. This exploratory secondary study examines 107,018 MOOC reviews and two sets of human annotations for 120 reviews, with 40 reviews in each rating stratum: 1–2, 3, and 4–5 stars. Exact text matching, cross-tabulation, percentage agreement, and Cohen’s kappa were used without changing annotators’ labels. The corpus contained 100,038 distinct texts; 54 groups of identical texts had conflicting sentiment categories derived from ratings. Agreement between Annotators A and B was 94.17% for sentiment (κ = 0.9018), 87.50% for primary pedagogical aspect (κ = 0.8316), and 100% for the criticism indicator (κ = 1.0000). Agreement with rating proxies was only 70.83% for A and 69.17% for B. Among 40 three-star reviews, only 12 and 8 were labeled neutral by A and B, respectively. Sixty-six reviews contained criticism, including three of the 40 reviews with four or five stars. Textual evidence concerned assessment alignment, learning support, and access to materials. Label agreement establishes neither indicator validity nor intervention effectiveness. Findings are descriptive for a sample with an undocumented selection mechanism; educational decisions require review of textual evidence and course context.

References

Awadh, W. A., Sulaiman, R. B., & Mahmoud, M. A. (2025). Aspect-based sentiment analysis in MOOCs: A systematic literature review introducing the MASC-MEF framework. Journal of King Saud University Computer and Information Sciences, 37, Article 2. https://doi.org/10.1007/s44443-025-00018-1
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 347–364. https://doi.org/10.1007/BF00138871
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Jørnø, R. L., & Gynther, K. (2018). What constitutes an “actionable insight” in learning analytics? Journal of Learning Analytics, 5(3), 198–221. https://doi.org/10.18608/jla.2018.53.13
Li, L., Johnson, J., Aarhus, W., & Shah, D. (2022). Key factors in MOOC pedagogy based on NLP sentiment analysis of learner reviews: What makes a hit. Computers & Education, 176, Article 104354. https://doi.org/10.1016/j.compedu.2021.104354
O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19. https://doi.org/10.1177/1609406919899220
Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85(3), 257–268. https://doi.org/10.1093/ptj/85.3.257
Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4