Analysis of the Use of ChatGPT and DeepSeek in Solving HOTS Physics Problems in Kinematics Teaching Material

Nadya Aidilla Fitri Tanjung; Sahyar Sahyar

doi:10.63922/ijevss.v5i04.6962

Nadya Aidilla Fitri Tanjung Universitas Negeri Medan, Indonesia
Sahyar Sahyar Universitas Negeri Medan, Indonesia

DOI: https://doi.org/10.63922/ijevss.v5i04.6962

Keywords: ChatGPT; DeepSeek; HOTS; Kinematics; Accuracy; Consistency.

Abstract

This study analyzes the ability of ChatGPT 5.1 and DeepSeek V3 to solve Higher Order Thinking Skills (HOTS) problems in kinematics, in terms of answer accuracy and response consistency. A descriptive qualitative design with a desk-research approach was used. The sample consisted of 20 expert-validated HOTS items at cognitive levels C4 (analyzing) and C5 (evaluating), ten text-based and ten image-based. Accuracy was scored in binary form against three criteria: correctness of the final answer, conceptual understanding, and correct use of equations. Consistency was assessed by posing every item 30 times, each in a new conversation. On text-based items, both chatbots reached 80% accuracy (Good), with average consistency of 89% for ChatGPT and 91% for DeepSeek (both Very Good). On image-based items, ChatGPT achieved 100% accuracy and 100% consistency (Very Good), whereas DeepSeek achieved 60% accuracy and 73.7% consistency (Fairly Good). C5 items were the hardest for both chatbots, and DeepSeek’s errors on image-based items came mainly from misreading graphs rather than from wrong formulas. Chatbot output should therefore be verified by teachers and students before it is used as a learning aid in physics.

References

Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.
Andriyeni, & Zakir. (2023). Analisis penggunaan artificial intelligence dalam pemahaman pembelajaran fisika di SMAN 1 Ampek Angkek. Jurnal Ilmiah Research Student, 2(1).
Arikunto, S. (2018). Dasar-dasar evaluasi pendidikan. Bumi Aksara.
Azizah, M. A., & Mansyur, J. (2024). Perbandingan kapabilitas respons ChatGPT dan Gemini terhadap pertanyaan konseptual tentang optik. JPFT (Jurnal Pendidikan Fisika Tadulako Online), 12(1), 15–25.
Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. ASCD.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
DeepSeek AI. (2024). DeepSeek technical documentation. https://www.deepseek.com
DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.
Docktor, J. L., & Mestre, J. P. (2014). Synthesis of discipline-based education research in physics. Physical Review Special Topics - Physics Education Research, 10(2), 020119.
Funk, P. F., Hoch, C. C., Knoedler, S., Knoedler, L., Cotofana, S., Sofo, G., Dezfouli, A. B., Wollenberg, B., Guntinas-Lichius, O., & Alfertshofer, M. (2024). ChatGPT’s response consistency: A study on repeated queries of medical examination questions. European Journal of Investigation in Health, Psychology and Education, 14(3), 657–668.
Giancoli, D. C. (2014). Physics: Principles with applications (7th ed.). Pearson.
Halliday, D., Resnick, R., & Walker, J. (2013). Fundamentals of physics (10th ed.). Wiley.
Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of artificial intelligence in education. Computers and Education: Artificial Intelligence, 1, 100001.
Kemendikbudristek [Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi]. (2022). Buku siswa fisika untuk SMA/MA kelas X. Pusat Kurikulum dan Perbukuan.
Magfira, A., Kamaluddin, & Darsikin. (2021). Kesalahan siswa dalam memahami konsep kinematika gerak lurus. Jurnal Pendidikan Fisika.
Miles, M. B., & Huberman, A. M. (1992). Qualitative data analysis: An expanded sourcebook (2nd ed.). SAGE Publications.
Moleong, L. J. (2017). Metodologi penelitian kualitatif (Edisi revisi). PT Remaja Rosdakarya.
Mollick, E. R., & Mollick, L. (2023). Using AI to implement effective teaching strategies in classrooms: Five strategies, including prompts. The Wharton School Research Paper.
Mustofa, H. A., Bilad, M. R., & Grendis, N. W. B. (2024). Utilizing AI for physics problem solving: A literature review and ChatGPT experience. Lensa: Jurnal Kependidikan Fisika, 12(1), 78–97.
Nurhuda, Kumala, & Widiyatun. (2023). Analysis of ChatGPT artificial intelligence in solving illustrated physics problems on resistors.
Oblitas, L., Adeeb, S., & Cruz-Noguez, C. (2025). Performance of AI chatbots on the fundamentals of engineering civil exam. Discover Artificial Intelligence, 5(1), 378.
Polverini, G., & Gregorcic, B. (2024). Evaluating vision-capable chatbots in interpreting kinematics graphs: A comparative study of free and subscription-based models. Frontiers in Education, 9, 1452414.
Russell, S. J., & Norvig, P. (2020). Artificial intelligence: A modern approach (4th ed.). Pearson.
Sabharwal, C. L. (2021). Cohen’s Kappa statistic and new Kappa statistic for measuring and interpreting inter-rater agreement. International Journal of Research in Engineering and Science, 9(2), 23–28.
Sani, R. A. (2019). Cara membuat soal HOTS (High Order Thinking Skills). Tira Smart.
Sugiyono. (2017). Metode penelitian kuantitatif, kualitatif, dan R&D. Alfabeta.
Sukmadinata, N. S. (2011). Metode penelitian pendidikan. PT Remaja Rosdakarya.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., & Fedus, W. (2022). Emergent abilities of large language models. Transactions on Machine Learning Research.
Zed, M. (2008). Metode penelitian kepustakaan (2nd ed.). Yayasan Obor Indonesia.