An Analysis of ChatGPT-Generated Cloze Tests for Reading Assessment of College Students: Practicality, Validity, and Reliability
DOI:
https://doi.org/10.24036/s4x9fz12Keywords:
ChatGPT, Cloze Test, Practicality, Validity, ReliabilityAbstract
The use of artificial intelligence in language assessment has increased interest in automated test generation. In reading assessment, cloze tests are widely used to measure comprehension through contextual processing. This study examines the practicality, validity, and reliability of a ChatGPT-generated cloze test for reading assessment of college students. Using a descriptive quantitative design, 25 students completed a 30-item cloze test generated by ChatGPT without human modification. Practicality was measured through a 20-item student perception questionnaire, while validity and reliability were analyzed using corrected item–total correlation and Cronbach’s Alpha. The results indicate high practicality and acceptable reliability, although only a limited number of items were valid. These findings suggest that ChatGPT can support language test generation with proper statistical analysis.
References
Alderson, J. C. (1979). The cloze procedure and proficiency in English as a foreign language. TESOL Quarterly, 13(2), 219–227. https://doi.org/10.2307/3586211
Alderson, J. C. (2000). Assessing reading. Cambridge University Press.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.
Brown, H. D. (2004). Language assessment: Principles and classroom practices. Pearson Education.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). SAGE Publications.
DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). SAGE Publications.
George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference, 11.0 update (4th ed.). Allyn & Bacon.
Hwang, G.-J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of artificial intelligence in education. Computers and Education: Artificial Intelligence, 1, Article 100001. https://doi.org/10.1016/j.caeai.2020.100001
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274
Oller, J. W., Jr. (1979). Language tests at school: A pragmatic approach. Longman.
Taylor, W. L. (1953). “Cloze procedure”: A new tool for measuring readability. Journalism Quarterly, 30(4), 415–433. https://doi.org/10.1177/107769905303000401
Yan, Z., Wang, T., & Wang, L. (2022). Rethinking the fairness of AI in education: Validity, bias, and transparency. British Journal of Educational Technology, 53(4), 838–856.
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education: Where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0
Zhai, X. (2022). ChatGPT user experience: Implications for education. SSRN. https://doi.org/10.2139/ssrn.4312418
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Muhammad Iqbal Julianda, Dinovia Fannil Kher (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
