Keywords
"embedded answers (cloze)"
Rasch’s model
polytomous model
R programming environment
packages eRm и ltm
MOODLE
How to Cite
Abstract
The article examines the issue of choosing a mathematical model for the analysis of computer tests in mathematical disciplines. A feature of this study is the exploration of test items of the "embedded answers (cloze)" type. Such test tasks have several “steps” interconnected by a certain logic and have been used precisely for remote control of knowledge in mathematical disciplines during quarantine at NTUU “Igor Sikorsky Kyiv Polytechnic Institute”.The introduction of such tasks into the test raises the question for the teacher: how to analyze the quality of such tests? The authors built the study on the basis of the well-known methods of Classical Test Theory (CTT) and Item Response Theory (IRT), which have proven their effectiveness in pedagogical testing as well as in other industries in which the test approach is used. The study was carried out on the example of a modular test quizzes on probability theory for second-year students of the Faculty of Informatics and Computer Engineering of NTUU "Igor Sikorsky Kyiv Polytechnic Institute", formed on the MOODLE platform during distance learning in November 2020. In the course of the study, the following was done: the quality of the test was analyzed both as a whole and as individual test items; the latent parameters of the tasks were estimated, the adequacy of the corresponding models to the testing data was checked; characteristic and information curves were built; using information criteria, were selected the IRT models the most suitable for the analysis of computer tests with tasks of the "embedded answers" type; it has been proven that both Rush's dichotomous model and Muraki's political model are the most appropriate for the analysis of such tasks. The calculation was carried out using special libraries (packages) of R programming language, in which the KTT and IRT algorithms are implemented, namely, eRm, ltm, mirt. It is shown how the tasks with “embedded answers (cloze)” can be used to analyze certain skills and knowledge of students. The results of the study contribute to improving the competence of teachers in assessing the quality of tests in higher mathematics, which is very important when compiling tests in various disciplines, especially in the situation of student distance learning.
References
Moodle.org: open-source community-based tools for learning. [Електронний ресурс]. Доступно: http://www.moodle.org
В. П. Сергієнко, В. М. Франчук, Л.О. Кухар, Методичні рекомендації зі створення тестових завдань та тестів у системі управління навчальними матеріалами MOODLE., К.: НПУ імені М.П. Драгоманова, 100 с., 2014. [Електронний ресурс]. Доступно: https://vfranchuk.fi.npu.edu.ua/images/files/statty/47.pdf
А. М. Польовий, О. В. Шаблій О. В., І. А. Хоменко, Збірник методичних вказівок «Організація навчального курсу в СДН Moodle». ОДЕКУ, 182 c., 2019. [Електронний ресурс]. Доступно: http://eprints.library.odeku.edu.ua/id/eprint/6238/1/PolevoyAM_Shabl%C3%ADyOV_Khomenko%C3%8DA_Zb%C3%ADrnik_metodichnikh_vkaz%C3%ADvok_Organ%C3%ADzats%C3%ADya_navchal'nogo_kursu_v_SND_Moodle_%20(yel_vidannya)%20_2019.pdf
П. І. Андронатій, В.В. Котяк, Комп’ютерні технології в освітніх вимірюваннях: Навчально-методичний посібник. Кіровоград: Лисенко В.Ф., 144 с., 2011. [Електронний ресурс]. Доступно: http://moodle.ndu.edu.ua/pluginfile.php/889/mod_page/content/1/Andronatiy_Kotiak_Komp_tehnol_v_Osvit_Vymir.pdf
А. Ф. Дудко, Комп’ютерно орієнтована методика оцінювання якості тестів з вищої математики викладачами вищої освіти. Автореферат дисертації на здобуття наукового ступеня кандидата педагогічних наук, Київ, 24 с., 2019.
О. О. Диховичний, та А. Ф. Дудко, “Комплексна методика аналізу якості тестів з вищої математики”, Науковий часопис НПУ імені М.П. Драгоманова. Серія №2. Комп’ютерно-орієнтовані системи навчання: Зб. наук. праць, №15 (22), с. 140-144, 2015.
Н. В. Круглова, О.О. Диховичний, та І.В. Алєксєєва, Особливості застосування математичних моделей тестів в умовах дистанційного контролю. Математичні машини і системи, №. 2, с. 105-116, 2020.
В. Ю. Биков, Ю. М. Богачков, та Ю. О. Жук, Моніторинг рівня навчальних досягнень з використанням Інтернет-технологій: монографія. Київ, Україна: Педагогічна думка, 128 с., 2008.
O. H. Kolhatin, Requirements on the design of the automated system of pedagogical diagnostics. Information Technologies and Learning Tools, 19(5), 2010, doi:https://doi.org/10.33407/itlt.v19i5.352
O. H. Kolhatin, Еxperience of the automated system application for diagnostics of the learning achievements in the mathematical statistics methods. Information Technologies and Learning Tools, 5(1),2008. doi:https://doi.org/10.33407/itlt.v5i1.153
Y. M. Bogachkov, and P. S. Ukhan, Application of qualifiers in the systems of measurement of the level of educational achievements. Information Technologies and Learning Tools, 4(3),2007. doi:https://doi.org/10.33407/itlt.v4i3.169
L. Crocker, and J. Algina, Introduction to Classical and Modern Test Theory. Belmont, CA:Wadsworth, 527 p., 2006.
М.Б. Челышкова, Теория и практика конструирования педагогических тестов.М.: Логос, 432 с., 2002.
G. Rasch, Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research, Copenhagen, 126 p., 1960.
M.Wilson, Constructing Measures: An Item Response Modeling Approach. Mahwah, NJ: Erlbaum, 2005.
F. Baker, The Basics of Item Response Theory. College Park, MD: ERIC Clearinghouse on Assessment and Evaluation, 2001. [Електронний ресурс]. Доступно: http://echo.edres.org:8080/irt/baker/ Дата звернення: Квітень 24, 2020.
W. Linden and R. Hambleton, Handbook of modern item response theory. New York: Springer, 2018.
В. С. Аванесов, Item Response Theory: Основные понятия и положення, Педагогические измерения, №2, с.3-28, 2007.
Т. В. Лісова, Моделі та методи сучасної теорії тестів. Ніжин, Україна: Видавець ПП Лисенко М. М., 2012.
E. Muraki, and B. Carlson, Full-information factor analysis for polytomous item responses. Applied Psychological Measurement. Vol. 19, N 1. pp. 73–90, 1995.
The Comprehensive R Archive Network. [Електронний ресурс]. Доступно: http://cran.us.r-project.org
W. Holmes Finch, Jocelyn E. Bolin, and Ken Kelley Multilevel Modeling Using R , 2019 by Chapman and all/CRC Press, 252 p., 2019.
David A. Armstrong II, Ryan Bakker, Royce Carroll, Christopher Hare, Keith T. Poole, and Howard Rosenthal Analyzing Spatial Models of Choice and Judgment with R, CRC Press, Boca Raton, 336 p., 2014.
В. С. Ким, Тестирование учебных достижений. Монография. Уссурийск: Издательство УГПИ, 214 с., 2007.
Энциклопедия психологических тестов. Личность Мотивация. Потребность. /Под ред. А. Карелина. М., 312 с., 1997.
Handbook of item response theory. volume two, statistical tools edited by Wim J. van der Linden, Boca Raton, FL, CRC Press, 427 p., 2016.
M. Chambers, Trevor Hastie, Statistical Models in S. Wadsworth & Brooks/Cole Advanced Books & Software, 608 р., 1992.
N. Kruglova, O. Dykhovychnyi, D. Lysenko, Application of IRT and MIRT models to analysis of analytical geometry tests. Міжвідомчий науково-технічний збірник «Адаптивні системи автоматичного управління», № 1 (38), pp. 36-49, 2021.
M. D. Reckase, Multidimensional item response theory. New York: Springer, 2009.
REFERENCES (TRANSLATED AND TRANSLITERATED)
Moodle.org: open-source community-based tools for learning. [Online]. Available: http://www.moodle.org (in English).
V. P. Serhiienko, V. M. Franchuk, L. O. Kukhar, Methodical recommendations for creating test tasks and tests in the management system of educational materials MOODLE, K.: NPU imeni M.P. Drahomanova, 100 ., 2014. [Online]. Available: https://vfranchuk.fi.npu.edu.ua/images/files/statty/47.pdf (in Ukrainian).
A. M. Polovyi, O. V. Shablii, I. A. Khomenko, Collection of methodical instructions "Organization of a training course in Moodle". ОDKEU, 182 p., 2019. [Online]. Available: http://eprints.library.odeku.edu.ua/id/eprint/6238/1/PolevoyAM_Shabl%C3%ADyOV_Khomenko%C3%8DA_Zb%C3%ADrnik_metodichnikh_vkaz%C3%ADvok_Organ%C3%ADzats%C3%ADya_navchal'nogo_kursu_v_SND_Moodle_%20(yel_vidannya)%20_2019.pdf (in Ukrainian).
P. I. Andronatii, V. V. Kotiak, Сomputer technology in educational measurements. Navchalno-metodychnyi posibnyk. Kirovohrad: Lysenko V.F, 144 p., 2011. [Online]. Available: http://moodle.ndu.edu.ua/pluginfile.php/889/mod_page/content/1/Andronatiy_Kotiak_Komp_tehnol_v_Osvit_Vymir.pdf (in Ukrainian).
A. F. Dudko, Computer-oriented methodology of assessing the quality of tests in higher mathematics by teachers of higher education establishments. Avtoreferat dysertatsii na zdobuttia naukovoho stupenia kandydata pedahohichnykh nauk, Kyiv, 24 р., 2019. (in Ukrainian) .
O. O. Dykhovychnyi, and A. F. Dudko, “The comprehensive procedure of analysis of tests in higher mathematics”, Naukovyi chasopys NPU imeni M.P. Drahomanova, Seriia №2. Kompiuterno-oriientovani systemy navchannia: Zb. nauk. prats, no. 15 (22), pp. 140-144, 2015. (in Ukrainian).
N. V. Kruglova , O. O. Dykhovychnyi, and I.V. Alekseeva, Features of mathematical application test models in the conditions of remote control. Mathematical Machines and Systems, no.. 2, рр. 105-116, 2020. (in Ukrainian).
V. Iu. Bykov, Yu. M. Bohachkov, ta Yu. O. Zhuk, Monitoring the evaluation of educational achievement using Internet technologies: monograph. Kyiv, Ukraine: Pedahohichna dumka, 2008. (in Ukrainian).
O. H. Kolhatin. Requirements on the design of the automated system of pedagogical diagnostics. Information Technologies and Learning Tools, 19(5), 2010, doi:https://doi.org/10.33407/itlt.v19i5.352 (in Ukrainian).
O. H. Kolhatin, Еxperience of the automated system application for diagnostics of the learning achievements in the mathematical statistics methods. Information Technologies and Learning Tools, 5(1),2008. doi:https://doi.org/10.33407/itlt.v5i1.153 (in Ukrainian).
Y. M. Bogachkov, & P. S. Ukhan, Application of qualifiers in the systems of measurement of the level of educational achievements. Information Technologies and Learning Tools, 4(3),2007. doi:https://doi.org/10.33407/itlt.v4i3.169 (in Ukrainian).
L. Crocker, and J. Algina, Introduction to Classical and Modern Test Theory. Belmont, CA:Wadsworth, 527 p., 2006. (in English).
M. B. Chelyshkova, Theory and practice of constructing of pedagogical tests. Moscow, RF: Logos, 2002. (in Russian).
G. Rasch, Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research, Copenhagen, 126 p., 1960 (in English).
M.Wilson, Constructing Measures: An Item Response Modeling Approach. Mahwah, NJ: Erlbaum, 2005. (in English).
F. Baker, The Basics of Item Response Theory. College Park, MD: ERIC Clearinghouse on Assessment and Evaluation, 2001. [Online]. Available: http://echo.edres.org:8080/irt/baker/ Accessed on: Apr. 24, 2020. (in English).
W. Linden and R. Hambleton, Handbook of modern item response theory. New York: Springer, 2018 (in English).
V. S. Avanesov, Item Response Theory: The basic concepts and propositions. Pedagogicheskie izmerenija, №2, pp. 3-28., 2007, (in Russian).
T. V. Lisova, Models and methods of the modern test theory. Nizhyn, Ukraina: Vydavets PP M. M., 2012. (in Ukrainian).
E. Muraki, and B. Carlson,. Full-information factor analysis for polytomous item responses. Applied Psychological Measurement. Vol. 19, N 1. pp. 73–90, 1995. (in English).
The Comprehensive R Archive Network. [Online]. Available: http://cran.us.r-project.org
W. Holmes Finch, Jocelyn E. Bolin, and Ken Kelley, Multilevel Modeling Using R , 2019 by Chapman and all/CRC Press, 252 p., 2019. (in English).
David A. Armstrong II, Ryan Bakker, Royce Carroll, Christopher Hare, Keith T. Poole, and Howard Rosenthal, Analyzing Spatial Models of Choice and Judgment with R, CRC Press, Boca Raton, 336 p., 2014. (in English).
V. S. Kym, Testing of educational achievements. Monograph. Ussuryisk: Yzdatelstvo UHPY, 214 p., 2007, (in Russian).
Encyclopedia of psychological tests. Personality Motivation. Requirement. Pod red. A. Karelyna. M., 312 p., 1997. (in Russian).
Handbook of item response theory. volume two, statistical tools edited by Wim J. van der Linden, Boca Raton, FL, CRC Press, 427 p., 2016. (in English).
M. Chambers, Trevor Hastie, Statistical Models in S. Wadsworth & Brooks/Cole Advanced Books & Software, 608 р., 1992. (in English).
N. Kruglova, O. Dykhovychnyi, D. Lysenko, Application of IRT and MIRT models to analysis of analytical geometry tests. Mizhvidomchyi naukovo-tekhnichnyi zbirnyk «Adaptyvni systemyavtomatychnoho upravlinnia», № 1 (38), pp. 36-49, 2021. (in English).
M. D. Reckase, Multidimensional item response theory. New York: Springer, 2009. (in English).
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright (c) 2022 Наталія Володимирівна Круглова, Олександр Олександрович Диховичний