How reliable is it to automatically score open-ended items? An application in the Turkish language

Basit Öğe Kaydı
dc.authorid0000-0002-6767-0362en_US
dc.authorid0000-0001-6274-2016en_US
dc.contributor.authorUysal, İbrahim
dc.contributor.authorDoğan, Nuri
dc.date.accessioned2023-07-14T12:17:51Z
dc.date.available2023-07-14T12:17:51Z
dc.date.issued2021en_US
dc.departmentBAİBÜ, Eğitim Fakültesi, Eğitim Bilimleri Bölümüen_US
dc.description.abstractThe use of open-ended items, especially in large-scale tests, created difficulties in scoring open-ended items. However, this problem can be overcome with an approach based on automated scoring of open-ended items. The aim of this study was to examine the reliability of the data obtained by scoring open-ended items automatically. One of the objectives was to compare different algorithms based on machine learning in automated scoring (support vector machines, logistic regression, multinominal Naive Bayes, long-short term memory, and bidirectional long-short term memory). The other objective was to investigate the change in the reliability of automated scoring by differentiating the data rate used in testing the automated scoring system (33%, 20%, and 10%). While examining the reliability of automated scoring, a comparison was made with the reliability of the data obtained from human raters. In this study, which demonstrated the first automated scoring attempt of open-ended items in the Turkish language, Turkish test data of the Academic Skills Monitoring and Evaluation (ABIDE) program administered by the Ministry of National Education were used. Cross-validation was used to test the system. Regarding the coefficients of agreement to show reliability, the percentage of agreement, the quadratic-weighted Kappa, which is frequently used in automated scoring studies, and the Gwet's AC1 coefficient, which is not affected by the prevalence problem in the distribution of data into categories, were used. The results of the study showed that automated scoring algorithms could be utilized. It was found that the best algorithm to be used in automated scoring is bidirectional long-short term memory. Long-short term memory and multinominal Naive Bayes algorithms showed lower performance than support vector machines, logistic regression, and bidirectional long-short term memory algorithms. In automated scoring, it was determined that the coefficients of agreement at 33% test data rate were slightly lower comparing 10% and 20% test data rates, but were within the desired range.en_US
dc.identifier.citationUysal, I., & DOĞAN, N. (2021). How Reliable Is It to Automatically Score Open-Ended Items? An Application in the Turkish Language. Journal of Measurement and Evaluation in Education and Psychology, 12(1), 28-53.en_US
dc.identifier.doi10.21031/epod.817396
dc.identifier.endpage54en_US
dc.identifier.issn1309-6575
dc.identifier.issue1en_US
dc.identifier.scopus2-s2.0-85112175890en_US
dc.identifier.scopusqualityQ4en_US
dc.identifier.startpage28en_US
dc.identifier.trdizinid460805en_US
dc.identifier.urihttp://dx.doi.org/10.21031/epod.817396
dc.identifier.urihttps://hdl.handle.net/20.500.12491/11294
dc.identifier.volume12en_US
dc.identifier.wosWOS:000636562100003en_US
dc.identifier.wosqualityN/Aen_US
dc.indekslendigikaynakWeb of Scienceen_US
dc.indekslendigikaynakScopusen_US
dc.indekslendigikaynakTR-Dizinen_US
dc.institutionauthorUysal, İbrahim
dc.language.isoenen_US
dc.publisherAssoc Measurement & Evaluation Education & Psychologyen_US
dc.relation.ispartofJournal of Measurement and Evaluation in Education and Psychology-Epoden_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/openAccessen_US
dc.subjectOpen-Ended Itemen_US
dc.subjectMachine Learning Algorithmsen_US
dc.subjectAutomated Scoringen_US
dc.subjectInter-Rater Reliabilityen_US
dc.subjectCoefficients of Agreementen_US
dc.subjectGwet's AC1en_US
dc.titleHow reliable is it to automatically score open-ended items? An application in the Turkish languageen_US
dc.typeArticleen_US

Dosyalar

Orijinal paket
Listeleniyor 1 - 1 / 1
Küçük Resim
İsim:
ibrahim-uysal.pdf
Boyut:
2.07 MB
Biçim:
Adobe Portable Document Format
Açıklama:
Tam metin/Full text
İndir
Lisans paketi
Listeleniyor 1 - 1 / 1
Küçük Resim Yok
İsim:
license.txt
Boyut:
1.44 KB
Biçim:
Item-specific license agreed upon to submission
Açıklama:
İndir

Koleksiyon

WoS İndeksli Yayınlar Koleksiyonu
Eğitim Bilimleri Bölümü Koleksiyonu
Scopus İndeksli Yayınlar Koleksiyonu
TR-Dizin İndeksli Yayınlar Koleksiyonu