Abstract:

Q-matrix is the core of cognitive diagnosis, and the Q-matrix constructed by experts usually has certain misspecifications, which will reduce the estimation accuracy and thus needs to be validated. New machine learning-based Q-matrix validation methods (RF-P, RF-L, and RF-R) is proposed using the random forest (RF) algorithm with PVAF, log-likelihood, and modified R statistics as the feature training models, and simulation and empirical studies are conducted to verify the performance. The results show that (1) the accuracy, recall, precision, F1, Kappa of the three models are above 0.75; (2) in the simulation study, the new methods based on the three RF models have better validation performance than the Wald-XPD method, which was latest published, among which RF-R has the best performance; (3) in the empirical study, RF-R suggested a more reasonable Q matrix with optimal model-data fitting results.