QuantDev Brownbag - Progress toward NLP-assisted formative assessment feedback

Google map

Progress toward NLP-assisted formative assessment feedback

Abstract: 

This seminar seeks to articulate the benefit of free-response tasks and timely formative assessment feedback, a roadmap for developing human-in-the-loop natural language processing (NLP) assisted feedback, and results from a pilot study establishing proof of principle. If we are to pursue Statistics and Data Science Education across disciplines, we will surely encounter both opportunity and necessity to develop scalable solutions for pedagogical best practices. Research suggests “write-to-learn” tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy when class sizes grow large. In the pilot study, several short-answer tasks completed by nearly 2000 introductory tertiary statistics students were evaluated by human raters and an NLP algorithm. After briefly describing the tasks, the student contexts, the algorithm and the raters, this talk discusses the results which indicate substantial inter-rater agreement and group consensus. With compelling rater agreement, the study then examines a preliminary cluster analysis of response text as a mechanism for scalable formative assessment. The talk will conclude with recent developments building upon this pilot, as well as implications for teaching and future research.

Resources about this talk: SSRI QuantDev Seminar (mdbeckman.github.io)