The Andes Physics Tutor [1,2] is a nice piece of work.\u00a0 Looking at the data from the Spring 2010 physics course at the US Naval Academy [3] there are obvious challenges facing the current generation of adaptive education vendors.<\/p>\n
Most of the modeling going on today is limited to questions where a learner gets the answer right or wrong with nothing in between.\u00a0 Most assessment systems use such simple “dichotomous” models based on item response theory (IRT).\u00a0 IRT models range in sophistication based on the number of “logistic parameters” that the models use to describe assessment items.\u00a0 IRT models also come in many variations that address graded responses (e.g., Likert scales [4]) or partial credit and onto multi-dimensional and\/or multi-level\/hierarchical and\/or longitudinal models.\u00a0 I am not aware of any model that addresses the richness of pedagogical data available from the physics tutor, however.<\/p>\n
Having spent too long working through too much of the research literature in the area, a summary path through all this may be helpful…\u00a0 There are 1, 2, and 3 parameter logistic IRT models that characterize assessment items in terms of their difficulty, discrimination, and guess-ability. These are easiest to understand initially in the context of an assessment item that assesses a single cognitive skill by scoring a learner’s performance as passing or failing on the item.\u00a0\u00a0 Wikipedia does a good job of illustrating how these three numbers describe the cumulative distribution function of the probability that a learner will pass as his or her skill increases [5].\u00a0\u00a0 A 1 parameter logistic (1PL) model describes an assessment item only in terms of a threshold of skill at which half the population falls on either side.\u00a0 The 2PL IRT model also considers the steepness of the curve.\u00a0 The steeper it is the more precisely the assessment item discriminates between above and below average levels of skill.\u00a0 And the 3PL model takes into consideration the odds that the a passing result is a matter of luck, such as in guessing the right answer from a multiple choice question.<\/p>\n
In the worlds of standardized testing and educational technology, especially with regard to personalized learning (as in adaptive assessment, curriculum sequencing, and intelligent tutoring), multiple choice and fill-in-the-blank questions dominate because grading can be automated.\u00a0 It is somewhat obvious then that 3PL IRT is the appropriate model for standardized testing (which is all multiple choice) and a large fraction of personalized learning technology.\u00a0 Fill-in-the-blank questions are sometimes less vulnerable to guessing, in which case 2PL may suffice.\u00a0 You may be surprised to learn, however, that even though 3PL is strongly indicated, its use is not pervasive because estimating the parameters of a 3PL model is mathematically and algorithmically sophisticated as well as computationally demanding.\u00a0 Nonetheless, algorithmic advances in Bayesian inference have matured over the last decade such that 3PL should become much more pervasive in the next few years.\u00a0 (To their credit, for example, Knewton appears to use such techniques [6].)<\/p>\n
It gets worse, though.\u00a0 Even if an ed-tech vendor is sophisticated enough to employ 3PL IRT, they are far less likely to model assessment items that involve multiple or hierarchical skills and assessments other than right or wrong. And there’s more beyond these complications, but let’s pause to consider two of these for a moment.\u00a0 In solving a word problem, such as in physics or math, for example, a learner needs linguistic skills to read and comprehend the problem before going on to formulate and solve the problem.\u00a0 These are different skills.\u00a0 They could be modeled coarsely, such as a skill for reading comprehension, a skill for formulating equations, and a skill for solving equations,\u00a0 but conflating these skills into what ITR folks sometimes call a single latent trait is behind the state of the art.<\/p>\n
Today, multi-dimensional IRT is hot given recent advances in Bayesian methods.\u00a0 Having said that, it’s worth noting that multiple skills have been on experts’ minds for a long time (e.g., [7]) and have been prevalent in higher-end educational technology, such as intelligent tutoring systems (aka, cognitive tutors), for years.\u00a0 These issues are prevalent in almost all the educational data available at PLSC’s DataShop [3].\u00a0 Unfortunately, the need to associate multiple skills with assessment items exacerbates a primary obstacle to broader deployment of better personalized learning solutions: pedagogical modeling.\u00a0 One key aspect of a pedagogical model is the relationship between assessment items and the cognitive skills they require (and assess).\u00a0 Given such information, multi-dimensional IRT can be employed, but even articulating a single learning objective per assessment item and normalizing those learning objectives over thousands of assessment items is a major component of the cost of developing curriculum sequencing solutions.\u00a0 (We’ll be announcing progress on this problem “soon”.)<\/p>\n
In addition to multi-dimensional IRT, which promises more cognitive tutors, there are other aspects of modeling assessment, such as in hierarchical or multi-level IRT.\u00a0\u00a0 Although the terms hierarchical and multi-level are sometimes used interchangeably with respect to models of assessment, we are more comfortable with the former being with respect to skills and the latter with regard to items.\u00a0 A hierarchical model is similar to a multi-dimensional model in that an item involves multiple skills but most typically where those skills have some taxonomic relationship.\u00a0 A multi-level model allows for shared structure or focus between assessment items, including multiple questions concerning a common passage\u00a0 or scenario, as well as drill-down items.\u00a0 All of the issues discussed in this paragraph are prevalent in the Andes Physics Tutor data.\u00a0 Many other data models available at PLSC’s Datashop also involve hierarchically organization of multiple skills (aka, “knowledge components”).<\/p>\n
And we have yet to address other critical aspects of a robust model of assessment!\u00a0 For example, we have not considered how the time taken to perform an assessment reflects on a learner’s skil nor graded responses or grades other than pass\/fail (i.e., polytomous vs. dichotomous models).\u00a0 The former is available in the data (along with multiple attempts, hints, and explanations that we have not touched on).\u00a0 The latter\u00a0 remains largely unaddressed despite being technically straightforward (albeit somewhat sophisticated).\u00a0 All of these are important, so put them on your assessment or pedagogical modeling and personalized learning checklist and stay posted!<\/p>\n
[1] http:\/\/www.andestutor.org\/<\/p>\n
[2] Vanlehn, Kurt, et al. “The Andes physics tutoring system: Lessons learned<\/a>.” International Journal of Artificial Intelligence in Education<\/em> 15.3 (2005): 147-204.<\/p>\n [3] available on-line at the Pittsburgh Learning Sciences Collaboration (PLSC) Datashop<\/a><\/p>\n [4] Likert scales & items at Wikipedia<\/a><\/p>\n [5] the item response function of ITR at Wikipedia<\/a><\/p>\n [6] see the references to Monte Carlo Markov Chain estimation of ITR models in this post from Knewton’s tech blog<\/a><\/p>\n