Authoring questions for education and assessment

Thesis: the overwhelming investment in educational technology will have its highest economic and social impact in technology that directly increases the rate of learning and the amount learned, not in technology that merely provides electronic access to abundant educational resources or on-line courses.  More emphasis is needed on the cognitive skills and knowledge required to achieve learning objectives and how to assess them.

Content is No Longer King

Given an educational resource, such as CK-12’s Biology textbook, how do we help students learn the material?  We need to do more than make the material easy to read, which is largely accomplished in this electronic textbook and others from Boundless, Pearson, Nature, Wilson, and many others.  These electronic textbooks, combined with thousands of educational resources from Khan Academy, YouTube, Vimeo, and too many other sites to enumerate, provide more than enough educational resources to teach the material.

The challenge of advancing education is moving beyond the development of open educational resources.  It has progressed towards electronically-enhanced learning, or e-learning, that enhances traditional education.  There are too many forms of e-learning to discuss here, as enumerated in the prior link to Wikipedia.  Here, we divide those forms of e-learning that address delivering educational resources electronically, including on-line education, virtual learning environments, and digital textbooks from those that take an active role in pedagogy.

A system that takes an active role in pedagogy must take some initiative to advance learning.  In doing so, it typically must assess the knowledge and cognitive skills of students, but assessment is not all that we require.  In addition, and more importantly, the system must somehow teach or otherwise take an active role in improving students’ knowledge and cognitive skills.

As discussed above, there is no shortage of educational resources available.  Especially in traditional areas of education, there are plenty of open educational resources available.  The challenge in educational technology has moved beyond developing such resources to harnessing them more effectively in advancing learning.

Access is not the Answer

Learning management systems (LMS) and virtual learning environments (VLE) are pervasive platforms in education.  According to the prior links to Wikipedia:

  • A learning management system (LMS) is a software application for the administration, documentation, tracking, reporting and delivery of e-learning education courses or training programs.
  • LMSs range from systems for managing training and educational records to software for distributing online or blended/hybrid college courses over the Internet with features for online collaboration.
  • A virtual learning environment (VLE), or learning platform, is an e-learning education system based on the web that models conventional in-person education by providing equivalent virtual access to classes, class content, tests, homework, grades, assessments, and other external resources such as academic or museum website links.

According to the Software & Information Industry Association (SIIA) report on the 2013 K-12 market for educational technology, revenue is balanced between content and “instructional support” the latter of which is dominated by testing and assessment.  Half as much of either, about $1.5B USD, was spent on platforms and administration, which includes LMS and VLE, but spending in this area was down over 30%.

Why?  Several reasons…  The report notes decreased public funding from states pressured by reduced tax revenue from the recession and reduced federal investment from lapsed stimulus spending.  There is also saturation in the LMS/VLE market.  But perhaps most importantly, the platforms don’t advance learning itself, although they have improved distribution or delivery and access.

Where the Action Is

An article in the Washington Post cites the IBIS Capital review of global investment in e-learning.  Essentially, the $5T of global expenditure on education is second only to healthcare expenditures and warrants investors’ attention.  This attention is warranted not as much by the rate of growth in the market but by its sheer magnitude and the clear imminence of disruption by technology which presents unprecedented opportunities for return on investment.

And the attention is focused on e-learning for good reason.  For example, consider this quote from a TechCrunch article, Education Technology Startups Raised over Half a Billion Dollars in Q1:

  • “It’s interesting because public education hasn’t changed that much in 150 to 200 years and there had been almost no technology going into it”
  • “It’s not only that there’s this huge behemoth sector of the economy that spends $1.2 trillion on educating kids, but that it’s old, it’s long in the tooth and it’s bound to get disrupted.”

There are many aspects of e-learning that are transforming education.  For example, social and gaming segments are growing very quickly, along with mobile and on-line education, although the latter are more about than distribution or delivery and access than learning itself.

Assessment is Not Enough

As mentioned above regarding the SIIA report, recent spending has remained balanced between content and instructional support with the highest area of focused investment being in testing and assessment.  There are many reasons for this, including many state and federal programs focused on increasingly standardized and electronic testing.  But those efforts are not about advancing education directly.  Their objective is measurement of outcomes so as to assess performance of academic institutions and educators.  Advocates of test-centric metrics hope that more measurement will improve educational outcomes over time through economic incentives they argue will improve educational performance.  The approach is not directly focused on enhancing the learning rate or level of education of students directly, however.  It’s more about improving traditional education than taking a quantum leap forward.

Learning More Faster

There are several areas that offer hype or hope for taking a quantum leap forward through educational technology.  These include social learning, educational games or immersive simulation or training, and evolutions of computer-based training or computer-assisted education.

Social learning is based on a social network that collectively engages in education.  Such engagement varies from communities of educators (whether teachers or tutors) helping students, to learners helping one another, or learners and educators collaborating on grading, and so on.

Gaming and simulation, which may include social learning, take a step beyond facilitating collaboration towards actually contributing to learning directly.  They are somewhat unique in that they combine assessment and education.  If you are killed more than your teammates you learn at the same time that you are assessed.  It’s an exciting area.  Unfortunately, it’s expensive to develop even a narrowly purposed educational game.  Furthermore, in part because of its costs but also due to its approach, gaming and simulation are not immediately applicable to broad swaths of primary, secondary, or post-secondary education which account for roughly 90% of the market.

The big potential is in the modern forms of what were known decades ago as computer-based training or computer-assisted education.  In particular, systems that combine assessment and instruction into a cycle of continuously adapting and personalizing engagement with educational resources and assessment towards improving educational outcomes.  These include cognitive tutors, which first became available a decade ago and more recent “adaptive” or personalized learning.

Cognitive Learning Systems

As noted in a previous post, personalized learning is a more general term that is not limited to e-learning but which can include one-on-one tutoring by a person.  In the space of e-learning, however, personalized learning and adaptive learning are, for all practical purposes, synonymous.

Cognitive Tutors and Models

Cognitive tutors were first developed by John Anderson at Carnegie Mellon.  He and Bob Longo spun that research out into Carnegie Learning.  They were the first to establish that personalized education in math improved learning outcomes.  The system is personalized in that it has a cognitive model of the problem solving required for each assessment item (i.e., math problem).  If you get it right, you have some degree of mastery of the cognitive skills required to solve the problem.  If you get it wrong, there is some deficiency in those cognitive skills (although you might simply have slipped up, which is also taken into account).  It’s the cognitive model, which involves mapping the knowledge and cognitive skills involved in solving problems, which distinguishes cognitive tutors from other approaches now taken in the adaptive learning space.

Adaptive Learning Systems

The investment required to develop a cognitive model as Carnegie did it is significant.  It took skilled technologists along with subject matter and pedagogical experts to develop a tutor as effective as that of Carnegie Learning.

ALEKS uses a different kind of model to provide a math tutor that competes effectively with Carnegie Learning.  The model uses the unfortunately vague and overused word “concept” to describe whatever needs to be known or mastered within a domain.  The basic approach is to organize these concepts into a graph of dependencies and to characterize an individual’s “knowledge state” as a position in a multi-dimensional space where each dimension is a degree of knowledge or mastery of such a concept.  Here is a good paper on “Knowledge Space Theory”.  The work, as with Anderson’s, dates back decades.

ALEKS’ approach is easy to understand (as depicted in Figure 3 of the reference given above) and is amenable to various applications of big data analytics (i.e., machine learning technology).  Knewton, for example, uses the same term, “concept”, and modeling approach.  Knewton is the leader in applying machine learning to big data (aka data science) for personalized education, as reflected by recent investor interest, in particular.

Learning Objectives

The modern term for these “concepts” is “learning objectives”.  Learning objectives can be formulated as sentences and are the core ingredient of educational standards such as the Common Core State Standards (CCSS) and the pivotal ingredient in the design of an effective course.

According to the Teacher Effectiveness Program at the University of Oregon:

  1. A learning objective should describe what students should know or be able to do at the end of the course that they couldn’t do before.
  2. Learning objectives should be about student performance.
  3. Good learning objectives shouldn’t be
    1. too abstract
      (“the students will understand what good literature is”);
    2. too narrow
      (“the students will know what a ground is”); or
    3. restricted to lower-level cognitive skills
      (“the students will be able to name the countries in Africa.”).

According to The Eberly Center for Teaching Excellence and Educational Innovation, learning objectives:

  1. Learning objectives should be student-centered.
  2. Learning objectives should break down the task and focus on specific cognitive processes.
    1. Many activities that faculty believe require a single skill (for example, writing or problem solving) actually involve a synthesis of many component skills.
    2. Breaking down the skills will allow us to select appropriate assessments and instructional strategies so that students practice all component skills.
    3. Learning objectives should use action verbs.
      1. Using action verbs enables you to more easily measure the degree to which students can do what you expect them to do.
      2. Learning objectives should be measurable.

In addition, the Eberly Center provides these links:

The ALEKS and Knewton approaches simplify learning objectives to nodes in a graph while Carnegie builds rich models of learning objectives including detail on the cognitive skills and knowledge involved in mastering them.  The trade-off is effective in that Knewton and McGraw-Hill (who acquired ALEKS) can produce effective systems at lower cost than the deep modeling of cognition and knowledge involved in the Carnegie and other more ambitious approaches.

The Whole Picture

A complete model supporting a more cognitive adaptive learning system is shown below.  In this model, learning objectives may be augmented with details about the cognitive skills and knowledge they involve.  The other aspects of this model are what is now pervasive in the adaptive learning space, although some vendors do not put as much emphasis on learning objectives as recommended by the authorities cited above.  This is most common, for example, in the test-prep market which can be so fixated on assessment that even instructional content is omitted.  (Note that the explanations in such offerings amount to instructional content and that some tagging of assessment items with skill or knowledge aspects is typically done for reporting purposes.)

Assessment is Essential

Technology cannot efficaciously actively advance learning in a personalized manner unless it has some assessment of a student’s mastery of something more abstract than an individual question or answer.  The word efficacious is important here.  Some approaches focus a great deal on simply tagging educational resources.  Educational resources that are tagged similarly to assessment items can be used to prepare for or remediate deficiencies of performance on such assessment items.  And machine learning can adaptively improve which instructional content is used in such preparation or remediation.

The key is whether naïve tagging will prove efficacious without amounting to the pedagogical and cognitive relationships depicted above.  If we tag something as “assessment” and we tag something as “solve equation” and we tag something as “involves fractions”, we are effectively building a cognitive model.  This is not what the simplest approaches are doing but is indicative of what it takes for efficacy.

The critical issue is having some fulcrum through which assessment can inform the recommendation of engagements.  Otherwise, the system is not helping to advance learning, it is simply a delivery vehicle, like Blackboard.  That fulcrum is some combination of learning objectives, cognitive skills, and knowledge, however those ingredients may be used to tag, describe, or relate educational resources.

Whatever the model, however, no technology can advance learning without assessment.  And there should multiple items to assess each learning objective.  This is the crux of the problem limiting the acceleration of learning via active pedagogical technology.  There is not enough emphasis upon assessment items and the learning objectives, cognitive skills, and knowledge they require.

There are far too few assessment items in any given textbook and, even in electronic textbooks, they are poorly organized, if at all, with respect to learning objectives (McGraw-Hill’s SmartBooks being a notable exception).

If you want to transform your course or publication into an educational system that actively engages students in a learning experience optimized to them, here’s what you need:

  1. Learning objectives
  2. Assessment Items
  3. Instructional Content

You don’t need to go overboard modeling your learning objectives to the level of cognitive skills and detailed knowledge, although you should understand them and articulate them as suggested by the authorities referenced above.

What you do need to do is to identify the learning objectives assessed, directly or indirectly, by your assessment items and make sure that you have as many assessment items per learning objectives as practical.  And for the best results, you want multiple instructional items per learning objective, too.

Here’s an example:  Suppose you have 200 pages of content to cover.  Typically, you will have a learning objective for every few pages and rarely more than several per page.  An average of one per two pages is not unreasonable for a good initial result.  You probably should not have less than 50 for those 200 pages and more than 200 is probably overkill, although not necessarily if you have the capacity.  But don’t overdo it.  You are more likely to be limited by your assessment items.

Suppose you have 50 learning objectives.  You should have a couple hundred assessment items, especially if they include practice items.  If you have 200 learning objectives, you need over 500 assessment items.  If you don’t have them, don’t bother.  If you do, you can deliver a state of the art solution.