systems that know and understand and think and learn
In preparing for some natural language generation[1], I came across some work on natural logic[2][3] and reasoning by textual entailment[4] (RTE) by Richard Bergmair in his PhD at Cambridge:
The work he describes overlaps our approach to robust inference from the deep, variable-precision semantics that result from linguistic analysis and disambiguation using the English Resource Grammar (ERG) and the Linguist™.
The title is in tribute to Raj Reddy‘s classic talk about how it’s hard to wreck a nice beach.
I came across interesting work on higher order and semantic dependency parsing today:
So I gave the software a try for the sentence I discussed in this post. The results discussed below were somewhat disappointing but not unexpected. So I tried a well know parser with similar results (also shown below).
There is no surprise here. Both parsers are marvels of machine learning and natural language processing technology. It’s just that understanding is far beyond the ken of even the best NLP. This may be obvious to some, but many are surprised given all the hype about Google and Watson and artificial intelligence or “deep learning” recently.
Consider the following disambiguation result from a user of Automata’s Linguist™.
Continue reading “Properly disambiguating a sentence using the Linguist™”
This is a must-watch video from the Allen Institute for AI for anyone seriously interested in artificial intelligence. It’s 70 minutes long, but worth it. Some of the highlights from my perspective are:
The astute viewer and blog reader will recognize this slide as discussed by Oren Etzioni here.
Thanks to John Sowa‘s comment on LinkedIn for this link which, although slightly dated, contains the following:
In August, I had the chance to speak with Peter Norvig, Director of Google Research, and asked him if he thought that techniques like deep learning could ever solve complicated tasks that are more characteristic of human intelligence, like understanding stories, which is something Norvig used to work on in the nineteen-eighties. Back then, Norvig had written a brilliant review of the previous work on getting machines to understand stories, and fully endorsed an approach that built on classical “symbol-manipulation” techniques. Norvig’s group is now working within Hinton, and Norvig is clearly very interested in seeing what Hinton could come up with. But even Norvig didn’t see how you could build a machine that could understand stories using deep learning alone.
Other quotes along the same lines come from Oren Etzioni in Looking to the Future of Data Science:
This is a significant statement from one of the best people in fact extraction on the planet!
As you know from elsewhere on this blog, I’ve been involved with the precursor to the AIAI (Vulcan’s Project Halo) and am a fan of Watson. But Watson is the best example of what Big Data, Deep Learning, fact extraction, and textual entailment aren’t even close to:
Sure, you can rationalize these things and hope that someday the machine will not need reliable knowledge (or that it will induce enough information with enough certainty). IBM does a lot of this (e.g., see the source of the quotes above). That day may come, but it will happen a lot sooner with curated knowledge.
For those of us that enjoy the intersection of machine learning and natural language, including “deep learning”, which is all the rage, here is an interesting paper on generalizing vector space models of words to broader semantics of English by Jayant Krishnamurthy, a PhD student of Tom Mitchell at Carnegie Mellon University:
Essentially, the paper demonstrates how the features of high-precision lexicalized grammars allow machines to learn the compositional semantics of English. More specifically, the paper demonstrates learning of compositional semantics beyond the capabilities of recurrent neural networks (RNN). In summary, the paper suggests that deep parsing is better than deep learning for understanding the meaning of natural language.
For more information and a different perspective, I recommend the following paper, too:
Note that the authors use Combinatory Categorial Grammar (CCG) while our work uses head-driven phrase structure grammar (HPSG), but this is a minor distinction. For example, compare the logical forms in the Groningen Meaning Bank with the logic produced by the Linguist. The former uses CCG to produce lambda calculus while the latter uses HPSG to produce predicate calculus (ignoring vagaries of under-specified representation which are useful for hypothetical reasoning and textual entailment).
IBM recently posted this video which suggests the relevance of Watson’s capabilities to medical education. The demo uses cases such as occur on the USMLE exam and Waton’s ability to perform evidentiary reason given large bodies of text. The “reasoning paths” followed by Watson in presenting explanations or decision support material use a nice, increasingly popular graphical metaphor.
One intriguing statement in the video concerns Watson “asking itself questions” during the reasoning process. It would be nice to know more about where Watson gets its knowledge about the domain, other than from statistics alone. As I’ve written previously, IBM openly admits that it avoided explicit knowledge in its approach to Jeopardy!
The demo does a nice job with questions in which it is given answers (e.g., multiple choice questions), in particular. I am most impressed, however, with its response on the case beginning at 3 minutes into the video.
Knewton is an interesting company providing a recommendation service for adaptive learning applications. In a recent post, Jonathon Goldman describes an algorithmic approach to generating questions. The approach focuses on improving the manual authoring of test questions (known in the educational realm as “assessment items“). It references work at Microsoft Research on the problem of synthesizing questions for a algebra learning game.
We agree that more automated generation of questions can enrich learning significantly, as has been demonstrated in the Inquire prototype. For information on a better, more broadly applicable approach, see the slides beginning around page 16 in Peter Clark’s invited talk.
What we think is most promising, however, is understanding the reasoning and cognitive skill required to answer questions (i.e., Deep QA). The most automated way to support this is with machine understanding of the content sufficient to answer the questions by proving answers (i.e., multiple choices) right or wrong, as we discuss in this post and this presentation.
Orin Etzioni is a marvelous choice to lead the Allen Institute for AI (aka AI2). The NL/ML path is the right path for scaling up the deep knowledge that Paul Allen’s vision of a Digital Aristotle requires. You can read more about it below and here’s more background on the change in the direction and on some evidence that the path holds great promise.
Benjamin Grosof, co-founder of Coherent Knowledge Systems, is also involved with developing a standard ontology for the financial services industry (i.e., FIBO). In the course of working on FIBO, he is developing a demonstration of defeasible logic concerning Regulation W of the The Federal Reserve Act. Regulation W specifies which transactions involving banks and their affiliates are prohibited under Section 23A of the Act. In the course of doing this, there are various documents which are being captured within the Linguist™ platform. This is a brief note of how those documents can be imported into the platform for curation into formal semantics and logic (as Benjamin and Coherent are doing). Continue reading “Affiliate Transactions covered by The Federal Reserve Act (Regulation W)”