{"id":1625,"date":"2019-09-02T12:47:04","date_gmt":"2019-09-02T17:47:04","guid":{"rendered":"http:\/\/haleyai.com\/wordpress\/?p=1625"},"modified":"2019-09-02T12:47:04","modified_gmt":"2019-09-02T17:47:04","slug":"cpc-and-the-grandmother-neuron","status":"publish","type":"post","link":"http:\/\/haleyai.com\/wordpress\/2019\/09\/02\/cpc-and-the-grandmother-neuron\/","title":{"rendered":"CPC and the Grandmother Neuron"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A lot of recent work has advanced the learning of increasingly context-sensitive distributed representations (i.e., so-called &#8217;embeddings&#8217;).  In particular.  DeepMind&#8217;s <a rel=\"noreferrer noopener\" aria-label=\"paper  (opens in a new tab)\" href=\"https:\/\/arxiv.org\/abs\/1807.03748\" target=\"_blank\">paper <\/a>on &#8220;Contrastive Predictive Coding&#8221; (CPC) is particularly interesting and advances on a number of fronts.  For example,   in  <a rel=\"noreferrer noopener\" aria-label=\"wav2vec (opens in a new tab)\" href=\"https:\/\/arxiv.org\/abs\/1904.05862\" target=\"_blank\">wav2vec<\/a>, Facebook AI Research (FAIR) uses CPC to obtain apparently superior acoustic modeling results  to DeepSpeech&#8217;s  connectionist temporal classification (CTC) approach.  In the CPC paper, the following image is particularly striking, harkening back to the early notion of a <a rel=\"noreferrer noopener\" aria-label=\"Grandmother Cell (opens in a new tab)\" href=\"https:\/\/en.wikipedia.org\/wiki\/Grandmother_cell\" target=\"_blank\">Grandmother Cell<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"649\" src=\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/Contrastive-Predictive-Coding-grandmother-neurons-1024x649.png\" alt=\"\" class=\"wp-image-1626\" srcset=\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/Contrastive-Predictive-Coding-grandmother-neurons-1024x649.png 1024w, http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/Contrastive-Predictive-Coding-grandmother-neurons-300x190.png 300w, http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/Contrastive-Predictive-Coding-grandmother-neurons-768x487.png 768w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><figcaption>grandmother cells resulting from CPC<br \/><\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>A lot of recent work has advanced the learning of increasingly context-sensitive distributed representations (i.e., so-called &#8217;embeddings&#8217;). In particular. DeepMind&#8217;s paper on &#8220;Contrastive Predictive Coding&#8221; (CPC) is particularly interesting and advances on a number of fronts. For example, in wav2vec, Facebook AI Research (FAIR) uses CPC to obtain apparently superior acoustic modeling results to DeepSpeech&#8217;s &hellip; <\/p>\n<p class=\"link-more\"><a href=\"http:\/\/haleyai.com\/wordpress\/2019\/09\/02\/cpc-and-the-grandmother-neuron\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;CPC and the Grandmother Neuron&#8221;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[195,1],"tags":[950,947,949,945,946,809,915,954,952,953,948,943,942,951],"class_list":["post-1625","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-uncategorized","tag-acoustic-modeling","tag-connectionist-temporal-classification","tag-contrastive-predictive-coding","tag-cpc","tag-ctc","tag-deepmind","tag-deepspeech","tag-embeddings","tag-facebook-ai-research","tag-fair","tag-grandmother-cell","tag-knowledge-representation","tag-unsupervised-learning","tag-wav2vec"],"_links":{"self":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1625","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/comments?post=1625"}],"version-history":[{"count":1,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1625\/revisions"}],"predecessor-version":[{"id":1627,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1625\/revisions\/1627"}],"wp:attachment":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/media?parent=1625"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/categories?post=1625"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/tags?post=1625"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}