{"id":1646,"date":"2023-04-18T19:49:28","date_gmt":"2023-04-19T00:49:28","guid":{"rendered":"http:\/\/haleyai.com\/wordpress\/?p=1646"},"modified":"2023-04-18T20:02:05","modified_gmt":"2023-04-19T01:02:05","slug":"democracy-and-ai","status":"publish","type":"post","link":"http:\/\/haleyai.com\/wordpress\/2023\/04\/18\/democracy-and-ai\/","title":{"rendered":"Democracy and AI"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">This is an earlier version of what became &#8216;<a href=\"https:\/\/www.merlyn.org\/blog\/truly-open-ai-metas-llama-offers-open-sourced-foundation-models\">Truly open AI: Meta\u2019s LLaMA offers open-source foundation models<\/a>&#8216; published at <a href=\"https:\/\/www.merlyn.org\">Merlyn Mind<\/a>.  It was drafted before subsequent developments which I aim to address shortly.   I am solely responsible for the opinions expressed herein.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The ability of large language models has made huge leaps with the launch of ChatGPT in late 2022 and now GPT-4, both from OpenAI (and, effectively, Microsoft). Unfortunately, the GPT family of language models is being held increasingly close to the chest. From GPT-3 on, the models are simply not available other than as hosted services.\u00a0 <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Much has\nbeen written suggesting that AI will become concentrated in a few Big Tech firms\nbecause language modeling at scale has become prohibitively expensive. In\nparticular, we have heard that democratic AI \u2013 whereby the state of the art is\ntruly open and available to all \u2013 is impractical given the amount of compute\nneeded to reach or surpass language models such as GPT-3, Google\u2019s PaLM, and\nothers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Papers on\nGPT-3 have gone into some detail about the proprietary models, such as the\nnumber of layers and attention heads, as well as model width. This has allowed the\nAI community to glean some insights and compare the performance of other models\nagainst GPT. There are many papers comparing smaller models such as DeepMind\u2019s Chinchilla\nagainst GPT-3, and it is not uncommon for the smaller models to outperform their\nlarger sibling.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In mid-March, OpenAI published a <a href=\"https:\/\/arxiv.org\/abs\/2303.08774\">paper describing GPT-4<\/a>, but it gives few details of the model architecture. For example, the number of parameters is not even disclosed. The authors attribute GPT-4\u2019s improvement on exam taking over GPT-3.5 to pre-training methodology, but they explicitly state that no details will be shared for competitive and other reasons. We simply don\u2019t know whether size matters as much in GPT-4 as it did previously. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The\npromise of open source<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Well, let\u2019s step back to Meta\u2019s late 2022 release of Galactica, a model 2\/3 the size of GPT-3 that is trained not on arbitrary internet content but on scientific literature and data. As soon as it was made available, however, it was harshly criticized, even though it is superior to GPT in many regards. The criticism was mostly regarding its \u201ctoxicity,\u201d which is a reasonable concern (though it would be hard to argue that it was more toxic than GPT has been). <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Galactica\nwas promising. The model could be obtained from Meta, unlike those of OpenAI,\nand it could be deployed on readily available and affordable hardware. And,\nagain, it was superior to GPT-3 in various regards. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Well, Meta\nhas upped the ante significantly with the release of <a href=\"https:\/\/research.facebook.com\/publications\/llama-open-and-efficient-foundation-language-models\/\">LLaMA: Open and Efficient Language\nModels<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">LLaMA models\nrange up to 1\/3 the size of GPT-3. They differ by leveraging architectural\nimprovements from the many works of Google, DeepMind, Meta, and others. This\nallows LLaMA to be trained more efficiently and to perform better given\nwhatever training budget.&nbsp; The largest\nLLaMA model competes handily with models three time its size (GPT-3) and eight\ntimes its size (PaLM).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here at\nMerlyn Mind, we\u2019ve given LLaMA a go, and it\u2019s impressive. It\u2019s truly open AI.&nbsp; <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But wait!\nThere\u2019s more \u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">InstructGPT\nand ChatGPT are much better at following instructions and chatting than\nlanguage models that have not been fine-tuned to follow instructions or chat.\nThere is a lot going on to train non-GPT models with such capabilities, but one\neffort, in particular warrants kudos. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">First, let\u2019s\ngiven Anthropic an honorable mention for its work on <a href=\"https:\/\/www.anthropic.com\/index\/measuring-progress-on-scalable-oversight-for-large-language-models\">Constitutional AI<\/a> and related data sets that will\nenable the community to address toxicity and harm in language models.&nbsp; <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now \u2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Stanford\u2019s\nwork on <a href=\"https:\/\/crfm.stanford.edu\/2023\/03\/13\/alpaca.html\">Alpaca: a Strong, Replicable,\nInstruction-Following Model<\/a> is quite fun and helpful. The Stanford team took a small LLaMA model and\ntaught it to follow instructions. The fun part is that they prompted ChatGPT to\ngive it instructions! The result is Alpaca, a fine-tuned LLaMA. It\u2019s worth your\ntime to check it out!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is an earlier version of what became &#8216;Truly open AI: Meta\u2019s LLaMA offers open-source foundation models&#8216; published at Merlyn Mind. It was drafted before subsequent developments which I aim to address shortly. I am solely responsible for the opinions expressed herein. The ability of large language models has made huge leaps with the launch &hellip; <\/p>\n<p class=\"link-more\"><a href=\"http:\/\/haleyai.com\/wordpress\/2023\/04\/18\/democracy-and-ai\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Democracy and AI&#8221;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[955],"tags":[967,966,965,960,963,969,964,961,968,956,962],"class_list":["post-1646","post","type-post","status-publish","format-standard","hentry","category-artificial-intelligence","tag-alpaca","tag-anthropic","tag-chatgpt","tag-constitutional-ai","tag-galactica","tag-gpt-4","tag-instructgpt","tag-llama","tag-meta","tag-open-ai","tag-palm"],"_links":{"self":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1646","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/comments?post=1646"}],"version-history":[{"count":9,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1646\/revisions"}],"predecessor-version":[{"id":1657,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1646\/revisions\/1657"}],"wp:attachment":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/media?parent=1646"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/categories?post=1646"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/tags?post=1646"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}