{"id":1662,"date":"2023-04-23T07:27:00","date_gmt":"2023-04-23T12:27:00","guid":{"rendered":"http:\/\/haleyai.com\/wordpress\/?p=1662"},"modified":"2023-04-23T07:36:37","modified_gmt":"2023-04-23T12:36:37","slug":"gpt-under-100000","status":"publish","type":"post","link":"http:\/\/haleyai.com\/wordpress\/2023\/04\/23\/gpt-under-100000\/","title":{"rendered":"GPT under $100,000?"},"content":{"rendered":"\n

For the last several years, we\u2019ve been hearing about how much\nit costs to build ever larger language models. \nToday, a state-of-the-art language model requires approaching a\nmillion-trillion-trillion (1024<\/sup>) arithmetic operations involving\nhundreds of billions of parameters. \nDoing the math, assuming a decent, if older GPU, such as an A100, you\ncome up with how many years this computation will take.  Then you figure out how many GPUs you need\ngiven how many days you have to complete the computation.  For example, Meta recently published that\ntraining a 65 billion parameter version of the LLaMA<\/a> model using over a trillion\ntokens of text took approximately 21 days on roughly 2,000 such GPUs.  That\u2019s almost exactly 1 million hours of GPU\ntime, which can be had for less than $1,000,000.<\/p>\n\n\n\n

So, for $1 million, given a few decent machine learning\nfolks, you could replicate a state-of-the-art language model or build your own\ntweaked to perform better in your domain, such as Bloomberg has done given its\nfinancial market.  Expect to see much\nmore of this from many corners, especially in healthcare, various areas of the\nlife sciences, and others.<\/p>\n\n\n\n

I would like to save the $1 million and start with the 30 or 65 billion parameter LLaMA model rather than train it from scratch.\u00a0 Unfortunately, Meta is not forthcoming with model weights for LLaMA beyond 13 billion parameters.\u00a0 The 13 billion parameter model is impressive enough.\u00a0 The 7B model is not capable enough for me. The 65 billion parameter model would be better, but not twice as good.\u00a0 The 30B parameter model is in the sweet spot.<\/p>\n\n\n\n

Note that if you’re fine with a smaller pre-trained language mode, you could try Stability AI’s LM. These are the folks who brought you Stable Diffusion. They promise to eventually release, for any use including commercial, language models up to 65B. When available, the 15B model may be a good option. For now, I’d like to stick with LLaMA because of some of its significant algorithmic improvements.<\/p>\n\n\n\n

Although available, even the 13 billion parameter model is not openly licensed.\u00a0 As is frustratingly common, the model weights are licensed only for research, not for commercial purposes.\u00a0 Meta invites commercial inquiry but, regrettably, based on my experience, is not eager to respond.\u00a0 So, wanting to use LLaMA commercially, you may have to train your own model.\u00a0 So, let\u2019s talk budgets?<\/p>\n\n\n\n

Training a 13 billion parameter model will cost about 20% of\ntraining a 65-billion parameter version. \nThat will cost less than $200,000. \nYou might be able to cut that cost by half, maybe more, however.  It\u2019s a little dicey, but you can cut back on\nthe training data.  Google\u2019s excellent\nresults from Chinchilla teach us to balance model size with training.  OK, but the truth is that you can get over\n90% of a language model\u2019s final performance with less than half a trillion\ntokens of training data.  <\/p>\n\n\n\n

If you can afford it, you can avoid cutting back on the training data by taking your time.\u00a0 Your language model will be pretty good in 30% of the time Meta took and you can just let it improve over time.\u00a0 That is, start using the model and keep training it, replacing the one you\u2019re using every once and a while. \u00a0This is viable even if you perform fine-tuning (and even reinforcement learning) because the relative costs for such tuning are quite small versus pre-training.\u00a0 <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The bottom line here is that you can build your own 13\nbillion parameter LLaMA for less than $100,000. \nIf you\u2019re going to do millions of transactions, you might not be able to\nafford not to go in this direction!<\/p>\n\n\n\n

LLaMA is essentially an improved version of Open AI\u2019s\nGPT.  LLaMA benefits from various\nalgorithmic improvements since GPT-3 was released a couple years ago.  Recently, Open AI introduce Instruct GPT,\nwhich follows instructions and Chat GPT, which holds conversations.  And GPT has advanced to version 4.<\/p>\n\n\n\n

LLaMA is GPT without the instruction following or\nconversational abilities.  These are\neasy, and inexpensive, to add, however. \nConsider instruction following, for example.  Researchers from Stanford generating tens of\nthousands of simple instructions and results using GPT-4 and trained the 7\nbillion parameter version of LLaMA with them. \nThe dataset is relatively simple, and I thought weak, but was remarkably\neffective.  I was quite surprised how\nwell it follows instructions given only that simple, synthetic dataset. <\/p>\n\n\n\n

On the other hand, it\u2019s not all that surprising, given that\nwe have seen much transfer of learned representations in vision.  The ease of improvement here is simply\nbecause any decent generative language model will quickly adapt to using its\nrepresentation to new linguistic sequences, such as those involving\ninstructions.  It doesn\u2019t have to\nconstruct much new representation to do so.<\/p>\n\n\n\n

The resulting language model is dubbed Alpaca; a cute play on\nwords.  Well, now there is Vicuna<\/a>! \nVicuna takes the 13 billion parameter LLaMA to approach ever more\nclosely to Open AI\u2019s state-of-the-art performance.  According to the researchers from CMU,\nStanford, and the University of California at Berkley and San Diego. <\/p>\n\n\n\n

Look them up.  It\u2019s\nstunning how easily they compete with Google and approach Open AI.  And the training cost to improve LLaMA to \u201cwithin\n10%\u201d of GPT was less than $1,000.  More\nand more is happening on this front.  For\nmore, see Microsoft\u2019s DeepSpeed-Chat<\/a>\n(which may seem odd given their investments in OpenAI, the company.).<\/p>\n","protected":false},"excerpt":{"rendered":"

For the last several years, we\u2019ve been hearing about how much it costs to build ever larger language models.  Today, a state-of-the-art language model requires approaching a million-trillion-trillion (1024) arithmetic operations involving hundreds of billions of parameters.  Doing the math, assuming a decent, if older GPU, such as an A100, you come up with how … <\/p>\n

Continue reading “GPT under $100,000?”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[955],"tags":[971,969,972,922,961,975,974,973],"class_list":["post-1662","post","type-post","status-publish","format-standard","hentry","category-artificial-intelligence","tag-chat-gpt","tag-gpt-4","tag-instruct-gpt","tag-language-model","tag-llama","tag-pre-training","tag-stability-ai","tag-vicuna"],"_links":{"self":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1662","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/comments?post=1662"}],"version-history":[{"count":4,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1662\/revisions"}],"predecessor-version":[{"id":1667,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/posts\/1662\/revisions\/1667"}],"wp:attachment":[{"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/media?parent=1662"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/categories?post=1662"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/haleyai.com\/wordpress\/wp-json\/wp\/v2\/tags?post=1662"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}