When OpenAI unveiled its GPT-3 engine and showed how its AI model could generate sensible text seemingly out of nothing based on a few cues, it set a new limit in how much processing and memory a language model could use. At 175 billion trainable parameters, GPT-3 is enormous. But it’s time at the top of the supermodels did not last long. Google, which introduced the Transformer concept on which many of today’s AI language models are based, soared past it earlier this year with the 1.6 trillion-parameter Switch.
At his company’s Spring Processor Conference in April, analyst Linley Gwennap claimed language models are currently growing at the rate of around 40 times a year. Apparently, the rapid growth is worth it and not just in natural language processing. The results from other areas, such as image recognition, show size improves overall accuracy though the rate of growth is a mere doubling each year for those types of machine-learning model.
Short...