Jerome Pesenti has a few reasons to celebrate Meta’s decision last week to release Llama 3, a powerful, open-source language model for large languages that anyone can download, run, and build on.
Pesenti was formerly vice president of artificial intelligence at Meta and says he often pushed the company to think about releasing its technology so others could use and build on it. But his main reason for celebration is that his new startup is getting access to an AI model that he says is very close in performance to OpenAI’s industry-leading text generator GPT-4, but is significantly less expensive to run and more open to external review and modification.
“Last Friday’s release really feels like a game-changer,” says Pesenti. His new company, Sizzle, an AI tutor, is currently using GPT-4 and other closed and open AI models to create problem sets and curricula for students. Its engineers are examining whether Llama 3 could replace OpenAI’s model in many cases.
Sizzle’s story could promise a broader shift in the balance of power in AI. OpenAI changed the world with ChatGPT, sparking a wave of AI investment and attracting more than 2 million developers to its cloud APIs. However, if open source models prove competitive, developers and entrepreneurs may decide to stop paying for access to the latest model from OpenAI or Google and instead choose Llama 3 or one of the other increasingly powerful open source models to use that come up.
“It’s going to be an interesting horse race,” Pesenti says of the competition between open models like Llama 3 and closed models like GPT-4 and Google’s Gemini.
Meta’s previous model, Llama 2, was already influential, but the company says it has made the latest version more powerful by feeding larger amounts of higher quality training data and developing new techniques to filter out redundant or garbled content and select the best Mixture of data sets to be used.
According to Pesenti, running Llama 3 on a cloud platform like Fireworks.ai costs one-twentieth of the cost of accessing GPT-4 via an API. He adds that Llama 3 can be configured to respond extremely quickly to requests – an important aspect for developers at companies like his that rely on models from different vendors. “It’s an equation between latency, cost and accuracy,” he says.
Open models seem to be falling at an impressive rate. A few weeks ago, I visited startup Databricks to witness the final stages of its efforts to develop DBRX, a language model that was briefly the best open around. That crown now belongs to Lama 3. Ali Ghodsi, CEO of Databricks, also describes Llama 3 as “game-changing,” saying that the larger model “reaches the quality of GPT 4 – leveling the playing field between open- and closed-source LLMs .”
Llama 3 also shows the potential to make AI models smaller so they can run on less powerful hardware. Meta has released two versions of its latest model, one with 70 billion parameters – a measure of the variables it uses to learn from training data – and one with 8 billion. The smaller model is compact enough to run on a laptop but is remarkably powerful, at least in WIRED’s tests.
Two days before Meta’s release, Mistral, a French AI company founded by graduates of Pesenti’s team at Meta, released Mixtral 8x22B as an open source solution. It has 141 billion parameters but only uses 39 billion of them at a time, a design known as expert mix. Thanks to this trick, the model is much more powerful than some models that are much larger.
Meta isn’t the only tech giant releasing open source AI. This week, Microsoft released Phi-3-mini and Apple OpenELM, two small but powerful free-to-use language models that can run on a smartphone.
The coming months will show whether Llama 3 and other open models can truly displace premium AI models like GPT-4 for some developers. And even more powerful open source AI is coming. The company is working on a massive 400 billion-parameter version of Llama 3, which chief AI scientist Yann LeCun says will be one of the most powerful in the world.
Of course, this openness is not purely altruistic. Meta CEO Mark Zuckerberg says he will open up his AI models should ultimately benefit the company by reducing the cost of the technologies on which it relies, for example by creating compatible tools and services that Meta can use for itself. He failed to mention that it could also be beneficial for Meta to prevent OpenAI, Microsoft or Google from dominating the field.