Presented by many as one of the greatest European hopes in the AI race, the French startup Mistral AI has just published a model that challenges GPT-4 (OpenAI) and Llama (Facebook). Her name ? Mixtral 8x7B.
The message, at first glance, is incomprehensible. On December 8, Mistral AI published a seemingly meaningless sequence of letters and numbers on. No word from the French company specializing in artificial intelligence details the meaning of its publication, which almost looks like a copy and paste error.
In reality, this gibberish is a magnet link, which is commonly used in peer-to-peer exchanges, via specific software. In this case, this link can be opened with a BitTorrent client – like qBittorrent – by copying/pasting it. Then, the link gives access to downloading a large file. Allow 87 GB of space on your PC.
Mixtral 8x7B, the new AI model from Mistral AI
These 87 GB are the space that Mixtral 8x7B will occupy. This is the new model of the French startupwhich she describes as a “ sparse mix of high-quality expert models (SMoE) with open weights “. Above all, Mistral emphasizes that to date this is the “ most powerful open weight model with a permissive license. »
Mistral AI has closed Monday December 11 a second fundraising, with 385 million euros at stake. In June, it had obtained 105 million euros. The valuation of the startup would be around 2 billion euros.
Open source forms one of the specificities of Mistral AI’s strategy in artificial intelligence. The company defends a logic of openness and sharing of models for the good of generative AI. It’s the ” safest way to fight censorship and bias in technology that shapes our future » she argued in September 2023.
Among the merits that Mistral AI highlights about Mixtral 8x7B are the understanding of several languages (English, French, Italian, German, Spanish) and computer code generation “ very efficient “. On a technical level, Mistral AI also highlights its ability to adapt to follow more specific instructions.
Fine-tuning a language model helps train it in a more specialized way on data or a task. The goal is for the model to become more effective for working in a targeted area. Mistral AI claims a rating of 8.3 on the MT-Bench test bench. For comparison, GPT-4 Turbo, OpenAI’s newest model for ChatGPT, achieves 9.32.
Mistral AI adds that Mixtral 8x7B manages a context of 32,000 tokens. Clearly, the model is supposed to process and understand a context of 32,000 tokens — here, this refers to units of text, which can be words, parts of words, or even characters, depending on how the model has been trained. The larger this capacity, the better.
Trained from data extracted from the web and freely accessible (the nature of this data is not specified by Mistral AI), the startup’s new generative AI architecture includes 45 billion parameters in total, but does not only uses 12 billion per token. The company uses a process that allows it to remain efficient, but with lower cost and latency.
A superior model to ChatGPT (GPT-3.5) and Llama 2
The company founded in May 2023 by three French people — Arthur Mensch, Guillaume Lample and Timothée Lacroix — is keen to set its latest product apart from the competition. Llama 2 70B from Meta (Facebook)? It outperforms on most tests, with processing six times faster. GPT-3.5 from OpenAI? It is equalized or exceeded on most reference points.
To support its claims, the startup shared several tables and graphs to demonstrate the superiority of Mixtral 8x7B on different criteria. Overall, Mistral AI considers to have the best model “ regarding cost/performance trade-offs “. In any case, compared to Facebook’s Llama 2 models, which are also open source.
Even on the issue of biases and hallucinations (an AI is said to hallucinate when it responds completely off the mark), Mistral AI is supposed to do better with Mixtral 8x7B. Based on the TruthfulQA test bench, Mixtral would thus be more reliable (73.9%) than Llama 2 (50.2%). It would also have less bias according to the BBQ test and would be more positive, according to the BOLD tests.
No comparison with the most recent models
If the performances displayed by Mistral AI regarding Mixtral 8x7B are notable, it should however be noted that the two points of comparison selected are now relatively old. GPT-3.5, in particular, is a model released in the month of November 2022. Llama 2 is a bit newer, since this model debuted in July 2023.
OpenAI released two newer models in 2023, with GPT-4 in the spring and GPT-4 Turbo in the fall. We don’t know what a head-to-head matchup would be like between one of these two models and Mixtral 8x7B. We can however note that, according to MT-Bench, has a score around 9 and GPT-4 exceeds 9.3. Clearly, the tool may not yet completely hold its own against OpenAI.
The fact remains that if Mixtral 8x7B is not GPT-4, it is undoubtedly the new open source reference in terms of model. At least, for now. Indeed, the sector has been experiencing great excitement over the past year and the positions acquired today cannot be retained without regular development. We see this with ChatGPT, which has received several updates.
A challenge that Mistral AI is undoubtedly not unaware of. Its first model, Mistral 7B, was released in September. The new one, Mixtral 8x7B, was released two and a half months later. This is the reality of the sector from now on: having a model to establish the foundations of generative AI is no longer enough: we must generate constantly new AI to stay in the race.