Meta AI introduced Llama 3

Llama 3 is the latest open-source large language model by Meta AI. This new model is set to bring forth improved reasoning and numerous novel capabilities, making it one of the most capable models available to date.

A Quick Overview

Llama 3, the successor of the popular Llama 2, is designed to be the most efficient and capable language model in its class, driving the next wave of innovation in AI. The model is set to be available on major platforms including AWS, Databricks, Google Cloud, Hugging Face, Kaggle, IBM WatsonX, Microsoft Azure, NVIDIA NIM, and Snowflake.

Llama 3 has been integrated into Meta AI, which is now one of the world’s leading AI assistants. It’s available on Facebook, Instagram, WhatsApp, Messenger, and the web. You can visit the Llama 3 website to download the models and reference the Getting Started Guide for a list of all available platforms.

State-of-the-Art Performance

Llama 3’s 8B and 70B parameter models have shown significant improvement over Llama 2, establishing a new standard for LLM models at these scales. Thanks to advancements in pretraining and post-training, these models are currently the best available models at the 8B and 70B parameter scale.

Key Features

The project was designed with four key components in mind:

1. Model Architecture: Llama 3 uses a tokenizer with a vocabulary of 128K tokens, which encodes language with high efficiency, leading to improved model performance.
2. Training Data: Llama 3 is pretrained on over 15T tokens collected from publicly available sources. The training dataset is seven times larger than that used for Llama 2.
3. Scaling up Pretraining: To effectively utilize the pretraining data in Llama 3 models, significant effort was put into scaling up pretraining.
4. Instruction Fine-Tuning: This approach played a major role in unlocking the potential of pretrained models in chat use cases.

Building with Llama 3

The release of Llama 3 provides new trust and safety tools, including updated components like Llama Guard 2 and Cybersec Eval 2, and introduces Code Shield—an inference time guardrail for filtering insecure code produced by LLMs.

Availability

Llama 3 will soon be available on all major platforms, promising to be a ubiquitous presence in the AI landscape. Despite the model having 1B more parameters compared to Llama 2 7B, the improved tokenizer efficiency and Group Query Attention (GQA) maintains the inference efficiency on par with Llama 2 7B.

Future of Llama 3

The Llama 3 8B and 70B models are just the beginning. In the coming months, multiple models with new capabilities including multimodality, multiple language compatibility, a much longer context window, and stronger overall capabilities are expected to be released.

The introduction of Llama 3 marks a new chapter in the development of open AI models. It promises to bring forth improved reasoning and numerous novel capabilities, setting a new standard for AI technology. As always, we look forward to seeing the innovative products and experiences that will be built with Llama 3.