Llama 3.1论文中英对照 - cold_moon - 博客园 (2024)

论文来自： https://ai.meta.com/research/publications/the-llama-3-herd-of-models/

Llama Team, Al @ Meta \({}^{1}\)

Llama 团队，Meta @ Al \({}^{1}\)

\({}^{1}\) A detailed contributor list can be found in the appendix of this paper.

\({}^{1}\) 详细的贡献者名单可在本文附录中找到。

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with \({405}\mathrm{\;B}\) parameters and a context window of up to \({128}\mathrm{\;K}\) tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

现代人工智能（AI）系统由基础模型驱动。本文介绍了一组新的基础模型，称为 Llama 3。它是一群原生支持多语言、编码、推理和工具使用的语言模型。我们最大的模型是一个密集型 Transformer，具有 \({405}\mathrm{\;B}\) 参数和高达 \({128}\mathrm{\;K}\) 个令牌的上下文窗口。本文对 Llama 3 进行了广泛的实证评估。我们发现 Llama 3 在众多任务上与 GPT-4 等领先语言模型相比质量相当。我们公开发布了 Llama 3，包括 405B 参数语言模型的预训练和后训练版本以及用于输入和输出安全的 Llama Guard 3 模型。本文还展示了通过组合方法将图像、视频和语音功能集成到 Llama 3 中的实验结果。我们观察到这种方法在图像、视频和语音识别任务上与最先进的技术表现竞争。由于这些模型仍在开发中，因此尚未广泛发布。

Date: July 23, 2024

日期：2024年7月23日

Website: https://llama.meta.com/

网站：https://llama.meta.com/

1 Introduction 引言

Foundation models are general models of language, vision, speech, and/or other modalities that are designed to support a large variety of AI tasks. They form the basis of many modern AI systems.

基础模型是用于语言、视觉、语音及其他模态的通用模型，旨在支持多种AI任务。它们构成了许多现代AI系统的基础。

The development of modern foundation models consists of two main stages: (1) a pre-training stage in which the model is trained at massive scale using straightforward tasks such as next-word prediction or captioning and (2) a post-training stage in which the model is tuned to follow instructions, align with human preferences, and improve specific capabilities (for example, coding and reasoning).

现代基础模型的开发包括两个主要阶段：（1）预训练阶段，模型通过简单的任务如下一个词预测或字幕生成进行大规模训练；（2）后训练阶段，模型调整以遵循指令、与人类偏好对齐并提升特定能力（例如，编码和推理）。

In this paper, we present a new set of foundation models for language, called LIama 3. The Llama 3 Herd of models natively supports multilinguality, coding, reasoning, and tool usage. Our largest model is dense Transformer with \({405}\mathrm{\;B}\) parameters,processing information in a context window of up to \({128}\mathrm{\;K}\) tokens. Each member of the herd is listed in Table 1. All the results presented in this paper are for the Llama 3.1 models, which we will refer to as Llama 3 throughout for brevity.

在本文中，我们提出了一套新的语言基础模型，称为 LIama 3。Llama 3 模型群天然支持多语言、编码、推理和工具使用。我们最大的模型是密集型 Transformer，具有 \({405}\mathrm{\;B}\) 参数，在最多 \({128}\mathrm{\;K}\) 个词元的上下文窗口中处理信息。模型群中的每个成员均列于表 1。本文展示的所有结果均针对 Llama 3.1 模型，为简洁起见，全文将简称其为 Llama 3。

We believe there are three key levers in the development of high-quality foundation models: data, scale, and managing complexity. We seek to optimize for these three levers in our development process:

我们认为，高质量基础模型的发展有三个关键杠杆：数据、规模和管理复杂性。我们在开发过程中力求优化这三个方面：

• Data. Compared to prior versions of Llama (Touvron et al., 2023a,b), we improved both the quantity and quality of the data we use for pre-training and post-training. These improvements include the development of more careful pre-processing and curation pipelines for pre-training data and the development of more rigorous quality assurance and filtering approaches for post-training data. We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.

• 数据。与之前的 Llama 版本（Touvron 等人，2023a,b）相比，我们不仅提高了预训练和后训练所用数据的数量，还提升了其质量。这些改进包括开发更细致的预训练数据预处理和筛选流程，以及开发更严格的后训练数据质量保证和过滤方法。我们使用约 15T 的多语言词元对 Llama 3 进行预训练，而 Llama 2 的预训练数据量为 1.8T 词元。

• Scale. We train a model at far larger scale than previous Llama models: our flagship language model was pre-trained using \({3.8} \times {10}^{25}\) FLOPs,almost \({50} \times\) more than the largest version of Llama 2. Specifically, we pre-trained a flagship model with \({405}\mathrm{\;B}\) trainable parameters on \({15.6}\mathrm{\;T}\) text tokens. As expected per scaling laws for foundation models, our flagship model outperforms smaller models trained using the same procedure. While our scaling laws suggest our flagship model is an approximately compute-optimal size for our training budget, we also train our smaller models for much longer than is compute-optimal. The resulting models perform better than compute-optimal models at the same inference budget. We use the flagship model to further improve the quality of those smaller models during post-training.

• 规模。我们训练的模型规模远超以往的 Llama 模型：我们的旗舰语言模型使用 \({3.8} \times {10}^{25}\) FLOPs 进行预训练，几乎比 Llama 2 的最大版本多 \({50} \times\)。具体来说，我们在 \({15.6}\mathrm{\;T}\) 个文本标记上预训练了一个具有 \({405}\mathrm{\;B}\) 个可训练参数的旗舰模型。根据基础模型的缩放定律预期，我们的旗舰模型优于使用相同程序训练的较小模型。尽管我们的缩放定律表明我们的旗舰模型对于我们的训练预算来说是一个近似计算最优的大小，但我们的小型模型训练时间远超过计算最优时间。因此，这些模型在相同的推理预算下表现优于计算最优模型。我们使用旗舰模型在训练后进一步提高那些较小模型的质量。

Table 1 Overview of the Llama 3 Herd of models. All results in this paper are for the Llama 3.1 models.

表1 Llama 3 模型群概览。本文中的所有结果均针对 Llama 3.1 模型。

• Managing complexity. We make design choices that seek to maximize our ability to scale the model development process. For example, we opt for a standard dense Transformer model architecture (Vaswani et al., 2017) with minor adaptations, rather than for a mixture-of-experts model (Shazeer et al., 2017) to maximize training stability. Similarly, we adopt a relatively simple post-training procedure based on supervised finetuning (SFT), rejection sampling (RS), and direct preference optimization (DPO; Rafailov et al. (2023)) as opposed to more complex reinforcement learning algorithms (Ouyang et al., 2022; Schulman et al., 2017) that tend to be less stable and harder to scale.

• 管理复杂性。我们做出设计选择，以最大化我们扩展模型开发过程的能力。例如，我们选择了一个标准的密集 Transformer 模型架构（Vaswani 等人，2017），并进行了微小的调整，而不是选择混合专家模型（Shazeer 等人，2017）以最大化训练稳定性。同样，我们采用了一个相对简单的训练后程序，基于监督微调（SFT）、拒绝采样（RS）和直接偏好优化（DPO；Rafailov 等人（2023）），而不是更复杂的强化学习算法（Ouyang 等人，2022；Schulman 等人，2017），这些算法往往稳定性较差且更难扩展。

The result of our work is Llama 3: a herd of three multilingual \({}^{1}\) language models with \(8\mathrm{\;B},{70}\mathrm{\;B}\) ,and \({405}\mathrm{\;B}\) parameters. We evaluate the performance of Llama 3 on a plethora of benchmark datasets that span a wide range of language understanding tasks. In addition, we perform extensive human evaluations that compare Llama 3 with competing models. An overview of the performance of the flagship Llama 3 model on key benchmarks is presented in Table 2. Our experimental evaluation suggests that our flagship model performs on par with leading language models such as GPT-4 (OpenAI, 2023a) across a variety of tasks, and is close to matching the state-of-the-art. Our smaller models are best-in-class, outperforming alternative models with similar numbers of parameters (Bai et al., 2023; Jiang et al., 2023). Llama 3 also delivers a much better balance between helpfulness and harmlessness than its predecessor (Touvron et al., 2023b). We present a detailed analysis of the safety of Llama 3 in Section 5.4.

我们的工作成果是 Llama 3：一个由三个多语言 \({}^{1}\) 语言模型组成的群体，具有 \(8\mathrm{\;B},{70}\mathrm{\;B}\) 和 \({405}\mathrm{\;B}\) 参数。我们在涵盖广泛语言理解任务的大量基准数据集上评估了 Llama 3 的性能。此外，我们还进行了广泛的人工评估，将 Llama 3 与竞争模型进行比较。旗舰 Llama 3 模型在关键基准上的性能概览见表 2。我们的实验评估表明，我们的旗舰模型在各种任务上与 GPT-4（OpenAI，2023a）等领先语言模型表现相当，并且接近达到最先进水平。我们的小型模型表现最佳，优于具有相似参数数量的替代模型（Bai 等，2023；Jiang 等，2023）。Llama 3 在有用性和无害性之间提供了比其前身更好的平衡（Touvron 等，2023b）。我们在第 5.4 节中详细分析了 Llama 3 的安全性。

We are publicly releasing all three Llama 3 models under an updated version of the Llama 3 Community License; see https://llama.meta.com. This includes pre-trained and post-trained versions of our 405B parameter language model and a new version of our Llama Guard model (Inan et al., 2023) for input and output safety. We hope that the open release of a flagship model will spur a wave of innovation in the research community, and accelerate a responsible path towards the development of artificial general intelligence (AGI).

我们根据更新版本的 Llama 3 社区许可证公开发布了所有三个 Llama 3 模型；详情请参见 https://llama.meta.com。这包括我们 405B 参数语言模型的预训练和后训练版本，以及我们 Llama Guard 模型（Inan 等，2023）的新版本，用于输入和输出安全。我们希望旗舰模型的开放发布将激发研究社区的创新浪潮，并加速通往负责任的人工通用智能（AGI）发展之路。

As part of the Llama 3 development process we also develop multimodal extensions to the models, enabling image recognition, video recognition, and speech understanding capabilities. These models are still under active development and not yet ready for release. In addition to our language modeling results, the paper presents results of our initial experiments with those multimodal models.

作为Llama 3开发过程的一部分，我们还开发了多模态扩展模型，使其具备图像识别、视频识别和语音理解能力。这些模型仍在积极开发中，尚未准备好发布。除了我们的语言建模成果外，本文还展示了我们与这些多模态模型初步实验的结果。

\({}^{1}\) The Llama 38B and 70B were pre-trained on multilingual data but were intended for use in English at the time.

\({}^{1}\) Llama 38B和70B在多语言数据上进行了预训练，但当时旨在用于英语。

Table 2 Performance of finetuned Llama 3 models on key benchmark evaluations. The table compares the performance of the \(8\mathrm{\;B},{70}\mathrm{\;B}\) ,and \({405}\mathrm{\;B}\) versions of Llama 3 with that of competing models. We boldface the best-performing model in each of three model-size equivalence classes. A Results obtained using 5-shot prompting (no CoT). Results obtained without CoT. \({}^{\diamond }\) Results obtained using zero-shot prompting.

表2 微调后的Llama 3模型在关键基准评估中的性能。该表比较了Llama 3的 \(8\mathrm{\;B},{70}\mathrm{\;B}\)和\({405}\mathrm{\;B}\) 版本与竞争模型的性能。我们在每个三个模型大小等价类别中加粗显示了表现最佳的模型。A 使用5次提示（无CoT）获得的结果。无CoT获得的结果。\({}^{\diamond }\) 使用零次提示获得的结果。

2 General Overview 总体概述

The model architecture of Llama 3 is illustrated in Figure 1. The development of our Llama 3 language models comprises two main stages:

Llama 3的模型架构如图1所示。我们的Llama 3语言模型的开发包括两个主要阶段：

• Language model pre-training. We start by converting a large, multilingual text corpus to discrete tokens and pre-training a large language model (LLM) on the resulting data to perform next-token prediction. In the language model pre-training stage, the model learns the structure of language and obtains large amounts of knowledge about the world from the text it is "reading". To do this effectively, pre-training is performed at massive scale: we pre-train a model with \({405}\mathrm{\;B}\) parameters on \({15.6}\mathrm{\;T}\) tokens using a context window of \(8\mathrm{\;K}\) tokens. This standard pre-training stage is followed by a continued pre-training stage that increases the supported context window to \({128}\mathrm{\;K}\) tokens. See Section 3 for details.

• 语言模型预训练。我们首先将一个大型多语言文本语料库转换为离散的标记，并在生成的数据上预训练一个大型语言模型（LLM）以进行下一个标记预测。在语言模型预训练阶段，模型学习语言的结构，并从其“阅读”的文本中获取大量关于世界的知识。为了有效实现这一点，预训练是在大规模上进行的：我们在\({405}\mathrm{\;B}\)个参数上预训练一个模型，使用\({15.6}\mathrm{\;T}\)个标记的上下文窗口为\(8\mathrm{\;K}\)个标记。这一标准预训练阶段之后是一个继续预训练阶段，将支持的上下文窗口增加到\({128}\mathrm{\;K}\)个标记。详见第3节。

• Language model post-training. The pre-trained language model has a rich understanding of language but it does not yet follow instructions or behave in the way we would expect an assistant to. We align the model with human feedback in several rounds, each of which involves supervised finetuning (SFT) on instruction tuning data and Direct Preference Optimization (DPO; Rafailov et al., 2024). At this post-training \({}^{2}\) stage,we also integrate new capabilities,such as tool-use,and observe strong improvements in other areas, such as coding and reasoning. See Section 4 for details. Finally, safety mitigations are also incorporated into the model at the post-training stage, the details of which are described in Section 5.4.

• 语言模型后训练。预训练的语言模型对语言有丰富的理解，但它还不遵循指令或以我们期望助手的方式行事。我们通过几轮与人类反馈的对齐，每一轮都包括在指令调优数据上的监督微调（SFT）和直接偏好优化（DPO；Rafailov et al., 2024）。在这个后训练 \({}^{2}\) 阶段，我们还集成了新的能力，例如工具使用，并在其他领域如编码和推理方面观察到显著改进。详见第4节。最后，安全缓解措施也在后训练阶段被纳入模型，详情见第5.4节。

The resulting models have a rich set of capabilities. They can answer questions in at least eight languages, write high-quality code, solve complex reasoning problems, and use tools out-of-the-box or in a zero-shot way.

由此产生的模型具有丰富的能力。它们至少能用八种语言回答问题，编写高质量的代码，解决复杂的推理问题，并能开箱即用地或以零样本方式使用工具。

We also perform experiments in which we add image, video, and speech capabilities to Llama 3 using a compositional approach. The approach we study comprises the three additional stages illustrated in Figure 28:

我们还进行了实验，通过组合方法将图像、视频和语音能力添加到Llama 3中。我们研究的方法包括图28所示的三个额外阶段：

• Multi-modal encoder pre-training. We train separate encoders for images and speech. We train our image encoder on large amounts of image-text pairs. This teaches the model the relation between visual content and the description of that content in natural language. Our speech encoder is trained using a

• 多模态编码器预训练。我们为图像和语音训练单独的编码器。我们在大量的图像-文本对上训练我们的图像编码器。这教会了模型视觉内容与自然语言中对该内容的描述之间的关系。我们的语音编码器是使用

\({}^{2}\) In this paper,we use the term "post-training" to refer to any model training that happens outside of pre-training.

\({}^{2}\) 在本文中，我们使用术语“后训练”来指代在预训练之外进行的任何模型训练。

Figure 1 Illustration of the overall architecture and training of Llama 3. Llama \(3\) is a Transformer language model trained to predict the next token of a textual sequence. See text for details.

图1 Llama 3的整体架构和训练示意图。Llama \(3\) 是一个Transformer语言模型，训练用于预测文本序列的下一个词元。详见正文。

self-supervised approach that masks out parts of the speech inputs and tries to reconstruct the masked out parts via a discrete-token representation. As a result, the model learns the structure of speech signals. See Section 7 for details on the image encoder and Section 8 for details on the speech encoder.

一种自监督方法，该方法掩盖语音输入的部分内容，并尝试通过离散标记表示来重建被掩盖的部分。因此，模型学习了语音信号的结构。有关图像编码器的详细信息，请参见第7节；有关语音编码器的详细信息，请参见第8节。

• Vision adapter training. We train an adapter that integrates the pre-trained image encoder into the pre-trained language model. The adapter consists of a series of cross-attention layers that feed image-encoder representations into the language model. The adapter is trained on text-image pairs. This aligns the image representations with the language representations. During adapter training, we also update the parameters of the image encoder but we intentionally do not update the language-model parameters. We also train a video adapter on top of the image adapter on paired video-text data. This enables the model to aggregate information across frames. See Section 7 for details.

• 视觉适配器训练。我们训练一个适配器，将预训练的图像编码器集成到预训练的语言模型中。该适配器由一系列交叉注意力层组成，这些层将图像编码器表示输入到语言模型中。适配器在文本-图像对上进行训练。这使得图像表示与语言表示对齐。在适配器训练期间，我们还更新图像编码器的参数，但我们有意不更新语言模型的参数。我们还在图像适配器之上对视频-文本数据对进行视频适配器训练。这使得模型能够跨帧聚合信息。详细信息请参见第7节。

• Speech adapter training. Finally, we integrate the speech encoder into the model via an adapter that converts speech encodings into token representations that can be fed directly into the finetuned language model. The parameters of the adapter and encoder are jointly updated in a supervised finetuning stage to enable high-quality speech understanding. We do not change the language model during speech adapter training. We also integrate a text-to-speech system. See Section 8 for details.

• 语音适配器训练。最后，我们通过一个适配器将语音编码器集成到模型中，该适配器将语音编码转换为可以直接输入到微调语言模型的标记表示。在监督微调阶段，适配器和编码器的参数被联合更新，以实现高质量的语音理解。在语音适配器训练期间，我们不改变语言模型。我们还集成了一个文本到语音系统。详细信息请参见第8节。

Our multimodal experiments lead to models that can recognize the content of images and videos, and support interaction via a speech interface. These models are still under development and not yet ready for release.

我们的多模态实验产生了能够识别图像和视频内容并支持通过语音界面进行交互的模型。这些模型仍在开发中，尚未准备好发布。

3 Pre-Training 预训练

Language model pre-training involves: (1) the curation and filtering of a large-scale training corpus, (2) the development of a model architecture and corresponding scaling laws for determining model size, (3) the development of techniques for efficient pre-training at large scale, and (4) the development of a pre-training recipe. We present each of these components separately below.

语言模型预训练涉及：(1) 大规模训练语料库的策划和过滤，(2) 模型架构的开发以及确定模型大小的相应缩放法则，(3) 大规模高效预训练技术的开发，以及 (4) 预训练方案的开发。我们将在下面分别介绍这些组件。

3.1 Pre-Training Data 预训练数据

We create our dataset for language model pre-training from a variety of data sources containing knowledge until the end of 2023. We apply several de-duplication methods and data cleaning mechanisms on each data source to obtain high-quality tokens. We remove domains that contain large amounts of personally identifiable information (PII), and domains with known adult content.

我们从多种数据源中创建用于语言模型预训练的数据集，这些数据源包含截至2023年底的知识。我们对每个数据源应用多种去重方法和数据清洗机制以获取高质量的令牌。我们移除了包含大量个人身份信息（PII）的领域，以及已知包含成人内容的领域。

3.1.1 Web Data Curation 网络数据精选

Much of the data we utilize is obtained from the web and we describe our cleaning process below.

我们使用的许多数据来自网络，下面描述我们的清洗过程。

PH and safety filtering. Among other mitigations, we implement filters designed to remove data from websites are likely to contain unsafe content or high volumes of PII, domains that have been ranked as harmful according to a variety of Meta safety standards, and domains that are known to contain adult content.

PH和安全过滤。除了其他缓解措施外，我们实施了旨在移除可能包含不安全内容或大量PII的网站数据的过滤器，根据多种Meta安全标准被评定为有害的域名，以及已知包含成人内容的域名。

Text extraction and cleaning. We process the raw HTML content for non-truncated web documents to extract high-quality diverse text. To do so, we build a custom parser that extracts the HTML content and optimizes for precision in boilerplate removal and content recall. We evaluate our parser's quality in human evaluations, comparing it with popular third-party HTML parsers that optimize for article-like content, and found it to perform favorably. We carefully process HTML pages with mathematics and code content to preserve the structure of that content. We maintain the image alt attribute text since mathematical content is often represented as pre-rendered images where the math is also provided in the alt attribute. We experimentally evaluate different cleaning configurations. We find markdown is harmful to the performance of a model that is primarily trained on web data compared to plain text, so we remove all markdown markers.

文本提取和清洗。我们对非截断的网页文档的原始HTML内容进行处理，以提取高质量的多样化文本。为此，我们构建了一个自定义解析器，用于提取HTML内容并优化样板删除和内容召回的精确度。我们通过人工评估来评估解析器的质量，将其与优化文章类内容的流行第三方HTML解析器进行比较，发现其表现更优。我们仔细处理包含数学和代码内容的HTML页面，以保留这些内容的结构。我们保留图像alt属性文本，因为数学内容通常以预渲染图像表示，其中数学内容也提供在alt属性中。我们实验性地评估不同的清洗配置。我们发现，与纯文本相比，Markdown对主要基于网络数据训练的模型的性能有害，因此我们移除了所有Markdown标记。

De-duplication. We apply several rounds of de-duplication at the URL, document, and line level:

去重。我们在URL、文档和行级别应用多轮去重：

• URL-level de-duplication. We perform URL-level de-duplication across the entire dataset. We keep the most recent version for pages corresponding to each URL.

• URL级别去重。我们对整个数据集进行URL级别的去重。对于每个URL对应的页面，我们保留最新版本。

• Document-level de-duplication. We perform global MinHash (Broder, 1997) de-duplication across the entire dataset to remove near duplicate documents.

• 文档级去重。我们对整个数据集执行全局MinHash（Broder，1997）去重，以删除近似重复的文档。

• Line-level de-duplication. We perform aggressive line-level de-duplication similar to coNet (Wenzek et al., 2019). We remove lines that appeared more than 6 times in each bucket of 30M documents. Although our manual qualitative analysis showed that the line-level de-duplication removes not only leftover boilerplate from various websites such as navigation menus, cookie warnings, but also frequent high-quality text, our empirical evaluations showed strong improvements.

• 行级去重。我们执行激进的行级去重，类似于coNet（Wenzek等，2019）。我们删除在每个30M文档桶中出现超过6次的行。尽管我们的手动定性分析显示，行级去重不仅删除了来自各种网站的残留样板文件，如导航菜单、cookie警告，还删除了频繁的高质量文本，但我们的实证评估显示了显著的改进。

Heuristic filtering. We develop heuristics to remove additional low-quality documents, outliers, and documents with excessive repetitions. Some examples of heuristics include:

启发式过滤。我们开发启发式方法来删除额外的低质量文档、异常值和包含过多重复内容的文档。一些启发式方法的例子包括：

• We use duplicated n-gram coverage ratio (Rae et al., 2021) to remove lines that consist of repeated content such as logging or error messages. Those lines could be very long and unique, hence cannot be filtered by line-dedup.

• 我们使用重复n-gram覆盖率（Rae等，2021）来删除包含重复内容（如日志或错误消息）的行。这些行可能非常长且独特，因此无法通过行去重过滤。

• We use "dirty word" counting (Raffel et al., 2020) to filter out adult websites that are not covered by domain block lists.

• 我们使用“脏词”计数（Raffel等，2020）来过滤未被域名阻止列表覆盖的成人网站。

• We use a token-distribution Kullback-Leibler divergence to filter out documents containing excessive numbers of outlier tokens compared to the training corpus distribution.

• 我们使用令牌分布的Kullback-Leibler散度来过滤包含相对于训练语料库分布过多异常令牌的文档。

Model-based quality filtering. Further, we experiment with applying various model-based quality classifiers to sub-select high-quality tokens. These include using fast classifiers such as fasttext (Joulin et al., 2017) trained to recognize if a given text would be referenced by Wikipedia (Touvron et al., 2023a), as well as more compute-intensive Roberta-based classifiers (Liu et al., 2019a) trained on Llama 2 predictions. To train a quality classifier based on Llama 2, we create a training set of cleaned web documents, describe the quality requirements, and instruct Llama 2's chat model to determine if the documents meets these requirements. We use DistilRoberta (Sanh et al., 2019) to generate quality scores for each document for efficiency reasons. We experimentally evaluate the efficacy of various quality filtering configurations.

基于模型的质量过滤。此外，我们尝试应用各种基于模型的质量分类器来筛选高质量的令牌。这些包括使用快速分类器，如 fasttext（Joulin 等人，2017），训练来识别给定文本是否会被维基百科引用（Touvron 等人，2023a），以及更计算密集型的基于 Roberta 的分类器（Liu 等人，2019a），这些分类器在 Llama 2 预测上进行训练。为了基于 Llama 2 训练质量分类器，我们创建了一个经过清理的网络文档训练集，描述了质量要求，并指示 Llama 2 的聊天模型来确定文档是否满足这些要求。出于效率原因，我们使用 DistilRoberta（Sanh 等人，2019）为每个文档生成质量分数。我们实验性地评估了各种质量过滤配置的有效性。

Code and reasoning data. Similar to DeepSeek-AI et al. (2024), we build domain-specific pipelines that extract code and math-relevant web pages. Specifically, both the code and reasoning classifiers are DistilRoberta models trained on web data annotated by Llama 2. Unlike the general quality classifier mentioned above, we conduct prompt tuning to target web pages containing math deduction, reasoning in STEM areas and code interleaved with natural language. Since the token distribution of code and math is substantially different than that of natural language, these pipelines implement domain-specific HTML extraction, customized text features and heuristics for filtering.

代码和推理数据。与 DeepSeek-AI 等人（2024）类似，我们构建了特定领域的管道，用于提取与代码和数学相关的网页。具体来说，代码和推理分类器都是基于 Llama 2 标注的网络数据训练的 DistilRoberta 模型。与上述通用质量分类器不同，我们进行提示调优，以针对包含数学推导、STEM 领域推理以及自然语言交织代码的网页。由于代码和数学的令牌分布与自然语言有显著不同，这些管道实现了特定领域的 HTML 提取、定制文本特征和过滤启发式方法。

Multilingual data. Similar to our processing pipelines for English described above, we implement filters to remove data from websites that are likely to contain PII or unsafe content. Our multilingual text processing pipeline has several unique features:

多语言数据。与上述针对英语的处理管道类似，我们实施过滤器以移除可能包含个人身份信息（PII）或不安全内容的网站数据。我们的多语言文本处理管道具有几个独特特点：

• We use a fasttext-based language identification model to categorize documents into 176 languages.

• 我们使用基于 fasttext 的语言识别模型将文档分类为 176 种语言。

• We perform document-level and line-level de-duplication within data for each language. - We apply language-specific heuristics and model-based filters to remove low-quality documents.

• 我们对每种语言的数据进行文档级和行级去重。 - 我们应用特定语言的启发式方法和基于模型的过滤器来移除低质量文档。

In addition, we perform quality ranking of multilingual documents using a multilingual Llama 2-based classifier to ensure that high-quality content is prioritized. We determine the amount of multilingual tokens used in pre-training experimentally, balancing model performance on English and multilingual benchmarks.

此外，我们使用基于多语言Llama 2的分类器对多语言文档进行质量排序，以确保高质量内容得到优先处理。我们通过实验确定预训练中使用的多语言令牌数量，平衡模型在英语和多语言基准上的性能。

3.1.2 Determining the Data Mix 确定数据混合比例

To obtain a high-quality language model, it is essential to carefully determine the proportion of different data sources in the pre-training data mix. Our main tools in determining this data mix are knowledge classification and scaling law experiments.

为了获得高质量的语言模型，必须仔细确定预训练数据混合中不同数据源的比例。我们在确定这一数据混合比例时的主要工具是知识分类和缩放定律实验。

Knowledge classification. We develop a classifier to categorize the types of information contained in our web data to more effectively determine a data mix. We use this classifier to downsample data categories that are over-represented on the web, for example, arts and entertainment.

知识分类。我们开发了一个分类器，用于对我们的网络数据中包含的信息类型进行分类，以便更有效地确定数据混合比例。我们使用这个分类器对在网络上过度代表的数据类别进行降采样，例如艺术和娱乐。

Scaling laws for data mix. To determine the best data mix, we perform scaling law experiments in which we train several small models on a data mix and use that to predict the performance of a large model on that mix (see Section 3.2.1). We repeat this process multiple times for different data mixes to select a new data mix candidate. Subsequently, we train a larger model on this candidate data mix and evaluate the performance of that model on several key benchmarks.

数据混合比例的缩放定律。为了确定最佳的数据混合比例，我们进行了缩放定律实验，在这些实验中，我们在一个数据混合上训练几个小型模型，并利用这些模型预测大型模型在该混合上的性能（见第3.2.1节）。我们对不同的数据混合重复这一过程，以选择一个新的数据混合候选。随后，我们在这个候选数据混合上训练一个更大的模型，并在几个关键基准上评估该模型的性能。

Data mix summary. Our final data mix contains roughly \({50}\%\) of tokens corresponding to general knowledge, \({25}\%\) of mathematical and reasoning tokens, \({17}\%\) code tokens,and \(8\%\) multilingual tokens.

数据混合总结。我们的最终数据混合包含大约\({50}\%\)的令牌对应于一般知识，\({25}\%\)的数学和推理令牌，\({17}\%\)的代码令牌，以及\(8\%\)的多语言令牌。

3.1.3 Annealing Data 退火数据

Empirically, we find that annealing (see Section 3.4.3) on small amounts of high-quality code and mathematical data can boost the performance of pre-trained models on key benchmarks. Akin to Li et al. (2024b), we perform annealing with a data mix that upsamples high-quality data in select domains. We do not include any training sets from commonly used benchmarks in our annealing data. This enables us to assess the true few-shot learning capabilities and out-of-domain generalization of Llama 3.

根据实证研究，我们发现对少量高质量的代码和数学数据进行退火处理（参见第3.4.3节）可以提升预训练模型在关键基准测试中的性能。类似于Li等人（2024b）的方法，我们通过混合数据对高质量数据在选定领域进行上采样来执行退火处理。在我们的退火数据中不包含任何常用基准测试的训练集。这使我们能够评估Llama 3的真实少样本学习能力和跨领域泛化能力。

Following OpenAI (2023a), we evaluate the efficacy of annealing on the GSM8k (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) training sets in annealing. We find that annealing improved the performance of a pre-trained Llama 3 8B model on the GSM8k and MATH validation sets by 24.0% and 6.4%, respectively. However, the improvements on the 405B model are negligible, suggesting that our flagship model has strong in-context learning and reasoning capabilities and does not require specific in-domain training samples to obtain strong performance.

遵循OpenAI（2023a）的方法，我们在退火过程中评估了退火在GSM8k（Cobbe等人，2021）和MATH（Hendrycks等人，2021b）训练集上的有效性。我们发现，退火分别提高了预训练的Llama 3 8B模型在GSM8k和MATH验证集上的性能24.0%和6.4%。然而，405B模型的改进微乎其微，这表明我们的旗舰模型具有强大的上下文学习和推理能力，不需要特定的领域内训练样本就能获得强大的性能。

Using annealing to assess data quality. Similar to Blakeney et al. (2024), we find that annealing enables us to judge the value of small domain-specific datasets. We measure the value of such datasets by annealing the learning rate of a \({50}\%\) trained Llama 3 8B model linearly to 0 on 40B tokens. In those experiments,we assign \({30}\%\) weight to the new dataset and the remaining \({70}\%\) weight to the default data mix. Using annealing to evaluate new data sources is more efficient than performing scaling law experiments for every small dataset.

使用退火评估数据质量。类似于Blakeney等人（2024）的方法，我们发现退火使我们能够判断小规模特定领域数据集的价值。我们通过将\({50}\%\)训练的Llama 3 8B模型的学习率在40B个令牌上线性退火至0来衡量这些数据集的价值。在这些实验中，我们将\({30}\%\)权重分配给新数据集，其余的\({70}\%\)权重分配给默认数据混合。使用退火来评估新数据源比为每个小数据集执行缩放定律实验更高效。

3.2 Model Architecture 模型架构

Llama 3 uses a standard, dense Transformer architecture (Vaswani et al., 2017). It does not deviate significantly from Llama and Llama 2 (Touvron et al., 2023a,b) in terms of model architecture; our performance gains are primarily driven by improvements in data quality and diversity as well as by increased training scale.

Llama 3 采用标准的密集 Transformer 架构（Vaswani 等人，2017）。在模型架构方面，它与 Llama 和 Llama 2（Touvron 等人，2023a,b）没有显著偏离；我们的性能提升主要源于数据质量和多样性的改进以及训练规模的扩大。

We make a few small modifications compared to Llama 2:

与 Llama 2 相比，我们进行了一些小的修改：

• We use grouped query attention (GQA; Ainslie et al. (2023)) with 8 key-value heads to improve inference speed and to reduce the size of key-value caches during decoding.

• 我们使用分组查询注意力（GQA；Ainslie 等人（2023）），采用 8 个键值头，以提高推理速度并减少解码过程中键值缓存的大小。

• We use an attention mask that prevents self-attention between different documents within the same sequence. We find that this change had limited impact during in standard pre-training, but find it to be important in continued pre-training on very long sequences.

• 我们使用了一个注意力掩码，防止同一序列内不同文档之间的自注意力。我们发现这一改变在标准预训练中影响有限，但在对非常长的序列进行连续预训练时显得非常重要。

Table 3 Overview of the key hyperparameters of Llama 3. We display settings for 8B, 70B, and 405B language models.

表 3 Llama 3 关键超参数概览。我们展示了 8B、70B 和 405B 语言模型的设置。

• We use a vocabulary with \({128}\mathrm{\;K}\) tokens. Our token vocabulary combines \({100}\mathrm{\;K}\) tokens from the tiktoken \({}^{3}\) tokenizer with \({28}\mathrm{\;K}\) additional tokens to better support non-English languages. Compared to the Llama 2 tokenizer, our new tokenizer improves compression rates on a sample of English data from 3.17 to 3.94 characters per token. This enables the model to "read" more text for the same amount of training compute. We also found that adding \({28}\mathrm{\;K}\) tokens from select non-English languages improved both compression ratios and downstream performance, with no impact on English tokenization.

• 我们使用了一个包含 \({128}\mathrm{\;K}\) 个词元的词汇表。我们的词元词汇结合了 tiktoken \({}^{3}\) 分词器中的 \({100}\mathrm{\;K}\) 个词元和 \({28}\mathrm{\;K}\) 个额外词元，以更好地支持非英语语言。与 Llama 2 分词器相比，我们的新分词器在英语数据样本上的压缩率从 3.17 提高到 3.94 个字符每词元。这使得模型能够以相同的训练计算量“读取”更多文本。我们还发现，添加来自选定非英语语言的 \({28}\mathrm{\;K}\) 个词元不仅提高了压缩比，还提升了下游性能，且对英语分词没有影响。

• We increase the RoPE base frequency hyperparameter to 500,000 . This enables us to better support longer contexts; Xiong et al. (2023) showed this value to be effective for context lengths up to 32,768.

• 我们将 RoPE 基频超参数提高到 500,000。这使我们能够更好地支持更长的上下文；Xiong 等人（2023）表明，这一数值对于长达 32,768 的上下文长度是有效的。

Llama 3405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128 attention heads; see Table 3 for details. This leads to a model size that is approximately compute-optimal according to scaling laws on our data for our training budget of \({3.8} \times {10}^{25}\mathrm{{FLOPs}}\) .

Llama 3405B 采用了一种架构，包含 126 层，令牌表示维度为 16,384，以及 128 个注意力头；详细信息参见表 3。根据我们的数据和训练预算 \({3.8} \times {10}^{25}\mathrm{{FLOPs}}\) 的缩放定律，这导致了一个近似计算最优的模型大小。

3.2.1 Scaling Laws 缩放定律

We develop scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) to determine the optimal model size for our flagship model given our pre-training compute budget. In addition to determining the optimal model size, a major challenge is to forecast the flagship model's performance on downstream benchmark tasks, due to a couple of issues: (1) Existing scaling laws typically predict only next-token prediction loss rather than specific benchmark performance. (2) Scaling laws can be noisy and unreliable because they are developed based on pre-training runs conducted with small compute budgets (Wei et al., 2022b).

我们根据预训练计算预算，开发了缩放定律（Hoffmann 等人，2022；Kaplan 等人，2020），以确定我们旗舰模型的最优大小。除了确定最优模型大小外，一个主要挑战是预测旗舰模型在下游基准任务上的性能，这由于以下几个问题：（1）现有的缩放定律通常仅预测下一个令牌预测损失，而不是特定的基准性能。（2）缩放定律可能嘈杂且不可靠，因为它们是基于使用小计算预算进行的预训练运行开发的（Wei 等人，2022b）。

To address these challenges, we implement a two-stage methodology to develop scaling laws that accurately predict downstream benchmark performance:

为了解决这些挑战，我们实施了一种两阶段方法来开发能够准确预测下游基准性能的缩放定律：

1. We first establish a correlation between the compute-optimal model's negative log-likelihood on downstream tasks and the training FLOPs.

2. 我们首先建立计算最优模型在下游任务上的负对数似然与训练浮点运算次数（FLOPs）之间的相关性。

3. Next, we correlate the negative log-likelihood on downstream tasks with task accuracy, utilizing both the scaling law models and older models trained with higher compute FLOPs. In this step, we specifically leverage the Llama 2 family of models.

4. 接下来，我们利用缩放定律模型和使用更高计算 FLOPs 训练的旧模型，将下游任务上的负对数似然与任务准确性相关联。在这一步骤中，我们特别利用了 Llama 2 系列模型。

This approach enables us to predict downstream task performance given a specific number of training FLOPs for compute-optimal models. We use a similar method to select our pre-training data mix (see Section 3.4).

这种方法使我们能够根据计算最优模型的特定训练 FLOPs 数量预测下游任务性能。我们使用类似的方法来选择我们的预训练数据混合（参见第 3.4 节）。

Scaling law experiments. Concretely, we construct our scaling laws by pre-training models using compute budgets between \(6 \times {10}^{18}\) FLOPs and \({10}^{22}\) FLOPs. At each compute budget,we pre-train models ranging in size between \({40}\mathrm{M}\) and \({16}\mathrm{\;B}\) parameters,using a subset of model sizes at each compute budget. In these training runs, we use a cosine learning rate schedule with a linear warmup for 2,000 training steps. The peak learning rate is set between \(2 \times {10}^{-4}\) and \(4 \times {10}^{-4}\) depending on the size of the model. We set the cosine decay to 0.1 of the peak value. The weight decay at each step is set to 0.1 times the learning rate at that step. We use a fixed batch size for each compute scale,ranging between \({250}\mathrm{\;K}\) and \(4\mathrm{M}\) .

规模法则实验。具体而言，我们通过使用介于 \(6 \times {10}^{18}\) FLOPs 和 \({10}^{22}\) FLOPs 之间的计算预算对模型进行预训练来构建我们的规模法则。在每个计算预算下，我们预训练大小介于 \({40}\mathrm{M}\) 和 \({16}\mathrm{\;B}\) 参数之间的模型，在每个计算预算下使用模型大小的一个子集。在这些训练运行中，我们使用余弦学习率调度，线性预热 2,000 个训练步骤。峰值学习率根据模型大小设置在 \(2 \times {10}^{-4}\) 和 \(4 \times {10}^{-4}\) 之间。我们将余弦衰减设置为峰值值的 0.1。每一步的权重衰减设置为该步学习率的 0.1 倍。我们为每个计算规模使用固定的批量大小，范围介于 \({250}\mathrm{\;K}\) 和 \(4\mathrm{M}\) 之间。

\({}^{3}\) https://github.com/openai/tiktoken/tree/main

Figure 2 Scaling law IsoFLOPs curves between \(6 \times {10}^{18}\) and \({10}^{22}\) FLOPs. The loss is the negative log-likelihood on a held-out validation set. We approximate measurements at each compute scale using a second degree polynomial.

图2 在 \(6 \times {10}^{18}\) 和 \({10}^{22}\) FLOPs 之间的规模法则等FLOPs曲线。损失是在一个独立的验证集上的负对数似然。我们使用二次多项式近似每个计算规模上的测量值。

Figure 3 Number of training tokens in identified compute-optimal models as a function of pre-training compute budget. We include the fitted scaling-law prediction as well. The compute-optimal models correspond to the parabola minimums in Figure 2.

图3 作为预训练计算预算函数的识别出的计算最优模型中的训练令牌数量。我们还包含了拟合的规模法则预测。计算最优模型对应于图2中的抛物线最小值。

These experiments give rise to the IsoFLOPs curves in Figure 2. The loss in these curves is measured on a separate validation set. We fit the measured loss values using a second-degree polynomial and identify the minimums of each parabola. We refer to minimum of a parabola as the compute-optimal model at the corresponding pre-training compute budget.

这些实验产生了图2中的等FLOPs曲线。这些曲线中的损失是在一个独立的验证集上测量的。我们使用二次多项式拟合测量的损失值，并识别每个抛物线的最小值。我们将抛物线的最小值称为相应预训练计算预算下的计算最优模型。

We use the compute-optimal models we identified this way to predict the optimal number of training tokens for a specific compute budget. To do so,we assume a power-law relation between compute budget, \(C\) ,and the optimal number of training tokens, \({N}^{ \star }\left( C\right)\) :

我们使用这种方式识别出的计算最优模型来预测特定计算预算下的最优训练令牌数量。为此，我们假设计算预算 \(C\) 和最优训练令牌数量 \({N}^{ \star }\left( C\right)\) 之间存在幂律关系：

\[{N}^{ \star }\left( C\right) = A{C}^{\alpha }.\]

We fit \(A\) and \(\alpha\) using the data from Figure 2. We find that \(\left( {\alpha ,A}\right) = \left( {{0.53},{0.29}}\right)\) ; the corresponding fit is shown in Figure 3. Extrapolation of the resulting scaling law to \({3.8} \times {10}^{25}\) FLOPs suggests training a 402B parameter model on \({16.55}\mathrm{\;T}\) tokens.

我们使用图2的数据来拟合 \(A\) 和 \(\alpha\)。我们发现 \(\left( {\alpha ,A}\right) = \left( {{0.53},{0.29}}\right)\)；相应的拟合结果如图3所示。将由此得出的缩放定律外推至 \({3.8} \times {10}^{25}\) FLOPs 表明，应在 \({16.55}\mathrm{\;T}\) 令牌上训练一个402B参数的模型。

An important observation is that IsoFLOPs curves become flatter around the minimum as the compute budget increases. This implies that performance of the flagship model is relatively robust to small changes in the trade-off between model size and training tokens. Based on this observation, we ultimately decided to train a flagship model with \({405}\mathrm{\;B}\) parameters.

一个重要的观察结果是，随着计算预算的增加，IsoFLOPs曲线在最小值附近变得更平坦。这意味着旗舰模型的性能对模型大小和训练令牌之间权衡的小变化相对稳健。基于这一观察，我们最终决定训练一个具有 \({405}\mathrm{\;B}\) 参数的旗舰模型。

Predicting performance on downstream tasks. We use the resulting compute-optimal models to forecast the performance of the flagship Llama 3 model on benchmark data sets. First, we linearly correlate the (normalized) negative log-likelihood of correct answer in the benchmark and the training FLOPs. In this analysis,we use only the scaling law models trained up to \({10}^{22}\) FLOPs on the data mix described above. Next, we establish a sigmoidal relation between the log-likelihood and accuracy using both the scaling law models and Llama 2 models, which were trained using the Llama 2 data mix and tokenizer. We show the results of this experiment on the ARC Challenge benchmark in Figure 4). We find this two-step scaling law prediction, which extrapolates over four orders of magnitude, to be quite accurate: it only slightly underestimates the final performance of the flagship Llama 3 model.

预测下游任务的性能。我们使用由此得出的计算最优模型来预测旗舰Llama 3模型在基准数据集上的性能。首先，我们在线性相关基准中正确答案的（归一化）负对数似然和训练FLOPs。在此分析中，我们仅使用上述数据混合训练至 \({10}^{22}\) FLOPs 的缩放定律模型。接下来，我们使用缩放定律模型和Llama 2模型（这些模型使用Llama 2数据混合和分词器进行训练）建立对数似然和准确性之间的S形关系。我们在ARC挑战基准上展示了这一实验的结果（如图4所示）。我们发现这种跨越四个数量级的两步缩放定律预测相当准确：它仅略微低估了旗舰Llama 3模型的最终性能。

3.3 Infrastructure, Scaling, and Efficiency 基础设施、扩展和效率

We describe our hardware and infrastructure that powered Llama 3405B pre-training at scale and discuss several optimizations that leads to improvements in training efficiency.

我们描述了支持 Llama 3405B 大规模预训练的硬件和基础设施，并讨论了几项优化措施，这些措施提高了训练效率。

3.3.1 Training Infrastructure 训练基础设施

The Llama 1 and 2 models were trained on Meta's AI Research SuperCluster (Lee and Sengupta, 2022). As we scaled further, the training for Llama 3 was migrated to Meta's production clusters (Lee et al., 2024).This setup optimizes for production-grade reliability, which is essential as we scale up training.

Llama 1 和 2 模型是在 Meta 的 AI 研究超级集群（Lee 和 Sengupta，2022）上训练的。随着我们进一步扩展，Llama 3 的训练迁移到了 Meta 的生产集群（Lee 等人，2024）。这种设置针对生产级可靠性进行了优化，这对于我们扩大训练规模至关重要。

Figure 4 Scaling law forecast for ARC Challenge. Left: Normalized negative log-likelihood of the correct answer on the ARC Challenge benchmark as a function of pre-training FLOPs. Right: ARC Challenge benchmark accuracy as a function of the normalized negative log-likelihood of the correct answer. This analysis enables us to predict model performance on the ARC Challenge benchmark before pre-training commences. See text for details.

图 4 ARC 挑战赛的规模法则预测。左侧：ARC 挑战赛基准测试中正确答案的归一化负对数似然性作为预训练 FLOP 的函数。右侧：ARC 挑战赛基准测试准确性作为正确答案的归一化负对数似然性的函数。这一分析使我们能够在预训练开始之前预测模型在 ARC 挑战赛基准测试上的性能。详见正文。

Compute. Llama 3 405B is trained on up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3, using Meta's Grand Teton AI server platform (Matt Bowman, 2022). Each server is equipped with eight GPUs and two CPUs. Within a server, the eight GPUs are connected via NVLink. Training jobs are scheduled using MAST (Choudhury et al., 2024), Meta's global-scale training scheduler.

计算。Llama 3 405B 在多达 16K H100 GPU 上进行训练，每个 GPU 以 700W TDP 运行，配备 80GB HBM3，使用 Meta 的 Grand Teton AI 服务器平台（Matt Bowman，2022）。每台服务器配备八块 GPU 和两块 CPU。在服务器内部，八块 GPU 通过 NVLink 连接。训练任务使用 MAST（Choudhury 等人，2024）进行调度，这是 Meta 的全球规模训练调度器。

Storage. Tectonic (Pan et al., 2021), Meta's general-purpose distributed file system, is used to build a storage fabric (Battey and Gupta, 2024) for Llama 3 pre-training. It offers 240 PB of storage out of 7,500 servers equipped with SSDs, and supports a sustainable throughput of 2 TB/s and a peak throughput of 7 TB/s. A major challenge is supporting the highly bursty checkpoint writes that saturate the storage fabric for short durations. Checkpointing saves each GPU's model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. We aim to minimize GPU pause time during checkpointing and increase checkpoint frequency to reduce the amount of lost work after a recovery.

存储。Tectonic（Pan et al., 2021），Meta的通用分布式文件系统，被用于构建Llama 3预训练的存储架构（Battey and Gupta, 2024）。它提供了来自7,500台配备SSD的服务器的240 PB存储容量，并支持2 TB/s的可持续吞吐量和7 TB/s的峰值吞吐量。一个主要挑战是支持高度突发的检查点写入，这些写入会在短时间内饱和存储架构。检查点保存每个GPU的模型状态，每个GPU的范围从1 MB到4 GB不等，用于恢复和调试。我们的目标是尽量减少检查点期间的GPU暂停时间，并增加检查点频率以减少恢复后的工作损失量。

Network. Llama 3405B used RDMA over Converged Ethernet (RoCE) fabric based on the Arista 7800 and Minipack2 Open Compute Project \({}^{4}\) OCP rack switches. Smaller models in the Llama 3 family were trained using Nvidia Quantum2 Infiniband fabric. Both RoCE and Infiniband clusters leverage 400 Gbps interconnects between GPUs. Despite the underlying network technology differences between these clusters, we tune both of them to provide equivalent performance for these large training workloads. We elaborate further on our RoCE network since we fully own its design.

网络。Llama 3405B使用了基于Arista 7800和Minipack2 Open Compute Project \({}^{4}\) OCP机架交换机的RDMA over Converged Ethernet（RoCE）架构。Llama 3系列中的较小模型则使用Nvidia Quantum2 Infiniband架构进行训练。无论是RoCE还是Infiniband集群，都利用了GPU之间的400 Gbps互连。尽管这些集群的底层网络技术存在差异，我们对其进行了调整，以在这些大型训练负载中提供等效性能。我们进一步详细阐述我们的RoCE网络，因为我们完全拥有其设计。

• Network topology. Our RoCE-based AI cluster comprises \({24}\mathrm{\;K}{\mathrm{{GPUs}}}^{5}\) connected by a three-layer Clos network (Lee et al., 2024). At the bottom layer, each rack hosts 16 GPUs split between two servers and connected by a single Minipack2 top-of-the-rack (ToR) switch. In the middle layer, 192 such racks are connected by Cluster Switches to form a pod of 3,072 GPUs with full bisection bandwidth, ensuring no oversubscription. At the top layer, eight such pods within the same datacenter building are connected via Aggregation Switches to form a cluster of \({24}\mathrm{\;K}\) GPUs. However,network connectivity at the aggregation layer does not maintain full bisection bandwidth and instead has an oversubscription ratio of 1:7. Our model parallelism methods (see Section 3.3.2) and training job scheduler (Choudhury et al., 2024) are all optimized to be aware of network topology, aiming to minimize network communication across pods.

• 网络拓扑。我们的基于 RoCE 的 AI 集群由 \({24}\mathrm{\;K}{\mathrm{{GPUs}}}^{5}\) 通过三层 Clos 网络连接而成（Lee et al., 2024）。在最底层，每个机架托管 16 个 GPU，分布在两台服务器上，并通过一个 Minipack2 架顶（ToR）交换机连接。在中层，192 个这样的机架通过集群交换机连接，形成一个包含 3,072 个 GPU 的 Pod，具有完全的双向带宽，确保没有超额订阅。在顶层，同一数据中心建筑内的八个这样的 Pod 通过聚合交换机连接，形成一个包含 \({24}\mathrm{\;K}\) 个 GPU 的集群。然而，聚合层的网络连接并不保持完全的双向带宽，而是具有 1:7 的超额订阅比。我们的模型并行方法（见第 3.3.2 节）和训练作业调度器（Choudhury et al., 2024）都经过优化，以了解网络拓扑，旨在最小化 Pod 间的网络通信。

• Load balancing. LLM training produces fat network flows that are hard to load balance across all available network paths using traditional methods such as Equal-Cost Multi-Path (ECMP) routing. To address this challenge, we employ two techniques. First, our collective library creates 16 network flows between two GPUs, instead of just one, thereby reducing the traffic per flow and providing more flows

• 负载均衡。LLM 训练产生的大量网络流量很难使用传统的等成本多路径（ECMP）路由等方法在所有可用网络路径上进行负载均衡。为了解决这一挑战，我们采用了两种技术。首先，我们的集合库在两个 GPU 之间创建 16 个网络流，而不是仅一个，从而减少了每个流的流量并提供了更多的流

\({}^{4}\) Open Compute Project: https://www.opencompute.org/

\({}^{4}\) 开放计算项目：https://www.opencompute.org/

\({}^{5}\) Note that we use only up to \({16}\mathrm{\;K}\) of these \({24}\mathrm{\;K}\) GPUs for Llama 3 pre-training.

\({}^{5}\) 请注意，我们仅使用这些 \({24}\mathrm{\;K}\) 个 GPU 中的最多 \({16}\mathrm{\;K}\) 个用于 Llama 3 预训练。

Table 4 – Scaling configurations and MFU for each stage of Llama 3 405B pre-training. See \(\mathrm{{text}}\) and Figure 5 for \(\mathrm{{descriptions}}\) of each type of parallelism.

表 4 – Llama 3 405B 预训练各阶段的扩展配置和 MFU。参见 \(\mathrm{{text}}\) 和图 5 了解每种并行性的 \(\mathrm{{descriptions}}\)。

for load balancing. Second, our Enhanced-ECMP (E-ECMP) protocol effectively balances these 16 flows across different network paths by hashing on additional fields in the RoCE header of packets.

用于负载均衡。其次，我们的增强型等价多路径（E-ECMP）协议通过在RoCE数据包头中对额外字段进行哈希处理，有效地将这16个流平衡到不同的网络路径上。

• Congestion control. We use deep-buffer switches in the spine (Gangidi et al., 2024) to accommodate transient congestion and buffering caused by collective communication patterns. This setup helps limit the impact of persistent congestion and network back pressure caused by slow servers, which is common in training. Finally, better load balancing through E-ECMP significantly reduces the chance of congestion. With these optimizations, we successfully run a 24K GPU cluster without traditional congestion control methods such as Data Center Quantized Congestion Notification (DCQCN).

• 拥塞控制。我们在骨干网中使用深度缓冲交换机（Gangidi等人，2024年）来容纳由集体通信模式引起的瞬态拥塞和缓冲。这种设置有助于限制持续拥塞和由慢速服务器引起的网络背压的影响，这在训练中很常见。最后，通过E-ECMP实现的更好的负载均衡显著降低了拥塞的可能性。通过这些优化，我们成功地运行了一个24K GPU集群，而没有使用诸如数据中心量化拥塞通知（DCQCN）等传统的拥塞控制方法。

3.3.2 Parallelism for Model Scaling 模型扩展的并行化

To scale training for our largest models, we use 4D parallelism - a combination of four different types of parallelism methods - to shard the model. This approach efficiently distributes computation across many GPUs and ensures each GPU's model parameters, optimizer states, gradients, and activations fit in its HBM. Our implementation of 4D parallelism is illustrated in Figure 5. It combines tensor parallelism (TP; Krizhevsky et al. (2012); Shoeybi et al. (2019); Korthikanti et al. (2023)), pipeline parallelism (PP; Huang et al. (2019); Narayanan et al. (2021); Lamy-Poirier (2023)), context parallelism (CP; Liu et al. (2023a)), and data parallelism (DP; Rajbhandari et al. (2020); Ren et al. (2021); Zhao et al. (2023b)).

为了扩展我们最大模型的训练，我们使用了4D并行化——结合了四种不同类型的并行化方法——来分片模型。这种方法有效地将计算分布到许多GPU上，并确保每个GPU的模型参数、优化器状态、梯度和激活值都适合其HBM。我们的4D并行化实现如图5所示。它结合了张量并行化（TP；Krizhevsky等人（2012年）；Shoeybi等人（2019年）；Korthikanti等人（2023年））、流水线并行化（PP；Huang等人（2019年）；Narayanan等人（2021年）；Lamy-Poirier（2023年））、上下文并行化（CP；Liu等人（2023a年））和数据并行化（DP；Rajbhandari等人（2020年）；Ren等人（2021年）；Zhao等人（2023b年））。

Tensor parallelism splits individual weight tensors into multiple chunks on different devices. Pipeline parallelism partitions the model vertically into stages by layers, so that different devices can process in parallel different stages of the full model pipeline. Context parallelism divides the input context into segments, reducing memory bottleneck for very long sequence length inputs. We use fully sharded data parallelism (FSDP; Rajbhandari et al., 2020; Ren et al., 2021; Zhao et al., 2023b), which shards the model, optimizer, and gradients while implementing data parallelism which processes data in parallel on multiple GPUs and synchronizes after each training step. Our use of FSDP for Llama 3 shards optimizer states and gradients, but for model shards we do not reshard after forward computation to avoid an extra all-gather communication during backward passes.

张量并行将单个权重张量分割到不同设备上的多个块中。流水线并行将模型垂直划分为多个阶段，通过层来实现，使得不同设备可以并行处理完整模型流水线的不同阶段。上下文并行将输入上下文分割成段，减少了对于非常长的序列长度输入的内存瓶颈。我们使用完全分片数据并行（FSDP；Rajbhandari 等人，2020；Ren 等人，2021；Zhao 等人，2023b），它在实现数据并行的同时，对模型、优化器和梯度进行分片，即在多个GPU上并行处理数据，并在每个训练步骤后进行同步。我们在Llama 3中使用FSDP分片优化器状态和梯度，但对于模型分片，我们在前向计算后不重新分片，以避免在后向传递期间进行额外的全收集通信。

GPU utilization. Through careful tuning of the parallelism configuration, hardware, and software, we achieve an overall BF16 Model FLOPs Utilization (MFU; Chowdhery et al. (2023)) of 38-43% for the configurations shown in Table 4. The slight drop in MFU to \({41}\%\) on \({16}\mathrm{\;K}\) GPUs with DP=128 compared to \({43}\%\) on \(8\mathrm{\;K}\) GPUs with DP=64 is due to the lower batch size per DP group needed to keep the global tokens per batch constant during training.

GPU利用率。通过仔细调整并行配置、硬件和软件，我们为表4所示的配置实现了总体BF16模型浮点运算利用率（MFU；Chowdhery 等人（2023））达到38-43%。在DP=128的\({16}\mathrm{\;K}\)GPU上，MFU略微下降至\({41}\%\)，相比于DP=64的\(8\mathrm{\;K}\)GPU上的\({43}\%\)，这是由于在训练过程中为了保持全局批次令牌数量恒定，需要降低每个DP组的批次大小。

Pipeline parallelism improvements. We encountered several challenges with existing implementations:

流水线并行改进。我们在现有实现中遇到了几个挑战：

• Batch size constraint. Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages. For the example in Figure 6, the depth-first schedule (DFS) of pipeline parallelism (Narayanan et al.,2021) requires \(N = \mathrm{{PP}} = 4\) ,while the breadth-first schedule (BFS; Lamy-Poirier (2023)) requires \(N = M\) ,where \(M\) is the total number of micro-batches and \(N\) is the number of contiguous micro-batches for the same stage’s forward or backward. However, pre-training often needs flexibility to adjust batch size.

• 批量大小约束。当前的实现对每个 GPU 支持的批量大小有约束，要求其必须能被流水线阶段数整除。以图 6 中的例子为例，深度优先调度（DFS）的流水线并行（Narayanan 等人，2021）要求 \(N = \mathrm{{PP}} = 4\)，而广度优先调度（BFS；Lamy-Poirier（2023））要求 \(N = M\)，其中 \(M\) 是微批量的总数，\(N\) 是同一阶段前向或后向的连续微批量数。然而，预训练通常需要灵活调整批量大小。

• Memory imbalance. Existing pipeline parallelism implementations lead to imbalanced resource consumption. The first stage consumes more memory due to the embedding and the warm-up micro-batches.

• 内存不平衡。现有的流水线并行实现导致资源消耗不平衡。第一阶段由于嵌入和预热微批量而消耗更多内存。

• Computation imbalance. After the last layer of the model, we need to calculate output and loss, making this stage the execution latency bottleneck.

• 计算不平衡。在模型的最后一层之后，我们需要计算输出和损失，这使得该阶段成为执行延迟瓶颈。

Figure 5 Illustration of 4D parallelism. GPUs are divided into parallelism groups in the order of [TP, CP, PP, DP], where DP stands for FSDP. In this example, \({16}\) GPUs are configured with a group size of \(\left| \mathrm{{TP}}\right| = 2,\left| \mathrm{{CP}}\right| = 2,\left| \mathrm{{PP}}\right| = 2\) ,and \(\left| \mathrm{{DP}}\right| = 2\) . A GPU’s position in 4D parallelism is represented as a vector, \(\left\lbrack {{D}_{1},{D}_{2},{D}_{3},{D}_{4}}\right\rbrack\) ,where \({D}_{i}\) is the index on the \(i\) -th parallelism dimension. In this example,GPU0[TP0,CP0,PP0,DP0] and GPU1[TP1,CP0,PP0,DP0] are in the same TP group, GPU0 and GPU2 are in the same CP group, GPU0 and GPU4 are in the same PP group, and GPU0 and GPU8 are in the same DP group.

图 5 展示了 4D 并行的示意图。GPU 按照 [TP, CP, PP, DP] 的顺序分成并行组，其中 DP 代表 FSDP。在这个例子中，\({16}\) 个 GPU 配置了 \(\left| \mathrm{{TP}}\right| = 2,\left| \mathrm{{CP}}\right| = 2,\left| \mathrm{{PP}}\right| = 2\) 的组大小，以及 \(\left| \mathrm{{DP}}\right| = 2\)。GPU 在 4D 并行中的位置表示为一个向量 \(\left\lbrack {{D}_{1},{D}_{2},{D}_{3},{D}_{4}}\right\rbrack\)，其中 \({D}_{i}\) 是第 \(i\) 个并行维度上的索引。在这个例子中，GPU0[TP0,CP0,PP0,DP0] 和 GPU1[TP1,CP0,PP0,DP0] 在同一个 TP 组中，GPU0 和 GPU2 在同一个 CP 组中，GPU0 和 GPU4 在同一个 PP 组中，GPU0 和 GPU8 在同一个 DP 组中。

To address these issues,we modify our pipeline schedule as shown in Figure 6,which allows setting \(N\) flexibly - in this case \(N = 5\) ,which can run a arbitrary number of micro-batches in each batch. This allows us to run: (1) fewer micro-batches than the number of stages when we have batch size limit at large scale; or (2) more micro-batches to hide point-to-point communication, finding a sweet spot between DFS and breadth first schedule (BFS) for the best communication and memory efficiency. To balance the pipeline, we reduce one Transformer layer each from the first and the last stages, respectively. This means that the first model chunk on the first stage has only the embedding, and the last model chunk on the last stage has only output projection and loss calculation. To reduce pipeline bubbles, we use an interleaved schedule (Narayanan et al.,2021) with \(V\) pipeline stages on one pipeline rank. Overall pipeline bubble ratio is \(\frac{\mathrm{{PP}} - 1}{V * M}\) . Further,we adopt asynchronous point-to-point communication in PP,which considerably speeds up training, especially in cases when the document mask introduces extra computation imbalance. We enable TORCH_NCCL_AVOID_RECORD_STREAMS to reduce memory usage from asynchronous point-to-point communication. Finally, to reduce memory cost, based on detailed memory allocation profiling, we proactively deallocate tensors that will not be used for future computation, including the input and output tensors of each pipeline stage, that will not be used for future computation. With these optimizations, we could pre-train Llama 3 on sequences of \(8\mathrm{\;K}\) tokens without activation checkpointing.

为了解决这些问题，我们修改了如图6所示的流水线调度，这使得可以灵活设置 \(N\) - 在这种情况下是 \(N = 5\)，它可以在每个批次中运行任意数量的微批次。这使我们能够运行：（1）在大规模下有批次大小限制时，少于阶段数量的微批次；或（2）更多的微批次以隐藏点对点通信，在DFS和广度优先调度（BFS）之间找到最佳通信和内存效率的平衡点。为了平衡流水线，我们分别从第一和最后阶段各减少一个Transformer层。这意味着第一阶段的第一模型块只有嵌入层，最后阶段的最后模型块只有输出投影和损失计算。为了减少流水线气泡，我们使用了一种交错调度（Narayanan等人，2021），在一个流水线等级上有 \(V\) 个流水线阶段。总体流水线气泡比率为 \(\frac{\mathrm{{PP}} - 1}{V * M}\)。此外，我们在PP中采用了异步点对点通信，这大大加快了训练速度，尤其是在文档掩码引入额外计算不平衡的情况下。我们启用了TORCH_NCCL_AVOID_RECORD_STREAMS来减少异步点对点通信的内存使用。最后，为了降低内存成本，基于详细的内存分配分析，我们主动释放那些不会用于未来计算的张量，包括每个流水线阶段的输入和输出张量，这些张量不会用于未来的计算。通过这些优化，我们可以在没有激活检查点的情况下预训练Llama 3，处理长度为 \(8\mathrm{\;K}\) 个令牌的序列。

Context parallelism for long sequences. We utilize context parallelism (CP) to improve memory efficiency when scaling the context length of Llama 3 and enable training on extremely long sequences up to \({128}\mathrm{\;K}\) in length. In CP, we partition across the sequence dimension, and specifically we partition the input sequence into \(2 \times \mathrm{{CP}}\) chunks so each CP rank receives two chunks for better load balancing. The \(i\) -th CP rank received both the \(i\) -th and the \(\left( {2 \times \mathrm{{CP}} - 1 - i}\right)\) -th chunks.

长序列的上下文并行。我们利用上下文并行（CP）在扩展Llama 3的上下文长度时提高内存效率，并能够在长度高达\({128}\mathrm{\;K}\)的极长序列上进行训练。在CP中，我们在序列维度上进行分区，具体地将输入序列分成\(2 \times \mathrm{{CP}}\)个块，以便每个CP等级接收两个块以实现更好的负载均衡。第\(i\)个CP等级接收第\(i\)个和第\(\left( {2 \times \mathrm{{CP}} - 1 - i}\right)\)个块。

Different from existing CP implementations that overlap communication and computation in a ring-like structure (Liu et al., 2023a), our CP implementation adopts an all-gather based method where we first all-gather the key (K) and value (V) tensors, and then compute attention output for the local query (Q) tensor chunk. Although the all-gather communication latency is exposed in the critical path, we still adopt this approach for two main reasons: (1) it is easier and more flexible to support different types of attention masks in all-gather based CP attention, such as the document mask; and (2) the exposed all-gather latency is small as the communicated \(\mathrm{K}\) and \(\mathrm{V}\) tensors are much smaller than \(\mathrm{Q}\) tensor due to the use of GQA (Ainslie et al., 2023). Hence, the time complexity of attention computation is an order of magnitude larger than all-gather \(\left( {O\left( {S}^{2}\right) }\right.\) versus \(O\left( S\right)\) ,where \(S\) represents the sequence length in the full causal mask),making the all-gather overhead negligible.

与现有在环状结构中重叠通信和计算的CP实现（Liu et al., 2023a）不同，我们的CP实现采用了一种基于all-gather的方法，其中我们首先all-gather键（K）和值（V）张量，然后计算本地查询（Q）张量块的注意力输出。尽管all-gather通信延迟暴露在关键路径中，我们仍然采用这种方法有两个主要原因：（1）在基于all-gather的CP注意力中，支持不同类型的注意力掩码（如文档掩码）更容易且更灵活；（2）由于使用了GQA（Ainslie et al., 2023），通信的\(\mathrm{K}\)和\(\mathrm{V}\)张量远小于\(\mathrm{Q}\)张量，因此暴露的all-gather延迟很小。因此，注意力计算的时间复杂度比all-gather大一个数量级（\(\left( {O\left( {S}^{2}\right) }\right.\)对\(O\left( S\right)\)，其中\(S\)表示完整因果掩码中的序列长度），使得all-gather的开销可以忽略不计。

Figure 6 Illustration of pipeline parallelism in Llama 3. Pipeline parallelism partitions eight pipeline stages (0 to 7) across four pipeline ranks (PP ranks 0 to 3), where the GPUs with rank 0 run stages 0 and 4, the GPUs with P rank 1 run stages 1 and 5,etc. The colored blocks ( 0 to 9 ) represent a sequence of micro-batches,where \(M\) is the total number of micro-batches and \(N\) is the number of continuous micro-batches for the same stage’s forward or backward. Our key insight is to make \(N\) tunable.

图6展示了Llama 3中的流水线并行性示意图。流水线并行性将八个流水线阶段（0到7）分布在四个流水线等级（PP等级0到3）上，其中等级0的GPU运行阶段0和4，等级1的GPU运行阶段1和5，依此类推。彩色块（0到9）代表一系列微批次，其中\(M\)是微批次总数，\(N\)是同一阶段的连续微批次数量，用于前向或后向传播。我们的关键见解是使\(N\)可调。

Network-aware parallelism configuration. The order of parallelism dimensions, [TP, CP, PP, DP], is optimized for network communication. The innermost parallelism requires the highest network bandwidth and lowest latency, and hence is usually constrained to within the same server. The outermost parallelism may spread across a multi-hop network and should tolerate higher network latency. Therefore, based on the requirements for network bandwidth and latency, we place parallelism dimensions in the order of [TP, CP, PP, DP]. DP (i.e., FSDP) is the outermost parallelism because it can tolerate longer network latency by asynchronously prefetching sharded model weights and reducing gradients. Identifying the optimal parallelism configuration with minimal communication overhead while avoiding GPU memory overflow is challenging. We develop a memory consumption estimator and a performance-projection tool which helped us explore various parallelism configurations and project overall training performance and identify memory gaps effectively.

网络感知的并行配置。并行维度的顺序[TP, CP, PP, DP]针对网络通信进行了优化。最内层的并行性需要最高的网络带宽和最低的延迟，因此通常限制在同一服务器内。最外层的并行性可能跨越多跳网络，并应容忍更高的网络延迟。因此，根据对网络带宽和延迟的要求，我们将并行维度按[TP, CP, PP, DP]的顺序排列。DP（即FSDP）是最外层的并行性，因为它可以通过异步预取分片模型权重和减少梯度来容忍更长的网络延迟。在避免GPU内存溢出的同时，确定具有最小通信开销的最佳并行配置是一个挑战。我们开发了一个内存消耗估计器和一个性能预测工具，帮助我们探索各种并行配置，有效预测整体训练性能并识别内存缺口。

Numerical stability. By comparing training loss between different parallelism setups, we fixed several numerical issues that impact training stability. To ensure training convergence, we use FP32 gradient accumulation during backward computation over multiple micro-batches and also reduce-scatter gradients in FP32 across data parallel workers in FSDP. For intermediate tensors, e.g., vision encoder outputs, that are used multiple times in the forward computation, the backward gradients are also accumulated in FP32.

数值稳定性。通过比较不同并行设置下的训练损失，我们解决了影响训练稳定性的几个数值问题。为了确保训练收敛，我们在多个微批次的后向计算中使用FP32梯度累积，并在FSDP中的数据并行工作者之间使用FP32梯度进行减少散布。对于在前向计算中多次使用的中间张量，例如视觉编码器输出，后向梯度也在FP32中累积。

3.3.3 Collective Communication 集体通信

Our collective communication library for Llama 3 is based on a fork of Nvidia's NCCL library, called NCCLX. NCCLX significantly improves the performance of NCCL, especially for higher latency networks. Recall that the order of parallelism dimensions is \(\left\lbrack {\mathrm{{TP}},\mathrm{{CP}},\mathrm{{PP}},\mathrm{{DP}}}\right\rbrack\) ,where \(\mathrm{{DP}}\) corresponds to FSDP. The outermost parallelism dimensions, PP and DP, may communicate through a multi-hop network, with latency up to tens of microseconds. The original NCCL collectives - all-gather and reduce-scatter in FSDP, and point-to-point in PP-require data chunking and staged data copy. This approach incurs several inefficiencies, including (1) requiring a large number of small control messages to be exchanged over the network to facilitate data transfer, (2) extra memory-copy operations, and (3) using extra GPU cycles for communication. For Llama 3 training, we address a subset of these inefficiencies by tuning chunking and data transfer to fit our network latencies, which can be as high as tens of microseconds for a large cluster. We also allow small control messages to traverse our network at a higher priority, especially avoiding being head-of-line blocked in deep-buffer core switches. Our ongoing work for future Llama versions involves making deeper changes in NCCLX to holistically address all the aforementioned problems.

我们为Llama 3开发的集体通信库基于Nvidia的NCCL库的一个分支，称为NCCLX。NCCLX显著提升了NCCL的性能，特别是在更高延迟的网络中。回想一下，并行维度的顺序是\(\left\lbrack {\mathrm{{TP}},\mathrm{{CP}},\mathrm{{PP}},\mathrm{{DP}}}\right\rbrack\)，其中\(\mathrm{{DP}}\)对应于FSDP。最外层的并行维度，PP和DP，可能通过多跳网络进行通信，延迟可达数十微秒。原始的NCCL集体操作——FSDP中的全收集和减少散布，以及PP中的点对点通信——需要数据分块和分阶段数据复制。这种方法会导致几个低效问题，包括（1）需要大量小控制消息通过网络交换以促进数据传输，（2）额外的内存复制操作，以及（3）使用额外的GPU周期进行通信。对于Llama 3的训练，我们通过调整分块和数据传输以适应我们的网络延迟（对于大型集群，延迟可能高达数十微秒）来解决这些低效问题的一部分。我们还允许小控制消息以更高优先级在我们的网络中传输，特别是避免在深度缓冲核心交换机中被头阻塞。我们正在进行的工作是为未来的Llama版本在NCCLX中进行更深入的更改，以全面解决上述所有问题。

Table 5 Root-cause categorization of unexpected interruptions during a 54-day period of Llama 3 405B pre-training. \(\mathrm{{About}}\) \({78}\%\) of unexpected interruptions were attributed to confirmed or suspected hardware issues.

表5 在Llama 3 405B预训练的54天期间，意外中断的根本原因分类。\(\mathrm{{About}}\) \({78}\%\)的意外中断归因于已确认或疑似硬件问题。

3.3.4 Reliability and Operational Challenges 可靠性与运营挑战

The complexity and potential failure scenarios of \({16}\mathrm{\;K}\) GPU training surpass those of much larger CPU clusters that we have operated. Moreover, the synchronous nature of training makes it less fault-tolerant - a single GPU failure may require a restart of the entire job. Despite these challenges, for Llama 3, we achieved higher than \({90}\%\) effective training time while supporting automated cluster maintenance,such as firmware and Linux kernel upgrades (Vigraham and Leonhardi, 2024), which resulted in at least one training interruption daily. The effective training time measures the time spent on useful training over the elapsed time.

\({16}\mathrm{\;K}\) GPU训练的复杂性和潜在故障场景超过了我们运营的许多大型CPU集群。此外，训练的同步性质使其容错性较低——单个GPU故障可能需要重新启动整个任务。尽管存在这些挑战，对于Llama 3，我们实现了超过\({90}\%\)的有效训练时间，同时支持自动化集群维护，如固件和Linux内核升级（Vigraham和Leonhardi，2024），这导致了至少每天一次的训练中断。有效训练时间衡量了在总时间中用于有效训练的时间。

During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator-initiated operations like configuration or dataset updates. The remaining 419 were unexpected interruptions, which are classified in Table 5. Approximately \({78}\%\) of the unexpected interruptions are attributed to confirmed hardware issues, such as GPU or host component failures, or suspected hardware-related issues like silent data corruption and unplanned individual host maintenance events. GPU issues are the largest category, accounting for \({58.7}\%\) of all unexpected issues. Despite the large number of failures,significant manual intervention was required only three times during this period, with the rest of issues handled by automation.

在预训练的54天快照期间，我们经历了总共466次任务中断。其中，47次是由于自动化维护操作（如固件升级）或操作员发起的操作（如配置或数据集更新）导致的计划中断。其余的419次是意外中断，分类见表5。大约\({78}\%\)的意外中断归因于已确认的硬件问题，如GPU或主机组件故障，或疑似硬件相关问题，如静默数据损坏和未计划的单个主机维护事件。GPU问题是最大的类别，占所有意外问题的\({58.7}\%\)。尽管故障数量众多，但在此期间仅三次需要大量人工干预，其余问题均由自动化处理。

To increase the effective training time, we reduced job startup and checkpointing time, and developed tools for fast diagnosis and problem resolution. We extensively use PyTorch's built-in NCCL flight recorder (Ansel et al., 2024), a feature that captures collective metadata and stack traces into a ring buffer, and hence allowing us to diagnose hangs and performance issues quickly at scale, particularly with regard to NCCLX. Using this, we efficiently record every communication event and the duration of each collective operation, and also automatically dump tracing data on NCCLX watchdog or heartbeat timeout. We enable more computationally intensive tracing operations and metadata collection selectively as needed live in production through online configuration changes (Tang et al., 2015) without needing a code release or job restart.

为了增加有效的训练时间，我们减少了作业启动和检查点时间，并开发了快速诊断和问题解决工具。我们广泛使用 PyTorch 内置的 NCCL 飞行记录器（Ansel et al., 2024），该功能捕获集体元数据和堆栈跟踪到环形缓冲区，从而使我们能够在大规模上快速诊断挂起和性能问题，特别是在 NCCLX 方面。通过使用这个工具，我们高效地记录每个通信事件和每次集体操作的持续时间，并在 NCCLX 看门狗或心跳超时时自动转储跟踪数据。我们根据需要在生产环境中通过在线配置更改（Tang et al., 2015）有选择地启用更多计算密集型的跟踪操作和元数据收集，而无需代码发布或作业重启。

Debugging issues in large-scale training is complicated by the mixed use of NVLink and RoCE in our network. Data transfer over NVLink typically occurs through load/store operations issued by CUDA kernels, and failures in either the remote GPU or NVLink connectivity often manifest as stalled load/store operations within CUDA kernels without returning a clear error code. NCCLX enhances the speed and accuracy of failure detection and localization through a tight co-design with PyTorch, allowing PyTorch to access NCCLX's internal state and track relevant information. While stalls due to NVLink failures cannot be completely prevented, our system monitors the state of the communication library and automatically times out when such a stall is detected. Additionally, NCCLX traces the kernel and network activities of each NCCLX communication and provides a snapshot of the failing NCCLX collective's internal state, including finished and pending data transfers between all ranks. We analyze this data to debug NCCLX scaling issues.

在大规模训练中调试问题因我们网络中混合使用 NVLink 和 RoCE 而变得复杂。通过 NVLink 的数据传输通常由 CUDA 内核发出的加载/存储操作完成，远程 GPU 或 NVLink 连接性的故障通常表现为 CUDA 内核中停滞的加载/存储操作，而不会返回明确的错误代码。NCCLX 通过与 PyTorch 的紧密协同设计，提高了故障检测和定位的速度和准确性，使 PyTorch 能够访问 NCCLX 的内部状态并跟踪相关信息。虽然无法完全防止因 NVLink 故障导致的停滞，但我们的系统监控通信库的状态，并在检测到此类停滞时自动超时。此外，NCCLX 跟踪每个 NCCLX 通信的内核和网络活动，并提供失败 NCCLX 集体的内部状态快照，包括所有等级之间的已完成和待处理数据传输。我们分析这些数据以调试 NCCLX 的扩展问题。

Sometimes, hardware issues may cause still-functioning but slow stragglers that are hard to detect. Even a single straggler can slow down thousands of other GPUs, often appearing as functioning but slow communications. We developed tools to prioritize potentially problematic communications from selected process groups. By investigating just a few top suspects, we were usually able to effectively identify the stragglers.

有时，硬件问题可能导致仍在运行但速度缓慢的落后者，这些落后者难以检测。即使单个落后者也可能拖慢成千上万台其他GPU的速度，通常表现为功能正常但通信缓慢。我们开发了工具，用于从选定的进程组中优先处理可能存在问题的通信。通过调查少数几个主要嫌疑对象，我们通常能够有效地识别出落后者。

One interesting observation is the impact of environmental factors on training performance at scale. For Llama 3405B, we noted a diurnal 1-2% throughput variation based on time-of-day. This fluctuation is the result of higher mid-day temperatures impacting GPU dynamic voltage and frequency scaling.

一个有趣的观察是环境因素对大规模训练性能的影响。对于Llama 3405B，我们注意到基于时间的日变化，吞吐量有1-2%的波动。这种波动是中午高温影响GPU动态电压和频率缩放的结果。

During training, tens of thousands of GPUs may increase or decrease power consumption at the same time, for example, due to all GPUs waiting for checkpointing or collective communications to finish, or the startup or shutdown of the entire training job. When this happens, it can result in instant fluctuations of power consumption across the data center on the order of tens of megawatts, stretching the limits of the power grid. This is an ongoing challenge for us as we scale training for future, even larger Llama models.

在训练过程中，成千上万台GPU可能会同时增加或减少功耗，例如，由于所有GPU等待检查点或集体通信完成，或者整个训练作业的启动或关闭。当这种情况发生时，可能会导致数据中心功耗瞬间波动，达到数十兆瓦的量级，这超出了电网的承受极限。随着我们为未来更大的Llama模型进行训练规模的扩大，这是一个持续的挑战。

3.4 Training Recipe 训练配方

The recipe used to pre-train Llama 3405B consists of three main stages: (1) initial pre-training, (2) long-context pre-training, and (3) annealing. The three stages are described separately below. We use similar recipes to pre-train the \(8\mathrm{\;B}\) and \({70}\mathrm{\;B}\) models.

用于预训练Llama 3405B的配方包含三个主要阶段：（1）初始预训练，（2）长上下文预训练，和（3）退火。以下分别描述这三个阶段。我们使用类似的配方来预训练\(8\mathrm{\;B}\)和\({70}\mathrm{\;B}\)模型。

3.4.1 Initial Pre-Training 初始预训练

We pre-train Llama 3 405B using AdamW with a peak learning rate of \(8 \times {10}^{-5}\) ,a linear warm up of 8,000 steps,and a cosine learning rate schedule decaying to \(8 \times {10}^{-7}\) over \(1,{200},{000}\) steps. We use a lower batch size early in training to improve training stability, and increase it subsequently to improve efficiency. Specifically, we use an initial batch size of \(4\mathrm{M}\) tokens and sequences of length 4,096,and double these values to a batch size of \(8\mathrm{M}\) sequences of \(8,{192}\) tokens after pre-training \({252}\mathrm{M}\) tokens. We double the batch size again to \({16}\mathrm{M}\) after pre-training on 2.87 T tokens. We found this training recipe to be very stable: we observed few loss spikes and did not require interventions to correct for model training divergence.

我们使用AdamW对Llama 3 405B进行预训练，设定峰值学习率为\(8 \times {10}^{-5}\)，采用线性预热8,000步，以及余弦学习率调度，在\(1,{200},{000}\)步内衰减至\(8 \times {10}^{-7}\)。在训练初期使用较低的批次大小以提高训练稳定性，随后增加批次大小以提高效率。具体而言，我们初始批次大小为\(4\mathrm{M}\)个令牌，序列长度为4,096，并在预训练\({252}\mathrm{M}\)个令牌后，将批次大小和序列长度分别翻倍至\(8\mathrm{M}\)和\(8,{192}\)。在预训练2.87万亿个令牌后，我们将批次大小再次翻倍至\({16}\mathrm{M}\)。我们发现这种训练配方非常稳定：观察到的损失峰值很少，且无需干预来纠正模型训练偏差。

Adjusting the data mix. We made a several adjustments to the pre-training data mix during training to improve model performance on particular downstream tasks. In particular, we increased the percentage of non-English data during pre-training to improve the multilingual performance of Llama 3. We also upsample mathematical data to improve the model's mathematical reasoning performance, we added more recent web data in the later stages of pre-training to advance the model's knowledge cut-off, and we downsampled subsets of the pre-training data that were later identified as being lower quality.

调整数据混合比例。在训练过程中，我们对预训练数据混合比例进行了多次调整，以提高模型在特定下游任务上的性能。特别是，我们增加了非英语数据的百分比，以提升Llama 3的多语言性能。我们还增加了数学数据的采样率，以提高模型的数学推理性能，在预训练后期阶段增加了更近期的网络数据，以推进模型的知识截止点，并对后来被识别为质量较低的预训练数据子集进行了降采样。

3.4.2 Long Context Pre-Training 长上下文预训练

In the final stages of pre-training, we train on long sequences to support context windows of up to \({128}\mathrm{K}\) tokens. We do not train on long sequences earlier because the compute in self-attention layers grows quadratically in the sequence length. We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves "needle in a haystack" tasks up to that length. In Llama 3405B pre-training, we increased context length gradually in six stages,starting from the original \(8\mathrm{\;K}\) context window and ending in the final \({128}\mathrm{\;K}\) context window. This long-context pre-training stage was performed using approximately \({800}\mathrm{\;B}\) training tokens.

在预训练的最后阶段，我们训练长序列以支持最长 \({128}\mathrm{K}\) 个标记的上下文窗口。我们不在早期训练长序列，因为自注意力层的计算量随序列长度呈二次方增长。我们逐步增加支持的上下文长度，预训练直到模型成功适应增加的上下文长度。我们通过测量（1）模型在短上下文评估中的性能是否完全恢复，以及（2）模型是否完美解决了长达该长度的“大海捞针”任务，来评估成功的适应。在 Llama 3405B 预训练中，我们分六个阶段逐步增加上下文长度，从最初的 \(8\mathrm{\;K}\) 上下文窗口开始，到最终的 \({128}\mathrm{\;K}\) 上下文窗口结束。这一长上下文预训练阶段使用了大约 \({800}\mathrm{\;B}\) 个训练标记。

Figure 7 Illustration of the overall post-training approach for Llama 3. Our post-training strategy involves rejection sampling, supervised finetuning, and direct preference optimization. See text for details.

图 7 展示了 Llama 3 的整体后训练方法。我们的后训练策略涉及拒绝采样、监督微调和直接偏好优化。详见正文。

3.4.3 Annealing 退火

During pre-training on the final \({40}\mathrm{M}\) tokens,we linearly annealed the learning rate to 0,maintaining a context length of \({128}\mathrm{\;K}\) tokens. During this annealing phase,we also adjusted the data mix to upsample data sources of very high quality; see Section 3.1.3. Finally, we compute the average of model checkpoints (Polyak (1991) averaging) during annealing to produce the final pre-trained model.

在预训练最后 \({40}\mathrm{M}\) 个标记的过程中，我们将学习率线性退火至 0，保持 \({128}\mathrm{\;K}\) 个标记的上下文长度。在这一退火阶段，我们还调整了数据混合，以增加高质量数据源的采样率；参见第 3.1.3 节。最后，我们在退火期间计算模型检查点的平均值（Polyak (1991) 平均），以生成最终的预训练模型。

4 Post-Training 后训练

We produce the aligned Llama 3 models by applying several rounds of post-training, \({}^{6}\) or aligning the model with human feedback (Ouyang et al., 2022; Rafailov et al., 2024) on top of a pre-trained checkpoint. Each round of post-training involves supervised finetuning (SFT) followed by Direct Preference Optimization (DPO; Rafailov et al., 2024) on examples collected either via human annotations or generated synthetically. Our post-training modeling and data approaches are described in Sections 4.1 and 4.2 respectively. We further detail custom data curation strategies to improve the reasoning, coding, factuality, multilingual, tool use, long context, and precise instruction following in Section 4.3.

我们通过应用多轮后训练，\({}^{6}\) 或在预训练检查点基础上结合人类反馈（Ouyang et al., 2022; Rafailov et al., 2024）来生成对齐的 Llama 3 模型。每轮后训练包括监督微调（SFT），随后在通过人工标注或合成生成的示例上进行直接偏好优化（DPO；Rafailov et al., 2024）。我们的后训练建模和数据方法分别在第 4.1 和 4.2 节中描述。我们进一步详细介绍了定制数据筛选策略，以提高推理、编码、事实性、多语言、工具使用、长上下文和精确指令遵循能力，详见第 4.3 节。

4.1 Modeling 建模

The backbone of our post-training strategy is a reward model and a language model. We first train a reward model on top of the pre-trained checkpoint using human-annotated preference data (see Section 4.1.2). We then finetune pre-trained checkpoints with supervised finetuning (SFT; see Section 4.1.3), and further align the checkpoints with Direct Preference Optimization (DPO; see Section 4.1.4). This process is illustrated in Figure 7. Unless otherwise noted, our modeling procedure applies to Llama 3 405B, and we refer to Llama 3 405B as Llama 3 for simplicity.

我们的后训练策略的核心是一个奖励模型和一个语言模型。我们首先在预训练检查点基础上使用人工标注的偏好数据（见第 4.1.2 节）训练一个奖励模型。然后，我们对预训练检查点进行监督微调（SFT；见第 4.1.3 节），并进一步通过直接偏好优化（DPO；见第 4.1.4 节）对检查点进行对齐。此过程如图 7 所示。除非另有说明，我们的建模过程适用于 Llama 3 405B，为简洁起见，我们将 Llama 3 405B 简称为 Llama 3。

4.1.1 Chat Dialog Format 聊天对话格式

To tune LLMs for human-AI interaction, we need to define a chat dialog protocol for the model to understand human instructions and perform conversational tasks. Compared to its predecessor, Llama 3 has new capabilities such as tool use (Section 4.3.5) which may require generating multiple messages and sending them to different locations (e.g., user, ipython) within a single dialog turn. To support this, we design a new multi-message chat protocol which uses various special header and termination tokens. The header tokens are used to indicate the source and destination of each message in a conversation. Similarly, the termination tokens indicate when it is the time to alternate between human and AI to speak.

为了调整大型语言模型以适应人机交互，我们需要为模型定义一个聊天对话协议，使其能够理解人类指令并执行对话任务。与前代产品相比，Llama 3 新增了工具使用（第 4.3.5 节）等能力，这可能需要在单个对话轮次内生成多条消息并发送至不同位置（例如，用户、ipython）。为此，我们设计了一种新的多消息聊天协议，该协议使用各种特殊头部和终止标记。头部标记用于指示对话中每条消息的来源和目的地。同样，终止标记用于指示何时轮到人类和人工智能交替发言。

\({}^{6}\) We use the term "post-training" to refer to any model training that happens outside of pre-training.

\({}^{6}\) 我们使用“后训练”这一术语来指代在预训练之外进行的任何模型训练。

4.1.2 Reward Modeling 奖励模型

We train a reward model (RM) covering different capabilities on top of the pre-trained checkpoint. The training objective is the same as Llama 2 except that we remove the margin term in the loss, as we observe diminishing improvements after data scaling. Following Llama 2, we use all of our preference data for reward modeling after filtering out samples with similar responses. In addition to standard preference pair of (chosen, rejected) response, annotations also create a third "edited response" for some prompts, where the chosen response from the pair is further edited for improvement (see Section 4.2.1). Hence, each preference ranking sample has two or three responses with clear ranking (edited \(>\) chosen \(>\) rejected). We concatenate the prompt and multiple responses into a single row during training with responses randomly shuffled. This is an approximation to the standard scenario of putting the responses in separate rows and computing the scores, but in our ablations, this approach improves training efficiency without a loss in accuracy.

我们在预训练检查点的基础上训练了一个涵盖不同能力的奖励模型（RM）。训练目标与 Llama 2 相同，只是我们去除了损失中的边际项，因为我们观察到数据规模扩大后改进效果递减。与 Llama 2 一样，我们在过滤掉响应相似的样本后，使用所有的偏好数据进行奖励模型训练。除了标准的偏好对（被选中的，被拒绝的）响应外，注释还为某些提示创建了第三种“编辑后的响应”，其中来自配对的被选中的响应被进一步编辑以改进（见第 4.2.1 节）。因此，每个偏好排序样本都有两个或三个响应，且排序明确（编辑后的 \(>\) 被选中的 \(>\) 被拒绝的）。我们在训练过程中将提示和多个响应连接成单行，并对响应进行随机洗牌。这是将响应放在单独行并计算分数的标准场景的近似，但在我们的消融实验中，这种方法提高了训练效率，且没有损失准确性。

4.1.3 Supervised Finetuning 监督微调

The reward model is then used to perform rejection sampling on our human annotation prompts, the details of which are described in Section 4.2. Together with this rejection-sampled data and other data sources (including synthetic data), we finetune the pre-trained language model using a standard cross entropy loss on the target tokens (while masking loss on prompt tokens). More details about the data mix can be found in Section 4.2. We refer to this stage as supervised finetuning (SFT; Wei et al., 2022a; Sanh et al., 2022; Wang et al., 2022b), even though many of the training targets are model-generated. Our largest models are finetuned with a learning rate of \({10}^{-5}\) over the course of \({8.5}\mathrm{\;K}\) to \(9\mathrm{\;K}\) steps. We found these hyperparameter settings to work well across different rounds and data mixes.

然后，奖励模型用于对我们的标注提示进行拒绝采样，详细信息在第4.2节中描述。结合这些拒绝采样数据和其他数据源（包括合成数据），我们使用标准交叉熵损失对目标词进行微调预训练语言模型（同时屏蔽提示词的损失）。有关数据混合的更多细节可以在第4.2节中找到。我们称这一阶段为监督微调（SFT；Wei et al., 2022a；Sanh et al., 2022；Wang et al., 2022b），尽管许多训练目标是模型生成的。我们的最大模型在\({8.5}\mathrm{\;K}\)到\(9\mathrm{\;K}\)步的过程中以\({10}^{-5}\)的学习率进行微调。我们发现这些超参数设置在不同轮次和数据混合中表现良好。

4.1.4 Direct Preference Optimization 直接偏好优化

We further train our SFT models with Direct Preference Optimization (DPO; Rafailov et al., 2024) for human preference alignment. For training, we primarily use the most recent batches of preference data collected using the best performing models from the previous alignment rounds. As a result, our training data conforms better to the distribution of the policy model that is being optimized in each round. We also explored on-policy algorithms such as PPO (Schulman et al., 2017), but found that DPO required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval (Zhou et al., 2023). For Llama 3,we use a learning rate of \({10}^{-5}\) and set the \(\beta\) hyper-parameter to be 0.1 . In addition,we apply the following algorithmic modifications to DPO:

我们进一步使用直接偏好优化（DPO；Rafailov et al., 2024）对SFT模型进行人类偏好对齐训练。在训练中，我们主要使用通过上一轮对齐中表现最佳的模型收集的最新偏好数据批次。因此，我们的训练数据更符合每一轮正在优化的策略模型的分布。我们还探索了如PPO（Schulman et al., 2017）这样的在线策略算法，但发现DPO对大规模模型需要的计算量更少，且表现更好，尤其是在遵循指令的基准测试如IFEval（Zhou et al., 2023）上。对于Llama 3，我们使用\({10}^{-5}\)的学习率，并将\(\beta\)超参数设置为0.1。此外，我们对DPO应用了以下算法修改：

• Masking out formatting tokens in DPO loss: We mask out special formatting tokens including header and termination tokens (described in Section 4.1.1) from both chosen and rejected responses in the loss to stabilize DPO training. We observe that having these tokens contribute to the loss may lead to undesired model behaviors such as tail repetition or abruptly generating termination tokens. We hypothesize that this is due to the contrastive nature of the DPO loss - the presence of common tokens in both chosen and rejected responses leads to a conflicting learning objective as the model needs to increase and reduce the likelihood of these tokens simultaneously.

• 在DPO损失中屏蔽格式化标记：我们从损失中的选定和拒绝响应中屏蔽掉包括标题和终止标记（在4.1.1节中描述）在内的特殊格式化标记，以稳定DPO训练。我们观察到，这些标记对损失的贡献可能导致不希望的模型行为，如尾部重复或突然生成终止标记。我们假设这是由于DPO损失的对比性质——选定和拒绝响应中共同标记的存在导致了一个冲突的学习目标，因为模型需要同时增加和减少这些标记的可能性。

• Regularization with NLL loss: We add an additional negative log-likelihood (NLL) loss term with a scaling coefficient of 0.2 on the chosen sequences, similar to Pang et al. (2024). This helps further stabilize DPO training by maintaining desired formatting for generation and preventing the decrease of log probability of chosen responses (Pang et al., 2024; Pal et al., 2024).

• 使用NLL损失进行正则化：我们在选定序列上添加一个额外的负对数似然（NLL）损失项，其缩放系数为0.2，类似于Pang等人（2024年）的做法。这有助于通过保持生成所需的格式并防止选定响应的对数概率下降来进一步稳定DPO训练（Pang等人，2024年；Pal等人，2024年）。

4.1.5 Model Averaging 模型平均

Finally, we average models obtained from experiments using various versions of data or hyperparameters at each RM, SFT, or DPO stage (Izmailov et al., 2019; Wortsman et al., 2022; Li et al., 2022).

最后，我们平均了在每个RM、SFT或DPO阶段使用不同版本数据或超参数获得的模型（Izmailov等人，2019年；Wortsman等人，2022年；Li等人，2022年）。

Table 6 Statistics of human preference data. We list statistics of the internally collected human preference data used for Llama 3 alignment. We ask annotators to perform multi-turn dialogues with the models and make comparisons among responses at each turn. In post-processing, we split each dialogue to multiple examples at a turn level. Each example consists of a prompt (including previous dialog if available) and a response (e.g., chosen or rejected response).

表6 人类偏好数据的统计。我们列出了用于Llama 3对齐的内部收集的人类偏好数据的统计信息。我们要求标注者与模型进行多轮对话，并在每轮中对响应进行比较。在后处理中，我们将每个对话拆分为多个以轮为单位的示例。每个示例包括一个提示（如果可用，包括之前的对话）和一个响应（例如，选定的或拒绝的响应）。

4.1.6 Iterative Rounds 迭代轮次

Following Llama 2, we apply the above methods in six rounds. In each cycle, we collect new preference annotations and SFT data, sampling synthetic data from the latest models.

继Llama 2之后，我们在六轮中应用上述方法。在每个周期中，我们收集新的偏好注释和SFT数据，从最新模型中抽取合成数据。

4.2 Post-training Data 后训练数据

The post-training data composition plays a critical role in the usefulness and behavior of language models. In this section, we discuss our human annotation procedures and preference data collection (Section 4.2.1), the composition of our SFT data (Section 4.2.2), and methods for data quality control and cleaning (Section 4.2.3).

后训练数据组成对语言模型的有用性和行为起着关键作用。本节中，我们将讨论我们的人工注释流程和偏好数据收集（第4.2.1节），我们的SFT数据组成（第4.2.2节），以及数据质量控制和清理的方法（第4.2.3节）。

4.2.1 Preference Data 偏好数据

Our preference data annotation process is similar to Llama 2. We deploy multiple models for annotation after each round and sample two responses from two different models for each user prompt. These models can be trained with different data mixes and alignment recipes, allowing for different capability strength (e.g., code expertise) and increased data diversity. We ask annotators to rate the strength of their preference by categorizing it into one of four levels, based on how much more they prefer the chosen response over the rejected one: significantly better, better, slightly better, or marginally better. We also incorporate an editing step after preference ranking to encourage annotators to further improve the preferred response. Annotators edit the chosen response directly or prompt the model with feedback to refine its own response. Consequently, a portion of our preference data has three responses ranked (edited \(>\) chosen \(>\) rejected).

我们的偏好数据注释流程与Llama 2类似。在每一轮之后，我们部署多个模型进行注释，并为每个用户提示从两个不同模型中抽取两个响应。这些模型可以使用不同的数据混合和调整方案进行训练，从而允许不同的能力强度（例如，代码专业知识）和增加数据多样性。我们要求注释者根据他们对所选响应相对于被拒绝响应的偏好程度，将其分类为四个级别之一：显著更好、更好、略好或稍好。我们还引入了一个编辑步骤，在偏好排序后鼓励注释者进一步改进所选响应。注释者直接编辑所选响应或通过反馈提示模型改进其自身响应。因此，我们的部分偏好数据包含三个排序的响应（编辑后的 \(>\) 所选的 \(>\) 被拒绝的）。

In Table 6, we report the statistics of preference annotations that we use for Llama 3 training. General English covers multiple subcategories such as knowledge-based question and answering or precise instruction-following, which fall outside the scope of specific capabilities. Compared to Llama 2, we observe an increase in the average length of prompt and response, suggesting that we train Llama 3 on more complex tasks. In addition, we implement a quality analysis and human evaluation process to rigorously assess the data collected, allowing us to refine our prompts and provide systematic, actionable feedback to annotators. For example, as Llama 3 improves after each round, we increase prompt complexity accordingly to target areas where the model lags.

在表6中，我们报告了用于Llama 3训练的偏好注释的统计数据。通用英语涵盖了多个子类别，如基于知识的问答或精确的指令遵循，这些都超出了特定能力的范围。与Llama 2相比，我们观察到提示和响应的平均长度增加，这表明我们训练Llama 3处理更复杂的任务。此外，我们实施了质量分析和人工评估流程，以严格评估收集的数据，使我们能够优化提示并提供系统性、可操作的反馈给注释者。例如，随着Llama 3在每一轮改进后，我们相应增加提示的复杂度，以针对模型落后的领域。

In each round of post-training, we use all the preference data that is available at the time for reward modeling, while only using the latest batches from various capabilities for DPO training. For both reward modeling and DPO, we use samples that are labeled as the chosen response being significantly better or better than the rejected counterpart for training and discard samples with similar responses.

在每一轮后训练中，我们使用当时可用的所有偏好数据进行奖励建模，而仅使用来自各种能力的最新批次进行DPO训练。对于奖励建模和DPO，我们使用标记为被选响应明显优于或优于被拒绝的对应响应的样本进行训练，并丢弃响应相似的样本。

4.2.2 SFT Data SFT数据

Our finetuning data is largely comprised of the following sources:

我们的微调数据主要由以下来源组成：

• Prompts from our human annotation collection with rejection-sampled responses.

• 来自我们人工注释收集的提示与拒绝采样的响应。

• Synthetic data targeting specific capabilities (see Section 4.3 for more details).

• 针对特定能力生成的合成数据（更多细节见第4.3节）。

Table 7 Statistics of SFT data. We list internally collected SFT data used for Llama 3 alignment. Each SFT example consists of a context (i.e., all conversation turns except the last one) and a final response.

表7 SFT数据的统计。我们列出了用于Llama 3对齐的内部收集的SFT数据。每个SFT示例包括一个上下文（即所有对话轮次，除最后一轮外）和一个最终响应。

• Small amounts of human-curated data (see Section 4.3 for more details).

• 少量人工精选数据（更多细节见第4.3节）。

As our post-training rounds progress, we develop stronger Llama 3 variants that we use to collect larger datasets that cover a wide range of complex capabilities. In this section, we discuss the details for the rejection-sampling procedure and overall composition of our final SFT datamix.

随着我们后训练轮次的进展，我们开发了更强大的Llama 3变体，用于收集涵盖广泛复杂能力的大型数据集。在本节中，我们将讨论拒绝采样程序的细节以及我们最终SFT数据集的整体组成。

Rejection sampling. During rejection sampling (RS), for each prompt collected during human annotation (Section 4.2.1) we sample \(K\) (typically between 10 and 30) outputs from the latest chat model policy (usually the best performing checkpoint from the previous post-training iteration, or the best performing checkpoint for a particular capability) and use our reward model to select the best candidate, consistent with Bai et al. (2022). In later rounds of post-training, we introduce system prompts to steer RS responses to conform with desirable tone, style, or formatting, which might be different for different capabilities.

拒绝采样。在拒绝采样（RS）过程中，对于在人工标注期间收集的每个提示（第4.2.1节），我们从最新的聊天模型策略（通常是上一轮后训练迭代中表现最佳的检查点，或针对特定能力表现最佳的检查点）中采样 \(K\)（通常在10到30之间）输出，并使用我们的奖励模型选择最佳候选，与Bai等人（2022）一致。在后训练的后续轮次中，我们引入系统提示以引导RS响应符合理想的语调、风格或格式，这些可能因不同能力而异。

To increase the efficiency of rejection sampling, we adopt PagedAttention (Kwon et al., 2023). PagedAttention enhances memory efficiency through dynamic key-value cache allocation. It supports arbitrary output lengths by dynamically scheduling requests based on the current cache capacity. Unfortunately, this carries the risk of swap-out when running out of memory. To eliminate such swap overhead, we define a maximum output length and perform a request only if sufficient memory is available to fit an output with that length. PagedAttention also enables us to share the key-value cache pages for a prompt across all corresponding outputs. Together, this leads to a throughput improvement of over \(2 \times\) during rejection sampling.

为了提高拒绝采样的效率，我们采用了PagedAttention（Kwon等人，2023）。PagedAttention通过动态键值缓存分配增强了内存效率。它通过根据当前缓存容量动态调度请求来支持任意输出长度。不幸的是，这带来了内存耗尽时交换出去的风险。为了消除这种交换开销，我们定义了最大输出长度，并且仅在有足够内存容纳该长度的输出时才执行请求。PagedAttention还使我们能够为提示共享所有相应输出的键值缓存页。这些共同导致了拒绝采样期间吞吐量提高了超过 \(2 \times\)。

Overall data composition. Table 7 shows data statistics for each broad category of our "helpfulness" mix. While SFT and preference data contain overlapping domains, they are curated differently, yielding distinct count statistics. In Section 4.2.3 we describe techniques for categorizing topic, complexity, and quality of our data samples. In each round of post-training, we adjust our overall data mix carefully across these axes to tune performance across a wide range of benchmarks. Our final data mix epochs multiple times on some high quality sources and downsamples others.

总体数据构成。表7显示了我们“帮助性”混合中每个大类别的数据统计。尽管SFT和偏好数据包含重叠的领域，但它们的策划方式不同，产生了不同的计数统计。在第4.2.3节中，我们描述了用于分类主题、复杂性和数据样本质量的技术。在每一轮后训练中，我们仔细调整这些轴上的总体数据混合，以在广泛的基准上调整性能。我们的最终数据混合在某些高质量来源上多次迭代，并对其他来源进行降采样。

4.2.3 Data Processing and Quality Control 数据处理和质量控制

Given that most of our training data is model-generated, it requires careful cleaning and quality control.

鉴于我们的大部分训练数据是模型生成的，因此需要仔细进行清洗和质量控制。

Data cleaning. In the early rounds, we observed a number of undesirable patterns common in our data, such as excessive use of emojis or exclamation points. Therefore, we implement a series of rule-based data removal and modification strategies to filter or clean problematic data. For example, to mitigate overly-apologetic tonal issues, we identify overused phrases (such as "I'm sorry" or "I apologize") and carefully balance the proportion of such samples in our dataset.

数据清洗。在早期轮次中，我们观察到数据中存在一些不良模式，例如过度使用表情符号或感叹号。因此，我们实施了一系列基于规则的数据删除和修改策略来过滤或清理有问题的数据。例如，为了缓解过度道歉的语调问题，我们识别出过度使用的短语（如“I'm sorry”或“I apologize”），并仔细平衡这些样本在我们数据集中的比例。

Data pruning. We also apply a collection of model-based techniques to remove low-quality training samples and improve overall model performance:

数据修剪。我们还应用了一系列基于模型的技术来移除低质量的训练样本，并提高整体模型性能：

• Topic classification: We first finetune Llama 3 8B into a topic classifier, and perform inference over all data to classify it into both coarsely-grained buckets ("mathematical reasoning") and fine-grained buckets ("geometry and trigonometry").

• 主题分类：我们首先将 Llama 3 8B 微调为一个主题分类器，并对所有数据进行推理，将其分类为粗粒度类别（如“数学推理”）和细粒度类别（如“几何与三角学”）。

• Quality scoring: We use both reward model and Llama-based signals to obtain a quality score for each sample. For an RM-based score, we consider data that is in the top quartile of RM scores as high quality. For a Llama-based score, we prompt Llama 3 checkpoint to rate each sample on a three-point scale for general English data (accuracy, instruction following, and tone/presentation) and a two-point scale for coding data (bug identification and user intention), and consider samples that obtain the maximum score as high quality. The RM and Llama-based scores have high disagreement rates, and we find that combining these signals yield the best recall on our internal test set. Ultimately, we select examples that are marked as high quality by the RM or the Llama-based filter.

• 质量评分：我们同时使用奖励模型和基于 Llama 的信号来为每个样本获取质量评分。对于基于 RM 的评分，我们认为位于 RM 评分前四分之一的数据显示为高质量。对于基于 Llama 的评分，我们提示 Llama 3 检查点对每个样本在一般英语数据（准确性、指令遵循和语调/呈现）上进行三点量表评分，在编程数据（错误识别和用户意图）上进行两点量表评分，并将获得最高分的样本视为高质量。RM 和基于 Llama 的评分存在高度不一致，我们发现结合这些信号在我们的内部测试集上获得了最佳召回率。最终，我们选择了被 RM 或基于 Llama 的过滤器标记为高质量的示例。

• Difficulty scoring: Because we are also interested in prioritizing examples that are more complex for the model, we score data using two measures of difficulty: Instag (Lu et al., 2023) and Llama-based scoring. For Instag, we prompt Llama 3 70B to perform intention tagging of SFT prompts, where more intentions implies more complexity. We also prompt Llama 3 to measure the difficulty (Liu et al., 2024c) of dialogs on a three-point scale.

• 难度评分：由于我们也对优先处理模型更复杂的示例感兴趣，我们使用两种难度度量来评分数据：Instag（Lu et al., 2023）和基于Llama的评分。对于Instag，我们提示Llama 3 70B对SFT提示进行意图标记，意图越多意味着越复杂。我们还提示Llama 3在三点量表上测量对话的难度（Liu et al., 2024c）。

• Semantic deduplication: Finally, we perform semantic deduplication (Abbas et al., 2023; Liu et al., 2024c). We first cluster complete dialogs using RoBERTa (Liu et al., 2019b) and within each cluster sort them by quality score \(\times\) difficulty score. We then do greedy selection by iterating through all sorted examples, and only keeping the ones that have maximum cosine similarity less than a threshold to the examples seen so far in the cluster.

• 语义去重：最后，我们进行语义去重（Abbas et al., 2023; Liu et al., 2024c）。我们首先使用RoBERTa（Liu et al., 2019b）对完整对话进行聚类，并在每个聚类中按质量分数\(\times\)难度分数进行排序。然后我们通过遍历所有排序后的示例进行贪心选择，只保留那些与当前聚类中已见示例的余弦相似度小于阈值的示例。

4.3 Capabilities 能力

We highlight special efforts to improve performance for specific capabilities such as code (Section 4.3.1), multilinguality (Section 4.3.2), math and reasoning (Section 4.3.3), long context (Section 4.3.4), tool use (Section 4.3.5), factuality (Section 4.3.6), and steerability (Section 4.3.7).

我们特别强调了针对特定能力提升性能的努力，如代码（第4.3.1节）、多语言能力（第4.3.2节）、数学和推理（第4.3.3节）、长上下文（第4.3.4节）、工具使用（第4.3.5节）、事实性（第4.3.6节）和可操控性（第4.3.7节）。

4.3.1 Code 代码

LLMs for code have received significant attention since the release of Copilot and Codex (Chen et al., 2021). Developers are now widely using these models to generate code snippets, debug, automate tasks, and improve code quality. For Llama 3, we target improving and evaluating code generation, documentation, debugging, and review capabilities for the following high priority programming languages: Python, Java, Javascript, \(\mathrm{C}/\mathrm{C} + +\) ,Typescript,Rust,PHP,HTML/CSS,SQL,bash/shell. Here,we present our work on improving these coding capabilities via training a code expert, generating synthetic data for SFT, improving formatting with system prompt steering, and creating quality filters to remove bad samples from our training data.

自Copilot和Codex（Chen et al., 2021）发布以来，用于代码的大型语言模型受到了广泛关注。开发者现在广泛使用这些模型来生成代码片段、调试、自动化任务以及提高代码质量。对于Llama 3，我们的目标是改进和评估代码生成、文档编写、调试和审查能力，针对以下高优先级编程语言：Python、Java、Javascript、\(\mathrm{C}/\mathrm{C} + +\)、Typescript、Rust、PHP、HTML/CSS、SQL、bash/shell。在这里，我们介绍了通过训练代码专家、生成合成数据用于SFT、通过系统提示引导改进格式以及创建质量过滤器从训练数据中移除不良样本来提高这些编码能力的工作。

Expert training. We train a code expert which we use to collect high quality human annotations for code throughout subsequent rounds of post-training. This is accomplished by branching the main pre-training run and continuing pre-training on a 1T token mix of mostly \(\left( { > {85}\% }\right)\) code data. Continued pre-training on domain-specific data has been shown to be effective for improving performance in a specific domain (Gururangan et al., 2020). We follow a recipe similar to that of CodeLlama (Rozière et al., 2023). For the last several thousand steps of training we perform long-context finetuning (LCFT) to extend the expert's context length to \({16}\mathrm{\;K}\) tokens on a high quality mix of repo-level code data. Finally,we follow the similar post-training modeling recipes described in Section 4.1 to align this model, except with SFT and DPO data mixes primarily targeting code. This model is also used for rejection sampling (Section 4.2.2) for coding prompts.

专家训练。我们训练了一个代码专家，用于在后续的训练后轮次中收集高质量的人工代码注释。这是通过从主预训练运行中分支出一个分支，并在大部分\(\left( { > {85}\% }\right)\)代码数据的1T令牌混合上继续预训练来实现的。继续在特定领域的数据上进行预训练已被证明对提高特定领域的表现有效（Gururangan et al., 2020）。我们遵循类似于CodeLlama（Rozière et al., 2023）的方法。在训练的最后几千步中，我们进行长上下文微调（LCFT），将专家的上下文长度扩展到\({16}\mathrm{\;K}\)令牌，使用高质量的仓库级代码数据混合。最后，我们遵循第4.1节中描述的类似的训练后建模方法来对齐这个模型，除了主要针对代码的SFT和DPO数据混合。该模型还用于编码提示的拒绝采样（第4.2.2节）。

Synthetic data generation. During development, we identified key issues in code generation, including difficulty in following instructions, code syntax errors, incorrect code generation, and difficulty in fixing bugs. While intensive human annotation could theoretically resolve these issues, synthetic data generation offers a complementary approach at a lower cost and higher scale, unconstrained by the expertise level of annotators. As such, we use Llama 3 and the code expert to generate a large quantity of synthetic SFT dialogs.

合成数据生成。在开发过程中，我们发现了代码生成中的关键问题，包括难以遵循指令、代码语法错误、错误的代码生成以及难以修复错误。虽然密集的人工标注理论上可以解决这些问题，但合成数据生成提供了一种成本更低、规模更高的补充方法，不受标注者专业水平的限制。因此，我们使用Llama 3和代码专家生成了大量合成SFT对话。

We describe three high-level approaches for generating synthetic code data. In total, we generate over 2.7M synthetic examples which were used during SFT.

我们描述了三种生成合成代码数据的高级方法。总共生成了超过270万个合成示例，这些示例在SFT期间被使用。

1. Synthetic data generation: execution feedback. The 8B and 70B models show significant performance improvements when trained on data generated by a larger, more competent model. However, our initial experiments revealed that training Llama 3405B on its own generated data is not helpful (and can even degrade performance). To address this limitation, we introduced execution feedback as a source of truth, enabling the model to learn from its mistakes and stay on track. In particular, we generate large dataset of approximately one million synthetic coding dialogues using the following process:

2. 合成数据生成：执行反馈。当8B和70B模型在由更大、更强的模型生成的数据上训练时，显示出显著的性能提升。然而，我们最初的实验表明，在自身生成的数据上训练Llama 3405B没有帮助（甚至可能降低性能）。为了解决这一限制，我们引入了执行反馈作为事实来源，使模型能够从错误中学习并保持在正确的轨道上。特别是，我们使用以下过程生成了大约一百万个合成编码对话的大型数据集：

• Problem description generation: First, we generate a large collection of programming problem descriptions that span a diverse range of topics, including those in the long tail distribution. To achieve this diversity, we sample random code snippets from various sources and prompt the model to generate programming problems inspired by these examples. This allowed us to tap into a wide range of topics and create a comprehensive set of problem descriptions (Wei et al., 2024).

• 问题描述生成：首先，我们生成了大量涵盖广泛主题的编程问题描述，包括长尾分布中的主题。为了实现这种多样性，我们从各种来源抽取随机代码片段，并提示模型根据这些示例生成编程问题。这使我们能够触及广泛的主题并创建全面的问题描述集（Wei等人，2024）。

• Solution generation: Then, we prompt Llama 3 to solve each problem in a given programming language. We observe that adding general rules of good programming to the prompt improves the generated solution quality. Also, we find it is helpful to require the model to explain its thought process in comments.

• 解决方案生成：然后，我们提示Llama 3用给定的编程语言解决每个问题。我们观察到，在提示中添加良好的编程通用规则可以提高生成解决方案的质量。此外，我们发现要求模型在注释中解释其思维过程是有帮助的。

• Correctness analysis: After generating a solution, it is crucial to recognize that its correctness is not guaranteed, and including incorrect solutions in the finetuning dataset could harm the model's quality. While we do not ensure complete correctness, we develop methods to approximate it. To achieve this, we extract the source code from the generated solution and applied a combination of static and dynamic analysis techniques to test its correctness, including:

• 正确性分析：在生成解决方案后，认识到其正确性并未得到保证至关重要，将不正确的解决方案包含在微调数据集中可能会损害模型的质量。虽然我们不保证完全正确，但我们开发了方法来近似正确性。为此，我们从生成的解决方案中提取源代码，并应用静态和动态分析技术的组合来测试其正确性，包括：

• Static analysis: We run all generated code through a parser and a linter to ensure syntactic correctness, catching errors such as syntax errors, use of uninitialized variables or non-imported functions, code style issues, typing errors, and others.

• 静态分析：我们将所有生成的代码通过解析器和代码检查工具运行，以确保语法正确性，捕捉诸如语法错误、未初始化变量的使用或未导入的函数、代码风格问题、类型错误等。

• Unit test generation and execution: For each problem and solution, we prompt the model to generate unit tests, executed in a containerized environment together with the solution, catching run-time execution errors and some semantic errors.

• 单元测试生成与执行：对于每个问题和解决方案，我们提示模型生成单元测试，并在容器化环境中与解决方案一起执行，捕捉运行时执行错误和一些语义错误。

• Error feedback and iterative self-correction: When a solution fails at any step, we prompt the model to revise it. The prompt included the original problem description, the faulty solution, and feedback from the parser/linter/tester (stdout, stderr/ and return code). After a unit test execution failure, the model could either fix the code to pass the existing tests or modify its unit tests to accommodate the generated code. Only dialogs that pass all checks are included in the final dataset,used for supervised finetuning (SFT). Notably,we observed that about \({20}\%\) of solutions were initially incorrect but self-corrected, indicating that the model learned from the execution feedback and improved its performance.

• 错误反馈与迭代自校正：当解决方案在任何步骤失败时，我们提示模型进行修订。提示包括原始问题描述、有问题的解决方案以及来自解析器/代码检查工具/测试器的反馈（标准输出、标准错误和返回代码）。在单元测试执行失败后，模型可以修复代码以通过现有测试，或修改其单元测试以适应生成的代码。只有通过所有检查的对话才包含在最终数据集中，用于监督微调（SFT）。值得注意的是，我们观察到大约 \({20}\%\) 的解决方案最初是不正确的，但经过自我校正，表明模型从执行反馈中学习并提高了其性能。

• Fine-tuning and iterative improvement: The finetuning process is conducted over multiple rounds, with each round building on the previous one. After each round, the model is improved, generating higher-quality synthetic data for the next round. This iterative process allows for progressive refinement and enhancement of the model's performance.

• 微调与迭代改进：微调过程在多个轮次中进行，每一轮都基于前一轮。每一轮之后，模型得到改进，为下一轮生成更高质量的合成数据。这种迭代过程允许逐步细化和增强模型的性能。

2. Synthetic data generation: programming language translation. We observe a performance gap between major programming languages (e.g., Python/C++) and less common ones (e.g., Typescript/PHP). This is not surprising as we have less training data for less common programming languages. To mitigate this, we supplement our existing data by translating data from common programming languages to less common languages (similar to Chen et al. (2023) in the context of reasoning). This is achieved by prompting Llama 3 and ensuring quality via syntax parsing, compilation, and execution. Figure 8 demonstrates an example of synthetic PHP code translated from Python. This improves performance significantly for less common languages as measured by the MultiPL-E (Cassano et al., 2023) benchmark.

3. 合成数据生成：编程语言翻译。我们观察到主流编程语言（如 Python/C++）与较少使用的编程语言（如 Typescript/PHP）之间存在性能差距。这并不令人惊讶，因为我们对于较少使用的编程语言的训练数据较少。为了缓解这一问题，我们通过将常见编程语言的数据翻译为较少使用的语言（类似于 Chen 等人在推理背景下的做法 (2023)）来补充现有数据。这一过程通过提示 Llama 3 并确保通过语法解析、编译和执行来保证质量。图 8 展示了一个从 Python 翻译成 PHP 的合成代码示例。这显著提高了较少使用语言的性能，如 MultiPL-E 基准测试（Cassano 等人，2023）所衡量。

4. Synthetic data generation: backtranslation. To improve certain coding capabilities (e.g., documentation, explanations) where execution feedback is less informative for determining quality, we employ an alternative multi-step approach. Using this procedure, we generated approximately 1.2M synthetic

5. 合成数据生成：回译。为了提高某些编码能力（例如，文档编写、解释），在这些能力中执行反馈对于确定质量的信息较少，我们采用了一种替代的多步骤方法。通过这一过程，我们生成了大约 1.2M 的合成数据。

Figure 8 Code translation example. We display an example of using Llama 3 to translate Python code (left) to PHP code (right) to augment our SFT dataset with a wider range of programming languages.

图8 代码翻译示例。我们展示了一个使用Llama 3将Python代码（左侧）翻译成PHP代码（右侧）的例子，以扩充我们的SFT数据集，涵盖更广泛的编程语言。

Figure 9 (c) Improving generated code quality with system prompts. \({Left}\) : without system prompt \({Right}\) : with system prompt.

图9 (c) 通过系统提示改进生成代码的质量。\({Left}\)：无系统提示 \({Right}\)：有系统提示。

dialogs related to code explanation, generation, documentation, and debugging. Beginning with code snippets from a variety of languages in our pre-training data:

与代码解释、生成、文档和调试相关的对话。从我们的预训练数据中多种语言的代码片段开始：

• Generate: We prompt Llama 3 to generate data that represents our target capability (e.g., we add comments and docstrings for the code snippet, or we ask the model to explain a piece of code).

• 生成：我们提示Llama 3生成代表我们目标能力的数据（例如，我们为代码片段添加注释和文档字符串，或者我们要求模型解释一段代码）。

• Backtranslate: We then prompt the model to "backtranslate" the synthetically generated data to the original code (e.g., we prompt the model to generate code only from its documentation, or we ask the model to generate code only from its explanation).

• 回译：然后我们提示模型进行“回译”，将合成生成的数据还原为原始代码（例如，我们提示模型仅从其文档生成代码，或者我们要求模型仅从其解释生成代码）。

• Filter: Using the original code as a reference, we prompt the Llama 3 to determine the quality of the output (e.g., we ask the model how faithful the backtranslated code is to the original). We then use the generated examples that have the highest self-verification scores in SFT.

• 过滤器：以原始代码为参考，我们引导 Llama 3 评估输出质量（例如，我们询问模型回译代码对原始代码的忠实度）。然后，我们使用在 SFT 中自验证得分最高的生成示例。

System prompt steering during rejection sampling. During the rejection sampling process, we used code specific system prompts to improve code readability, documentation, thoroughness, and specificity. Recall, from Section 7 this data is used to finetune the language model. Figure 9 shows an example of how the system prompt helps improve the generated code quality - it adds necessary comments, uses more informative variable names, saves memory, etc.

拒绝采样期间的系统提示引导。在拒绝采样过程中，我们使用特定于代码的系统提示来提高代码的可读性、文档化、全面性和具体性。回顾第7节，这些数据用于微调语言模型。图9展示了一个例子，说明系统提示如何帮助提高生成代码的质量——它添加了必要的注释，使用了更具信息量的变量名，节省了内存等。

Filtering training data with execution and model-as-judge signals. As described in Section 4.2.3, we occasionally encounter quality issues in our rejection-sampled data, such as code blocks containing bugs. Detecting these issues in our rejection-sampled data is not as straightforward as it is for our synthetic code data, as the rejection-sampled responses typically contain a mix of natural language and code for which the code may not always be expected to be executable. (For example, user prompts may explicitly ask for pseudo-code or edits to only a very small snippet of an executable program.) To address this, we utilize the "model-as-judge" approach, where earlier versions of Llama 3 assess and assign a binary \(\left( {0/1}\right)\) score based on two criteria: code correctness and code style. We retain only those samples that achieve a perfect score of 2. Initially, this stringent filtering led to a regression in downstream benchmark performance, primarily because it disproportionately removed examples with challenging prompts. To counteract this, we strategically revise the responses of some coding data categorized as most challenging until they met the Llama-based "model-as-judge" criteria. By refining these challenging problems, the coding data achieves a balance between quality and difficulty, resulting in optimal downstream performance.

使用执行和模型即判断信号过滤训练数据。如第4.2.3节所述，我们在拒绝采样数据中偶尔会遇到质量问题，例如包含错误的代码块。在我们的拒绝采样数据中检测这些问题不像合成代码数据那样直接，因为拒绝采样的响应通常包含自然语言和代码的混合，其中代码不一定总是可执行的。（例如，用户提示可能明确要求伪代码或仅对可执行程序的一小部分进行编辑。）为了解决这个问题，我们采用了“模型即判断”方法，其中早期的Llama 3版本根据两个标准：代码正确性和代码风格，评估并赋予一个二进制\(\left( {0/1}\right)\)分数。我们仅保留那些获得满分2分的样本。最初，这种严格的过滤导致下游基准性能下降，主要是因为它不均衡地移除了具有挑战性提示的示例。为了抵消这一点，我们战略性地修改了一些被归类为最具挑战性的编码数据的响应，直到它们符合基于Llama的“模型即判断”标准。通过改进这些具有挑战性的问题，编码数据在质量和难度之间达到了平衡，从而实现了最佳的下游性能。

4.3.2 Multilinguality 多语言性

We describe how we improve Llama 3's multilingual capabilities, including training an expert specialized on substantially more multilingual data, sourcing and generating high quality multilingual instruction tuning data for German, French, Italian, Portuguese, Hindi, Spanish, and Thai, and tackling specific challenges of multilingual language steering to enhance the overall performance of our model.

我们描述了如何提升Llama 3的多语言能力，包括训练一个专注于更多多语言数据的专家，为德语、法语、意大利语、葡萄牙语、印地语、西班牙语和泰语获取和生成高质量的多语言指令调整数据，以及解决多语言语言引导的具体挑战，以提高我们模型的整体性能。

Expert training. Our Llama 3 pre-training data mix contains significantly more English tokens than non-English tokens. To collect higher quality human annotations in non-English languages, we train a multilingual expert by branching off the pre-training run and continuing to pre-train on a data mix that consists of \({90}\%\) multilingual tokens. We then perform post-training on this expert following Section 4.1. This expert model is then used to collect higher quality annotations in non-English languages until pre-training was fully complete.

专家培训。我们的 Llama 3 预训练数据混合中包含的英语标记明显多于非英语标记。为了在非英语语言中收集更高质量的人工标注，我们通过分支出预训练运行并继续在由 \({90}\%\) 多语言标记组成的数据混合上进行预训练，来训练一个多语言专家。然后，我们按照第 4.1 节对该专家进行后训练。这个专家模型随后被用于在预训练完全完成之前收集非英语语言中更高质量的标注。

Multilingual data collection. Our multilingual SFT data is derived primarily from sources described below. The overall distribution is \({2.4}\%\) human annotations, \({44.2}\%\) data from other NLP tasks, \({18.8}\%\) rejection sampled data,and \({34.6}\%\) translated reasoning data.

多语言数据收集。我们的多语言 SFT 数据主要来源于以下描述的来源。总体分布为 \({2.4}\%\) 人工标注，\({44.2}\%\) 来自其他 NLP 任务的数据，\({18.8}\%\) 拒绝采样数据，以及 \({34.6}\%\) 翻译推理数据。

• Human annotations: We collect high-quality, manually annotated data from linguists and native speakers. These annotations mostly consist of open-ended prompts that represent real world use cases.

• 人工标注：我们从语言学家和母语使用者那里收集高质量的手工标注数据。这些标注主要由代表真实世界使用场景的开放式提示组成。

• Data from other NLP tasks: To further augment, we use multilingual training data from other tasks and rewrite into dialog format. For example, we use data from exams-qa (Hardalov et al., 2020) and Conic10k (Wu et al., 2023). To improve language alignment, we also use parallel texts from GlobalVoices (Prokopidis et al., 2016) and Wikimedia (Tiedemann, 2012). We use LID based filtering and Blaser2.0 (Seamless Communication et al., 2023) to remove low quality data. For parallel text data, instead of using the bitext pairs directly, we apply a multilingual template inspired by Wei et al. (2022a) to better simulate real-life conversations in translation and language learning scenarios.

• 来自其他 NLP 任务的数据：为了进一步增强，我们使用来自其他任务的多语言训练数据并重写为对话格式。例如，我们使用来自 exams-qa（Hardalov 等人，2020）和 Conic10k（Wu 等人，2023）的数据。为了改善语言对齐，我们还使用来自 GlobalVoices（Prokopidis 等人，2016）和 Wikimedia（Tiedemann，2012）的平行文本。我们使用基于 LID 的过滤和 Blaser2.0（Seamless Communication 等人，2023）来去除低质量数据。对于平行文本数据，我们不是直接使用双语对，而是应用受 Wei 等人（2022a）启发的多语言模板，以更好地模拟翻译和语言学习场景中的真实对话。

• Rejection sampled data: We apply rejection sampling on our human annotated prompts to generate high-quality samples for finetuning, with few modifications compared to the process for English data:

• 拒绝采样数据：我们在人工标注的提示上应用拒绝采样，以生成高质量的样本用于微调，与英语数据的处理过程相比，改动较少：

• Generation: We explored randomly choosing the temperature hyperparameter from the range \({0.2} - 1\) for diverse generations in early rounds of post-training. With high temperature,responses for multilingual prompts can get creative and inspiring, but are also susceptible to unnecessary or unnatural code-switching. In the final round of post-training, we use a constant value of 0.6 to balance the trade-off. Additionally, we used specialized system prompts to improve response format, structure and general readability.

• 生成：我们在后训练的早期轮次中探索从范围 \({0.2} - 1\) 随机选择温度超参数以实现多样化的生成。在高温度下，多语言提示的响应可以变得富有创意和启发性，但也容易出现不必要的或不自然的代码切换。在后训练的最后一轮，我们使用恒定值 0.6 来平衡这种权衡。此外，我们使用了专门的系统提示来改进响应格式、结构和整体可读性。

• Selection: Prior to reward model based selection, we implement multilingual-specific checks to ensure high language-match rate between the prompt and response (e.g., a romanized Hindi prompt should not expect a response in Hindi Devanagari script).

• 选择：在基于奖励模型的选择之前，我们实施了多语言特定的检查，以确保提示和响应之间的语言匹配率高（例如，罗马化的印地语提示不应期望以印地语梵文脚本形式的响应）。

• Translated data: We try to avoid using machine-translated data to finetune the model in order to prevent translationese (Bizzoni et al., 2020; Muennighoff et al., 2023) or possible name bias (Wang et al., 2022a), gender bias (Savoldi et al., 2021), or cultural bias (Ji et al., 2023). Moreover, we aim to prevent the model from being exposed only to tasks that are rooted in English cultural context, which may not be representative of the linguistic and cultural diversity we aim to capture. We made one exception to this and translated our synthetic quantitative reasoning data (see Section 4.3.3 for details) to improve performance in quantitative reasoning in non-English languages. Due to the simple nature of

• 翻译数据：我们尽量避免使用机器翻译的数据来微调模型，以防止翻译体（Bizzoni 等人，2020；Muennighoff 等人，2023）或可能的名称偏见（Wang 等人，2022a）、性别偏见（Savoldi 等人，2021）或文化偏见（Ji 等人，2023）。此外，我们的目标是防止模型仅接触根植于英语文化背景的任务，这可能无法代表我们旨在捕捉的语言和文化多样性。我们对此做了一个例外，将我们的合成定量推理数据（详见第 4.3.3 节）翻译成其他语言，以提高非英语语言的定量推理性能。由于这些数学问题中语言的简单性，翻译样本被发现几乎没有质量问题。我们观察到在 MGSM（Shi 等人，2022）上添加这些翻译数据后取得了显著的提升。

the language in these math problems, the translated samples were found to have little to no quality issues. We observed strong gains on MGSM (Shi et al., 2022) from adding this translated data.

这些数学问题中的语言简单，翻译样本几乎没有质量问题。我们观察到在 MGSM（Shi 等人，2022）上添加这些翻译数据后取得了显著的提升。

4.3.3 Math and Reasoning 数学与推理

We define reasoning as the ability to perform multi-step computations and arrive at the correct final answer. Several challenges guide our approach to training models that excel in mathematical reasoning:

我们将推理定义为执行多步骤计算并得出正确最终答案的能力。在训练擅长数学推理的模型时，我们面临以下几个挑战：

• Lack of prompts: As the complexity of questions increases, the number of valid prompts or questions for Supervised Fine-Tuning (SFT) decreases. This scarcity makes it difficult to create diverse and representative training datasets for teaching models various mathematical skills (Yu et al., 2023; Yue et al., 2023; Luo et al., 2023; Mitra et al., 2024; Shao et al., 2024; Yue et al., 2024b).

• 提示缺失：随着问题复杂性的增加，适用于监督微调（SFT）的有效提示或问题的数量减少。这种稀缺性使得难以创建多样化和具有代表性的训练数据集，以教授模型各种数学技能（Yu et al., 2023; Yue et al., 2023; Luo et al., 2023; Mitra et al., 2024; Shao et al., 2024; Yue et al., 2024b）。

• Lack of ground truth chain of thought: Effective reasoning requires a step-by-step solution to facilitate the reasoning process (Wei et al., 2022c). However, there is often a shortage of ground truth chains of thought, which are essential for guiding the model how to break down the problem step-by-step and reach the final answer (Zelikman et al., 2022).

• 缺乏真实思维链：有效的推理需要逐步解决方案来促进推理过程（Wei et al., 2022c）。然而，通常缺乏真实的思维链，这些思维链对于指导模型如何逐步分解问题并达到最终答案至关重要（Zelikman et al., 2022）。

• Incorrect intermediate steps: When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a). This inaccuracy can lead to incorrect final answers and needs to be addressed.

• 中间步骤错误：在使用模型生成的思维链时，中间步骤可能并不总是正确的（Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a）。这种不准确性可能导致最终答案错误，需要解决。

• Teaching models to use external tools: Enhancing models to utilize external tools, such as code interpreters, allows them to reason by interleaving code and text (Gao et al., 2023; Chen et al., 2022; Gou et al., 2023). This capability can significantly improve their problem-solving abilities.

• 教授模型使用外部工具：增强模型利用外部工具（如代码解释器）的能力，使它们能够通过交错代码和文本来进行推理（Gao et al., 2023; Chen et al., 2022; Gou et al., 2023）。这种能力可以显著提高它们的问题解决能力。

• Discrepancy between training and inference: There is often a discrepancy between how the model is finetuned during training and how it is used during inference. During inference, the finetuned model may interact with humans or other models, requiring it to improve its reasoning using feedback. Ensuring consistency between training and real-world usage is crucial for maintaining reasoning performance.

• 训练与推理之间的差异：模型在训练期间的微调方式与推理期间的使用方式之间常常存在差异。在推理过程中，微调后的模型可能与人类或其他模型交互，需要利用反馈来改进其推理能力。确保训练与实际应用之间的一致性对于保持推理性能至关重要。

To address these challenges, we apply the following methodologies:

为了应对这些挑战，我们采用了以下方法：

• Addressing the lack of prompts: We source relevant pre-training data from mathematical contexts and converted it into a question-answer format which can then be used for supervised finetuning. Additionally, we identify mathematical skills where the model under-performs and actively sourced prompts from humans to teach models such skills. To facilitate this process, we create a taxonomy of mathematical skills (Didolkar et al., 2024) and ask humans to provide relevant prompts/questions accordingly.

• 解决提示缺失问题：我们从数学上下文中获取相关的预训练数据，并将其转换为问答格式，以便用于监督式微调。此外，我们识别模型表现不佳的数学技能，并主动从人类那里获取提示来教授模型这些技能。为了促进这一过程，我们创建了一个数学技能分类体系（Didolkar et al., 2024），并请人类根据此体系提供相关的提示/问题。

• Augmenting training data with step-wise reasoning traces: We use Llama \(3\) to generate step-by-step solutions for a set of prompts. For each prompt, the model produces a variable number of generations. These generations are then filtered based on the correct answer (Li et al., 2024a). We also do self-verification where Llama 3 is used to verify whether a particular step-by-step solution is valid for a given question. This process improves the quality of the finetuning data by eliminating instances where the model does not produce valid reasoning traces.

• 通过逐步推理追踪增强训练数据：我们使用 Llama \(3\) 为一组提示生成逐步解决方案。对于每个提示，模型生成数量可变的生成结果。然后根据正确答案（Li et al., 2024a）对这些生成结果进行筛选。我们还进行自我验证，使用 Llama 3 来验证某个逐步解决方案是否适用于给定问题。这一过程通过消除模型未产生有效推理追踪的实例来提高微调数据的质量。

• Filteringincorrect reasoning traces: We train outcome and stepwise reward models (Lightman et al., 2023; Wang et al., 2023a) to filter training data where the intermediate reasoning steps were incorrect. These reward models are used to eliminate data with invalid step-by-step reasoning, ensuring high-quality data for finetuning. For more challenging prompts, we use Monte Carlo Tree Search (MCTS) with learned step-wise reward models to generate valid reasoning traces, further enhancing the collection of high-quality reasoning data (Xie et al., 2024).

• 过滤不正确的推理轨迹：我们训练结果和逐步奖励模型（Lightman et al., 2023; Wang et al., 2023a）来过滤中间推理步骤不正确的训练数据。这些奖励模型用于消除无效的逐步推理数据，确保用于微调的高质量数据。对于更具挑战性的提示，我们使用蒙特卡洛树搜索（MCTS）与学习的逐步奖励模型来生成有效的推理轨迹，进一步增强高质量推理数据的收集（Xie et al., 2024）。

• Interleaving code and text reasoning: We prompt Llama 3 to solve reasoning problems through a combination of textual reasoning and associated Python code (Gou et al., 2023). Code execution is used as a feedback signal to eliminate cases where the reasoning chain was not valid, ensuring the correctness of the reasoning process.

• 交错代码和文本推理：我们提示Llama 3通过文本推理和相关的Python代码（Gou et al., 2023）来解决推理问题。代码执行用作反馈信号，以消除推理链无效的情况，确保推理过程的正确性。

• Learning from feedback and mistakes: To simulate human feedback, we utilize incorrect generations (i.e., generations leading to incorrect reasoning traces) and perform error correction by prompting Llama 3 to

• 从反馈和错误中学习：为了模拟人类反馈，我们利用不正确的生成（即导致不正确推理轨迹的生成）并通过提示Llama 3进行错误纠正

yield correct generations (An et al., 2023b; Welleck et al., 2022; Madaan et al., 2024a). The iterative process of using feedback from incorrect attempts and correcting them helps improve the model's ability to reason accurately and learn from its mistakes.

产生正确的生成（An et al., 2023b; Welleck et al., 2022; Madaan et al., 2024a）。利用不正确尝试的反馈并纠正它们的迭代过程有助于提高模型准确推理和从错误中学习的能力。

4.3.4 Long Context 长上下文

During the final pre-training stage, we extend the context length of Llama 3 from 8K tokens to 128K tokens (see Section 3.4 for more details). Similar to pre-training, we find that during finetuning we must carefully tune the recipe to balance short and long-context capabilities.

在最终的预训练阶段，我们将Llama 3的上下文长度从8K令牌扩展到128K令牌（更多细节见第3.4节）。与预训练类似，我们发现在微调过程中，我们必须仔细调整配方以平衡短上下文和长上下文能力。

SFT and synthetic data generation. Naively applying our existing SFT recipe with only short-context data resulted in significant regressions in long-context capabilities from pre-training, highlighting the need to incorporate long-context data in our SFT data mix. In practice, however, it is largely impractical to get humans to annotate such examples due to the tedious and time-consuming nature of reading lengthy contexts, so we predominantly rely on synthetic data to fill this gap. We use earlier versions of Llama 3 to generate synthetic data based on the key long-context use-cases: (possibly multi-turn) question-answering, summarization for long documents, and reasoning over code repositories, and describe them in greater detail below.

SFT 和合成数据生成。简单地应用我们现有的 SFT 配方，仅使用短上下文数据，导致从预训练中长上下文能力显著退化，这凸显了在我们的 SFT 数据混合中加入长上下文数据的必要性。然而，在实践中，由于阅读长上下文的繁琐和耗时性质，让人类来标注这些例子基本上是不切实际的，因此我们主要依赖合成数据来填补这一空白。我们使用早期的 Llama 3 版本基于关键的长上下文用例生成合成数据：（可能是多轮的）问答、长文档的摘要和代码仓库的推理，并在下面详细描述它们。

• Question answering: We carefully curate a set of long documents from our pre-training mix. We split these documents into chunks of \(8\mathrm{\;K}\) tokens,and prompted an earlier version of the Llama 3 model to generate QA pairs conditional on randomly selected chunks. During training, the whole document is used as context.

• 问答：我们从预训练混合数据中精心挑选了一组长文档。我们将这些文档分割成 \(8\mathrm{\;K}\) 个令牌的块，并提示早期版本的 Llama 3 模型根据随机选择的块生成 QA 对。在训练过程中，整个文档被用作上下文。

• Summarization: We applied hierarchical summarization of long-context documents by first summarizing the chunks of \(8\mathrm{\;K}\) input length using our strongest Llama \({38}\mathrm{\;K}\) context model and then summarizing the summaries. During training we provide the full document and prompt the model to summarize the document while preserving all the important details. We also generate QA pairs based on the summaries of the documents and prompt the model with questions that require global understanding of the whole long document.

• 摘要：我们通过首先使用我们最强的 Llama \({38}\mathrm{\;K}\) 上下文模型对 \(8\mathrm{\;K}\) 输入长度的块进行摘要，然后对摘要进行摘要，应用了长上下文文档的分层摘要。在训练过程中，我们提供完整的文档并提示模型在保留所有重要细节的同时对文档进行摘要。我们还基于文档的摘要生成 QA 对，并提示模型回答需要对整个长文档进行全局理解的问题。

• Long context code reasoning: We parse Python files to identify import statements and determine their dependencies. From here, we select the most commonly depended-upon files, specifically those referenced by at least five other files. We remove one of these key files from a repository and prompt the model to identify which files depended on the missing file and to generate the necessary missing code.

• 长上下文代码推理：我们解析 Python 文件以识别导入语句并确定它们的依赖关系。从这里，我们选择最常被依赖的文件，特别是那些至少被其他五个文件引用的文件。我们从仓库中移除其中一个关键文件，并提示模型识别哪些文件依赖于缺失的文件并生成必要的缺失代码。

We further categorize these synthetically generated samples based on the sequence length \(({16}\mathrm{\;K},{32}\mathrm{\;K},{64}\mathrm{\;K}\) and \({128}\mathrm{\;K}\) ) to enable more fine-grained targeting of input lengths.

我们进一步根据序列长度 \(({16}\mathrm{\;K},{32}\mathrm{\;K},{64}\mathrm{\;K}\) 和 \({128}\mathrm{\;K}\) 对这些综合生成的样本进行分类，以实现对输入长度的更精细定位。

Through careful ablations,we observe that mixing \({0.1}\%\) of synthetically generated long-context data with the original short-context data optimizes the performance across both short-context and long-context benchmarks.

通过仔细的消融实验，我们观察到将综合生成的长上下文数据 \({0.1}\%\) 与原始短上下文数据混合，可以优化短上下文和长上下文基准的性能。

DPO. We observe that using only short context training data in DPO did not negatively impact long-context performance as long as the SFT model is high quality in long context tasks. We suspect this is due to the fact that our DPO recipe has fewer optimizer steps than SFT. Given this finding, we keep the standard short-context recipe for DPO on top of our long-context SFT checkpoints.

DPO。我们观察到，只要SFT模型在长上下文任务中质量高，仅使用短上下文训练数据在DPO中并不会对长上下文性能产生负面影响。我们怀疑这是因为我们的DPO配方比SFT的优化步骤少。鉴于这一发现，我们在长上下文SFT检查点的基础上保持标准的短上下文配方用于DPO。

4.3.5 Tool Use 工具使用

Teaching LLMs to use tools such as search engines or code interpreters hugely expands the range of tasks they can solve, transforming them from pure chat models into more general assistants (Nakano et al., 2021; Thoppilan et al., 2022; Parisi et al., 2022; Gao et al., 2023; Mialon et al., 2023a; Schick et al., 2024). We train Llama 3 to interact with the following tools:

教授LLM使用搜索引擎或代码解释器等工具，极大地扩展了它们能解决的任务范围，使它们从纯粹的聊天模型转变为更通用的助手（Nakano et al., 2021; Thoppilan et al., 2022; Parisi et al., 2022; Gao et al., 2023; Mialon et al., 2023a; Schick et al., 2024）。我们训练Llama 3与以下工具进行交互：

• Search engine. Llama 3 is trained to use Brave Search \({}^{7}\) to answer questions about recent events that go beyond its knowledge cutoff or that require retrieving a particular piece of information from the web.

• 搜索引擎。Llama 3被训练使用Brave Search \({}^{7}\) 来回答关于近期事件的问题，这些问题超出了其知识截止点，或者需要从网络上检索特定信息。

• Python interpreter. Llama 3 can generate and execute code to perform complex computations, read files uploaded by the user and solve tasks based on them such as question answering, summarization, data analysis or visualization.

• Python解释器。Llama 3能够生成并执行代码，进行复杂计算，读取用户上传的文件，并基于这些文件解决任务，如问答、总结、数据分析或可视化。

\({}^{7}\) https://brave.com/search/api/

• Mathematical computational engine. Llama 3 can use the Wolfram Alpha API \({}^{8}\) to more accurately solve math, science problems, or retrieve accurate information from Wolfram's database.

数学计算引擎。Llama 3 可以使用 Wolfram Alpha API \({}^{8}\) 更准确地解决数学、科学问题，或从 Wolfram 的数据库中检索准确信息。

The resulting model is able to use these tools in a chat setup to solve the user's queries, including in multi-turn dialogs. If a query requires multiple tool calls, the model can write a step-by-step plan, call the tools in sequence, and do reasoning after each tool call.

生成的模型能够在聊天环境中使用这些工具来解决用户的查询，包括在多轮对话中。如果一个查询需要多次工具调用，模型可以编写一个逐步计划，按顺序调用工具，并在每次工具调用后进行推理。

We also improve Llama 3's zero-shot tool use capabilities - given in-context, potentially unseen tool definitions and a user query, we train the model to generate the correct tool call.

我们还改进了 Llama 3 的零样本工具使用能力——在给定上下文中，潜在未见过的工具定义和用户查询的情况下，我们训练模型生成正确的工具调用。

Implementation. We implement our core tools as Python objects with different methods. Zero-shot tools can be implemented as Python functions with descriptions, documentation (i.e., examples for how to use them), and the model only needs the function's signature and docstring as context to generate the appropriate call. We also convert function definitions and calls to JSON format, e.g., for web API calls. All tool calls are executed by the Python interpreter, that must be enabled in the Llama 3 system prompt. Core tools can be individually enabled or disabled in the system prompt.

实现。我们将核心工具实现为具有不同方法的 Python 对象。零样本工具可以实现为带有描述、文档（即使用示例）的 Python 函数，模型只需要函数的签名和文档字符串作为上下文来生成适当的调用。我们还把函数定义和调用转换为 JSON 格式，例如，用于网络 API 调用。所有工具调用都由 Python 解释器执行，必须在 Llama 3 系统提示中启用。核心工具可以在系统提示中单独启用或禁用。

Data collection. Different from Schick et al. (2024), we rely on human annotations and preferences to teach Llama 3 to use tools. There are two main differences with the post-training pipeline generally used in Llama 3:

数据收集。与 Schick 等人 (2024) 不同，我们依赖人工注释和偏好来教导 Llama 3 使用工具。这与 Llama 3 中通常使用的后训练流程有两个主要区别：

• For tools, dialogs often contain more than a single assistant message (e.g., calling the tool and reasoning about the tool output). Thus, we annotate at the message level to collect granular feedback: annotators provide a preference between two assistant messages with the same context or, if both contain major problems, edit one of the messages. The chosen or edited message is then added to the context and the dialog continues. This provides human feedback for both the assistant's ability of calling the tools and reasoning about the tool outputs. Annotators cannot rank or edit the tool outputs.

• 对于工具，对话通常包含不止一条助手消息（例如，调用工具并对工具输出进行推理）。因此，我们在消息级别进行标注以收集细粒度的反馈：标注者在一个相同上下文中的两条助手消息之间提供偏好选择，或者如果两者都存在重大问题，则编辑其中一条消息。被选定或编辑的消息随后被添加到上下文中，对话继续进行。这为助手调用工具和推理工具输出的能力提供了人类反馈。标注者不能对工具输出进行排序或编辑。

• We do not perform rejection sampling, as we did not observe gains in our tool benchmarks.

• 我们不进行拒绝采样，因为我们未在我们的工具基准测试中观察到收益。

To accelerate the annotation process, we start by bootstrapping basic tool use capabilities by finetuning on synthetically generated data from previous Llama 3 checkpoints. Thus, annotators have fewer edits to perform. In a similar spirit, as Llama 3 gradually improves through its development, we progressively complexify our human annotation protocols: we start by single-turn tool use annotations, before moving to tool use in dialogs, and finally annotating for multi-step tool use and data analysis.

为了加速标注过程，我们首先通过在从前Llama 3检查点合成生成的数据上进行微调来引导基本工具使用能力。因此，标注者需要进行的编辑较少。同样地，随着Llama 3通过其开发逐步改进，我们逐步复杂化我们的人类标注协议：我们首先从单轮工具使用标注开始，然后过渡到对话中的工具使用，最后标注多步骤工具使用和数据分析。

Tool datasets. To create data for tool usage applications, we leverage the following procedure:

工具数据集。为了创建工具使用应用的数据，我们采用以下步骤：

• Single-step tool use: We start by few-shot generation of synthetic user prompts which, by construction, require a call to one of our core tools (for example, questions that exceed our knowledge cutoff date). Then, still relying on few-shot generation, we generate appropriate tool calls for these prompts, execute them, and add the output to the model's context. Finally, we prompt the model again to generate a final answer to the user's query based on the tool output. We end up with trajectories of the following form: system prompt,user prompt,tool call,tool output,final answer. We also filter around \({30}\%\) this dataset to remove tool calls that cannot be executed or other formatting issues.

• 单步骤工具使用：我们首先通过少样本生成合成用户提示，这些提示按设计需要调用我们的核心工具之一（例如，超出我们知识截止日期的问题）。然后，仍然依赖少样本生成，我们为这些提示生成适当的工具调用，执行它们，并将输出添加到模型的上下文中。最后，我们再次提示模型，根据工具输出生成对用户查询的最终答案。我们最终得到以下形式的轨迹：系统提示、用户提示、工具调用、工具输出、最终答案。我们还会过滤掉无法执行的工具调用或其他格式问题。

• Multi-step tool use: We follow a similar protocol and first generate synthetic data to teach the model basic multi-step tool use capabilities. To do this, we first prompt Llama 3 to generate user prompts that require at least two tool calls, that can be the same or different tools from our core set. Then, conditioned on these prompts, we few-shot prompt Llama 3 to generate a solution consisting of interleaved reasoning steps and tool calls, similar to ReAct (Yao et al., 2022). See Figure 10 for an example of Llama 3 performing a task involving multi-step tool usage.

• 多步骤工具使用：我们遵循类似的协议，首先生成合成数据来教授模型基本的多步骤工具使用能力。为此，我们首先提示 Llama 3 生成需要至少两次工具调用的用户提示，这些工具可以是我们核心集合中的相同或不同工具。然后，基于这些提示，我们通过少量示例提示 Llama 3 生成一个解决方案，该解决方案包含交错的推理步骤和工具调用，类似于 ReAct（Yao 等人，2022）。参见图 10，Llama 3 执行涉及多步骤工具使用的任务的示例。

• File uploads: We annotate for the following filetypes: .TXT, .DOCX, .PDF, .PPTX, .XLSX, .CSV, .TSV, PY, JSON, JSONL, HTML, XML. Our prompts are based on a provided file, and ask to summarize the contents of the file, find and fix bugs, optimize a piece of code, perform data analysis or visualization. See Figure 11 for an example of Llama 3 performing a task involving a file upload.

• 文件上传：我们为以下文件类型进行标注：.TXT、.DOCX、.PDF、.PPTX、.XLSX、.CSV、.TSV、PY、JSON、JSONL、HTML、XML。我们的提示基于提供的文件，要求总结文件内容、查找并修复错误、优化一段代码、执行数据分析或可视化。参见图 11，Llama 3 执行涉及文件上传的任务的示例。

After finetuning on this synthetic data, we gather human annotations in diverse and challenging scenarios including multi-turn interactions, more than three step tool use, and instances where a tool call does not yield a satisfying answer. We augment our synthetic data with different system prompts to teach the model to use tools only when activated. To train the model to avoid calling tools for simple queries, we also add queries from easy math or question answering datasets (Berant et al., 2013; Koncel-Kedziorski et al., 2016; Joshi et al., 2017; Amini et al., 2019) and their responses without tools, but with tools activated in system prompt.

在对此合成数据进行微调后，我们在多样化和具有挑战性的场景中收集人工标注，包括多轮交互、超过三个步骤的工具使用，以及工具调用未产生结果的情况。一个令人满意的答案。我们通过不同的系统提示来增强我们的合成数据，以教导模型仅在激活时使用工具。为了训练模型避免对简单查询调用工具，我们还添加了来自简单数学或问答数据集（Berant et al., 2013; Koncel-Kedziorski et al., 2016; Joshi et al., 2017; Amini et al., 2019）的查询及其不使用工具的响应，但在系统提示中激活了工具。

\({}^{8}\) https://products.wolframalpha.com/llm-api/documentation

Figure 10 Multi-step tool usage. Example of Llama 3 performing multi-step planning, reasoning, and tool calling to solve a task.
图10 多步骤工具使用。Llama 3执行多步骤规划、推理和工具调用的示例，以解决任务。

Zero-shot tool use data. We improve Llama 3 zero-shot tool use abilities (also referred to as function calling) by finetuning on a large and diverse set of partly synthetic (functions definitions, user query, corresponding call) tuples. We evaluate our model on a set of unseen tools.
零样本工具使用数据。我们通过在一个大型且多样化的部分合成（函数定义、用户查询、相应调用）元组上进行微调，来提高Llama 3的零样本工具使用能力（也称为函数调用）。我们在一组未见过的工具上评估我们的模型。

• Single, nested, and parallel function calling: Calls can be simple, nested, i.e. we pass a function call as an argument of another function, or parallel, i.e. the model returns a list of independent function calls. Generating a diverse set of functions, queries and ground truths can be challenging (Mekala et al., 2024), and we resort to mining the Stack (Kocetkov et al., 2022) to ground our synthetic user queries in real functions. More precisely, we extract function calls and their definitions, clean and filter them, e.g. for missing docstrings or non-executable functions, and use Llama 3 to generate a natural language query corresponding to the function call.

• 单一、嵌套和并行函数调用：调用可以是简单的、嵌套的，即我们将一个函数调用作为另一个函数的参数，或者是并行的，即模型返回一组独立的函数调用。生成多样化的函数、查询和基本事实可能具有挑战性（Mekala 等人，2024），我们求助于挖掘 Stack（Kocetkov 等人，2022）以使我们的合成用户查询基于真实函数。更确切地说，我们提取函数调用及其定义，清理和过滤它们，例如缺失文档字符串或不可执行的函数，并使用 Llama 3 生成与函数调用相对应的自然语言查询。

• Multi-turn function calling: We also generate synthetic data for multi-turn dialogs with function calls, following a protocol similar to the one proposed in Li et al. (2023b). We use multiple agents that generate domains, APIs, user queries, API calls, and responses, while also ensuring that the generated data covers a set of diverse domains and realistic APIs. All agents are variants of Llama 3 prompted in different ways depending on their roles and collaborate in a step-by-step manner.

• 多轮函数调用：我们还为带有函数调用的多轮对话生成合成数据，遵循类似于 Li 等人（2023b）提出的协议。我们使用多个代理生成领域、API、用户查询、API 调用和响应，同时还确保生成的数据涵盖一组多样化的领域和真实的 API。所有代理都是 Llama 3 的变体，根据其角色以不同方式提示，并以逐步方式协作。

4.3.6 Factuality 事实性

Hallucinations remain a major challenge for large language models. Models tend to be overconfident, even in domains where they have little knowledge. Despite these shortcomings, they are often used as knowledge bases, which can lead to risky outcomes such as the spread of misinformation. While we recognize that factuality can go beyond hallucinations, we took a hallucination-first approach here.
幻觉仍然是大型语言模型的主要挑战。模型往往过于自信，即使在它们知识有限的领域也是如此。尽管存在这些缺点，它们经常被用作知识库，这可能导致风险结果，如错误信息的传播。虽然我们认识到事实性可以超越幻觉，但我们在这里采取了以幻觉为先的方法。

Figure 11 Processing file uploads. Example of Llama 3 performing analysis and visualization of an uploaded file.

图 11 处理文件上传。Llama 3 执行分析和可视化上传文件的示例。

We follow the principle that post-training should align the model to "know what it knows" rather than add knowledge (Gekhman et al., 2024; Mielke et al., 2020). Our primary approach involves generating data that aligns model generations with subsets of factual data present in the pre-training data. To achieve this, we develop a knowledge probing technique that takes advantage of Llama 3's in-context abilities. This data generation process involves the following procedure:

我们遵循的原则是，训练后应使模型“知道自己知道什么”，而不是增加知识（Gekhman 等人，2024；Mielke 等人，2020）。我们的主要方法涉及生成与预训练数据中存在的部分事实数据相一致的数据。为此，我们开发了一种知识探查技术，利用 Llama 3 的上下文能力。此数据生成过程包括以下步骤：

1. Extract a data snippet from the pre-training data.
2. Generate a factual question about these snippets (context) by prompting Llama 3.
3. Sample responses from Llama 3 to the question.
4. Score the correctness of the generations using the original context as a reference and Llama 3 as a judge.
5. Score the informativeness of the generations using Llama 3 as a judge.
6. Generate a refusal for responses which are consistently informative and incorrect across the generations, using Llama 3.

1. 从预训练数据中提取数据片段。
2. 通过提示 Llama 3 生成关于这些片段（上下文）的事实问题。
3. 从 Llama 3 对问题的回答中抽样。
4. 使用原始上下文作为参考和 Llama 3 作为评判，对生成的正确性进行评分。
5. 使用 Llama 3 作为评判，对生成的信息量进行评分。
6. 对于在生成中持续表现出信息丰富但错误的回答，使用 Llama 3 生成拒绝回答。

We use data generated from the knowledge probe to encourage the model to only answer questions which it has knowledge about, and refuse answering those questions that it is unsure about. Further, pre-training data is not always factually consistent or correct. We therefore also collect a limited set of labeled factuality data that deals with sensitive topics where factually contradictory or incorrect statements are prevalent.

我们利用从知识探查中生成的数据，鼓励模型仅回答其了解的问题，并拒绝回答其不确定的问题。此外，预训练数据并不总是事实一致或正确。因此，我们还收集了一组有限的标记事实数据，这些数据涉及敏感话题，其中事实矛盾或不正确的陈述普遍存在。

4.3.7 Steerability 可操控性

Steerability is the ability to direct the model's actions and outcomes to meet developer and user specifications. As Llama 3 is a generic foundational model, it should be maximally steerable to different downstream use cases easily. For Llama 3, we focus on enhancing its steerability through system prompt with natural language instructions, especially around response length, format, tone and character/persona.

可操控性是指能够引导模型的行为和结果以满足开发者和用户的需求。由于 Llama 3 是一个通用的基础模型，它应该能够最大限度地轻松适应不同的下游使用场景。对于 Llama 3，我们通过使用自然语言指令的系统提示来增强其可操控性，特别是在响应长度、格式、语气和角色/人物方面。

Data collection. We collect steerability preference samples within the general English category by asking annotators to design different system prompts for Llama 3. Annotators then engage in conversations with the models to evaluate their consistency in following instructions defined in system prompts over the course of the conversation. We show an example customized system prompt used for enhancing steerability below:

数据收集。我们在英语类别中收集可操控性偏好样本，方法是要求标注者为 Llama 3 设计不同的系统提示。然后，标注者与模型进行对话，评估模型在整个对话过程中遵循系统提示中定义的指令的一致性。我们在下面展示了一个用于增强可操控性的自定义系统提示示例：

You are a helpful and cheerful AI Chatbot that acts as a meal plan assistant for busy families. The family consists of 2 adults, 3 teenagers, and 2 preschoolers. Plan two or three days at a time and use leftovers or extra ingredients for the second day's plan. The user will let you know if they want two or three days. If they don't, assume three days. Each plan should include breakfast, lunch, snack, and dinner. Ask the user if they approve of the plan or need adjustments. After they approve provide a grocery list with family size in mind. Always keep family preferences in mind and if there's something that they don't like provide a substitution. If the user is not feeling inspired then ask them what's the one place they wish they could visit on vacation this week and then suggest meals based on that location’s culture. Weekend meals can be more complex. Weekday meals should be quick and easy. For breakfast and lunch, easy food like cereal, English muffins with pre-cooked bacon, and other quick easy foods are preferred. The family is busy. Be sure to ask if they have essentials and favorites on hand like coffee or energy drinks so they don't forget to buy it. Remember to be budget-conscious unless it's a special occasion.

您是一位乐于助人且开朗的AI聊天机器人，担任忙碌家庭的膳食计划助手。这个家庭包括2名成人、3名青少年和2名学龄前儿童。每次计划两到三天的饮食，并利用剩余食材或额外配料安排第二天的计划。用户会告知他们需要两到三天的计划。如果未指定，则默认三天。每个计划应包括早餐、午餐、小吃和晚餐。询问用户是否批准该计划或需要调整。在他们批准后，提供一份考虑到家庭规模的购物清单。始终牢记家庭偏好，如果有他们不喜欢的食物，请提供替代品。如果用户缺乏灵感，询问他们本周最希望去哪里度假，然后根据该地点的文化推荐餐食。周末餐食可以更复杂。工作日餐食应快速简便。早餐和午餐偏好简单食物，如麦片、英式松饼配预煮培根等快捷简便食品。家庭忙碌，务必询问他们是否有咖啡或能量饮料等必需品和最爱，以免忘记购买。记得要考虑预算，除非是特殊场合。

Modeling. After we collect the preference data, we leverage this data in reward modeling, rejection sampling, SFT, and DPO to enhance Llama 3's steerability.

建模。在我们收集偏好数据后，我们利用这些数据在奖励建模、拒绝采样、SFT和DPO中增强Llama 3的可引导性。

5 Results 结果

We performed an extensive series of evaluations of Llama 3, investigating the performance of: (1) the pre-trained language model, (2) the post-trained language model, and (3) the safety characteristics of Llama 3. We present the results of these evaluations in separate subsections below.

我们对Llama 3进行了一系列广泛的评估，调查了以下方面的性能：（1）预训练语言模型，（2）后训练语言模型，以及（3）Llama 3的安全特性。我们在下面的单独小节中展示了这些评估的结果。

5.1 Pre-trained Language Model 预训练语言模型

In this section, we report evaluation results for our pre-trained Llama 3 (Section 3), comparing with various other models of comparable sizes. We reproduce results of competitor models whenever possible. For non-Llama models, we report the best score across results that are publicly reported or (where possible) that we reproduced ourselves. The specifics of these evaluations, including configurations such as the number of shots, metrics, and other pertinent hyperparameters and settings, can be accessed on our Github repository here. Additionally, we are releasing the data generated as part of evaluations with publicly available benchmarks which can be found on Huggingface here. We evaluate the quality of our models on standard benchmarks (Section 5.1.1), for robustness to changes in multiple-choice question setups (Section 5.1.2), and on adversarial evaluations (Section 5.1.3). We also conduct a contamination analysis to estimate the extent to which our evaluations are impacted by contamination of training data (Section 5.1.4).

在本节中，我们报告了我们预训练的Llama 3（第3节）的评估结果，并与各种其他尺寸相当的模型进行了比较。我们尽可能重现了竞争对手模型的结果。对于非Llama模型，我们报告了公开报告或（如果可能）我们自己重现的结果中的最佳分数。这些评估的具体细节，包括诸如样本数量、指标以及其他相关超参数和设置等配置，可以在我们的Github仓库中访问。此外，我们还发布了作为评估一部分生成的数据，这些数据可以在Huggingface上找到。我们在标准基准（5.1.1节）上评估我们模型的质量，对于多选题设置变化的鲁棒性（5.1.2节），以及对抗性评估（5.1.3节）。我们还进行了污染分析，以估计我们的评估受到训练数据污染影响的程度（5.1.4节）。

5.1.1 Standard Benchmarks 标准基准

To compare our models with the current state-of-the-art, we evaluate Llama 3 on a large number of standard benchmark evaluations shown in Table 8. These evaluations cover eight top-level categories: (1) commonsense reasoning; (2) knowledge; (3) reading comprehension; (4) math, reasoning, and problem solving; (5) long context; (6) code; (7) adversarial evaluations; and (8) aggregate evaluations.

为了将我们的模型与当前最先进的技术进行比较，我们在表8所示的大量标准基准评估上对Llama 3进行了评估。这些评估涵盖了八个顶级类别：（1）常识推理；（2）知识；（3）阅读理解；（4）数学、推理和问题解决；（5）长上下文；（6）代码；（7）对抗性评估；以及（8）综合评估。

Table 8 Pre-training benchmarks by category. Overview of all benchmarks we use to evaluate pre-trained Llama 3 models, grouped by capability category.

表8 按类别划分的预训练基准。我们用于评估预训练Llama 3模型的所有基准概览，按能力类别分组。

Experimental setup. For each benchmark, we compute scores for Llama 3 as well as various other pre-trained models of comparable sizes. Where possible, we recompute numbers with our own pipeline for other models. To ensure a fair comparison, we then select the best score between the score that we computed and the reported number for that model with comparable or more conservative settings. You can find additional details on our evaluation setup here. For some models, it is not possible to (re)compute benchmark values, for instance, because the pre-trained model is not released or because the API does not provide access to log-probabilities. In particular, this is true for all models comparable to Llama 3 405B. Thus, we do not report category averages for Llama 3405B, which requires that all numbers are available for all benchmarks.

实验设置。对于每个基准测试，我们计算 Llama 3 以及其他各种大小相当的预训练模型的分数。在可能的情况下，我们使用自己的流程重新计算其他模型的分数。为了确保公平比较，我们随后在可比较或更保守的设置下，选择我们计算的分数和该模型报告的分数中的最佳值。您可以在此处找到有关我们评估设置的更多详细信息。对于某些模型，无法（重新）计算基准值，例如因为未发布预训练模型或 API 不提供对对数概率的访问。特别是，这对于所有与 Llama 3 405B 相当的模型都是如此。因此，我们不报告 Llama 3405B 的类别平均值，这要求所有基准测试的所有数值都可用。

Significance estimates. Benchmark scores are estimates of a model's true performance. These estimates have variance because benchmark sets are finite samples drawn from some underlying distribution. We follow Madaan et al. (2024b) and report on this variance via \({95}\%\) confidence intervals (CIs),assuming that benchmark scores are Gaussian distributed. While this assumption is incorrect (e.g., benchmark scores are bounded), preliminary bootstrap experiments suggest CIs (for discrete metrics) are a good approximation:

显著性估计。基准分数是对模型真实性能的估计。这些估计具有方差，因为基准集是从某个潜在分布中抽取的有限样本。我们遵循 Madaan 等人（2024b）的方法，通过 \({95}\%\) 置信区间（CIs）报告这一方差，假设基准分数呈高斯分布。虽然这一假设不正确（例如，基准分数是有界的），但初步的 bootstrap 实验表明，对于离散指标，置信区间是一个很好的近似：

\[{CI}\left( S\right) = {1.96} \times \sqrt{\frac{S \times \left( {1 - S}\right) }{N}}.\]

Herein, \(S\) is the observed benchmark score (e.g.,accuracy or EM) and \(N\) the sample size of the benchmark. We omit CIs for benchmark scores that are not simple averages. We note that because subsampling is not the only source of variation, our CI values lower bound the actual variation in the capability estimate.

在此，\(S\) 是观察到的基准分数（例如，准确性或 EM），\(N\) 是基准的样本大小。我们省略了不是简单平均值的基准分数的置信区间。我们注意到，因为子抽样不是变化的唯一来源，我们的置信区间值低估了能力估计中的实际变化。

Results for 8B and 70B models. Figure 12 reports the average performance of Llama 3 8B and 70B on the commonsense reasoning, knowledge, reading comprehension, math and reasoning, and code benchmarks. The results show that Llama 3 8B outperforms competing models in virtually every category, both in terms of per-category win rate and in terms of average per-category performance. We also find that Llama 3 70B outperforms its predecessor Llama 2 70B by a large margin on most benchmarks, with the exception of commonsense benchmarks that are likely saturated. Llama 3 70B also outperforms Mixtral 8x22B.

8B 和 70B 模型的结果。图 12 报告了 Llama 3 8B 和 70B 在常识推理、知识、阅读理解、数学和推理以及代码基准上的平均表现。结果显示，Llama 3 8B 在几乎所有类别中都优于竞争模型，无论是在每个类别的胜率还是在每个类别的平均表现方面。我们还发现，Llama 3 70B 在大多数基准测试中都大幅领先于其前身 Llama 2 70B，除了可能已经饱和的常识基准。Llama 3 70B 也优于 Mixtral 8x22B。

Detailed results for all models. Table 9, 10, 11, 12, 13, and 14 present the benchmark performance of pre-trained Llama 3 8B, 70B, and 405B models on reading comprehension tasks, coding tasks, commonsense understanding tasks, mathematical reasoning tasks, and general tasks. The tables compare Llama 3's performance with that of models of similar size. The results show that Llama 3405B performs competitively with other models in its class. In particular, Llama 3405B substantially outperforms prior open-source models. For long-context, we present more comprehensive results (including probing tasks like needle-in-a-haystack) in Section 5.2.

所有模型的详细结果。表 9、10、11、12、13 和 14 展示了预训练的 Llama 3 8B、70B 和 405B 模型在阅读理解任务、编码任务、常识理解任务、数学推理任务和一般任务上的基准表现。这些表格比较了 Llama 3 与相似尺寸模型的表现。结果显示，Llama 3405B 在其类别中与其他模型竞争激烈。特别是，Llama 3405B 大幅领先于之前的开源模型。对于长上下文，我们在第 5.2 节中提供了更全面的结果（包括诸如“大海捞针”之类的探测任务）。

Figure 12: Performance of pre-trained Llama 3 8B and 70B models on pre-training benchmarks. Results are \(\mathrm{{aggregated}\;{by}}\) capability category by averaging accuracies across all benchmarks corresponding to that category.

图 12：预训练的 Llama 3 8B 和 70B 模型在预训练基准上的表现。结果通过在所有对应于该类别的基准上平均准确率来表示 \(\mathrm{{aggregated}\;{by}}\) 能力类别。

Table 9 Pre-trained model performance on reading comprehension tasks. Results include \({95}\%\) confidence intervals.

表 9 预训练模型在阅读理解任务上的表现。结果包括 \({95}\%\) 置信区间。

Table 10 Pre-trained model performance on coding tasks. Results include \({95}\%\) confidence intervals.

表 10 预训练模型在编码任务上的表现。结果包括 \({95}\%\) 置信区间。

5.1.2 Model Robustness 模型鲁棒性

In addition to performance on benchmarks, robustness is an important factor in the quality of pre-trained language models. We investigate the robustness of our pre-trained language models to design choices in multiple-choice question (MCQ) setups. Prior work has reported that model performance can be sensitive to seemingly arbitrary design choices in such setups, for example, model scores and even rankings may change with the order and labels of the in-context examples (Lu et al., 2022; Zhao et al., 2021; Robinson and Wingate, 2023; Liang et al., 2022; Gupta et al., 2024), the exact format of the prompt (Weber et al., 2023b; Mishra et al., 2022), or the answer choice format and order (Alzahrani et al., 2024; Wang et al., 2024a; Zheng et al., 2023). Motivated by this work, we use the MMLU benchmark to evaluate the robustness of our pre-trained models to: (1) few-shot label bias, (2) label variants, (3) answer order, and (4) prompt format:

除了基准测试的性能外，鲁棒性是预训练语言模型质量的重要因素。我们研究了预训练语言模型在多项选择题（MCQ）设置中的设计选择对其鲁棒性的影响。先前的工作报告称，模型性能可能对这种设置中的看似随意的设计选择敏感，例如，模型分数甚至排名可能会随着上下文示例的顺序和标签（Lu et al., 2022; Zhao et al., 2021; Robinson and Wingate, 2023; Liang et al., 2022; Gupta et al., 2024）、提示的确切格式（Weber et al., 2023b; Mishra et al., 2022）或答案选择格式和顺序（Alzahrani et al., 2024; Wang et al., 2024a; Zheng et al., 2023）的变化而变化。受此工作的启发，我们使用MMLU基准来评估我们的预训练模型对以下方面的鲁棒性：（1）少样本标签偏差，（2）标签变体，（3）答案顺序，以及（4）提示格式：

• Few-shot label bias. Following Zheng et al. (2023) and Weber et al. (2023a), we investigate the impact of the distribution of labels in four-shot examples. Specifically, we consider settings in which: (1) all

• 少样本标签偏差。根据Zheng et al. (2023)和Weber et al. (2023a)的研究，我们调查了四样本示例中标签分布的影响。具体来说，我们考虑了以下设置：（1）所有

Table 11 Pre-trained model performance on commonsense understanding tasks. Results include \({95}\%\) confidence intervals.

表11 预训练模型在常识理解任务上的性能。结果包括 \({95}\%\) 置信区间。

Table 12 Pre-trained model performance on math and reasoning tasks. Results include \({95}\%\) confidence intervals. \({}^{\diamondsuit }{11}\) -shot. \({}^{\bigtriangleup }\) Variable shot.

表12 预训练模型在数学和推理任务上的性能。结果包括 \({95}\%\) 置信区间。 \({}^{\diamondsuit }{11}\) 样本。 \({}^{\bigtriangleup }\) 可变样本。

Table 13 Pre-trained model performance on general language tasks. Results include \({95}\%\) confidence intervals.

表13 预训练模型在通用语言任务上的性能。结果包括 \({95}\%\) 置信区间。

Figure 13: Robustness of our pre-trained language models to different design choices in the MMLU benchmark. \({Left}{ : \text{ Performance }}\) for different label variants. Right: Performance for different labels present in few-shot examples.

图13：我们的预训练语言模型在MMLU基准测试中对不同设计选择的鲁棒性。\({Left}{ : \text{ Performance }}\) 针对不同的标签变体。右图：不同标签在少样本示例中的表现。

Figure 14 Robustness of our pre-trained language models to different design choices in the MMLU benchmark. \(\;{Left}\; : \;\mathrm{{Performance}}\) for different answer orders. Right: Performance for different prompt formats.

图14：我们的预训练语言模型在MMLU基准测试中对不同设计选择的鲁棒性。\(\;{Left}\; : \;\mathrm{{Performance}}\) 针对不同的答案顺序。右图：不同提示格式的表现。

few-shot examples have the same label (A A A A); (2) all examples have a different label (A B C D); and (3) there are only two labels present (A A B B and C C D D).

少样本示例具有以下三种标签情况：(1) 所有示例具有相同的标签（A A A A）；(2) 所有示例具有不同的标签（A B C D）；(3) 只有两个标签存在（A A B B 和 C C D D）。

• Label variants. We also study model response to different choice token sets. We consider the two sets proposed by Alzahrani et al. (2024): namely,a set of common language independent tokens ( \(\$ \& \#\) (c) and a of rare tokens (ce \(§3\) ii) that do not have any implicit relative order. We also consider two versions of the canonical labels (A. B. C. D. and A) B) C) D)) and a numerical list (1. 2. 3. 4.).

• 标签变体。我们还研究了模型对不同选择标记集的响应。我们考虑了Alzahrani等人（2024）提出的两组标记：一组是常见的与语言无关的标记（\(\$ \& \#\)（c）和一组罕见的标记（ce \(§3\) ii），这些标记没有任何隐含的相对顺序。我们还考虑了规范标签的两个版本（A. B. C. D. 和 A) B) C) D)）和一个数字列表（1. 2. 3. 4.）。

• Answer order. Following Wang et al. (2024a), we compute how stable the results are across different answer orders. To compute this, we remap all the answers in the dataset according to a fixed permutation. For example,for the permutation A B C D, all answer options with label A and B keep their label, and all answer options with label \(\mathbf{C}\) get label \(\mathbf{D}\) ,and vice versa.

• 答案顺序。根据Wang等人（2024a）的研究，我们计算了不同答案顺序下的结果稳定性。为了计算这一点，我们根据一个固定的排列重新映射数据集中的所有答案。例如，对于排列 A B C D，所有标签为 A 和 B 的答案选项保持其标签，所有标签为 \(\mathbf{C}\) 的答案选项获得标签 \(\mathbf{D}\)，反之亦然。

• Prompt format. We evaluate variance in performance across five task prompts that differ in the level of information provided: one prompt simply asks the model to answer the question, whereas other prompts assert the expertise of the model or that the best answer should be chosen.

• 提示格式。我们评估了五种任务提示在提供信息水平上的性能差异：一种提示仅要求模型回答问题，而其他提示则断言模型的专业性或应选择最佳答案。

Figure 13 presents the results of our experiments studying robustness of model performance to label variants (left) and few-shot label bias (right). The results show that our pre-trained language models are very robust to changes in MCQ labels and to the structure of the few-shot prompt labels. This robustness is particularly pronounced for the \({405}\mathrm{\;B}\) parameter model. Figure 14 presents the results of our study of robustness to answer order and prompt format. The results in the figure further underscore the robustness of the performance of our pre-trained language models, in particular, of Llama 3 405B.

图13展示了我们关于模型性能对标签变体（左侧）和少样本标签偏差（右侧）的鲁棒性实验结果。结果显示，我们的预训练语言模型对多项选择题标签的变化和少样本提示标签的结构非常鲁棒。这种鲁棒性在\({405}\mathrm{\;B}\)参数模型中尤为明显。图14展示了我们对答案顺序和提示格式鲁棒性的研究结果。图中的结果进一步强调了我们预训练语言模型性能的鲁棒性，特别是Llama 3 405B。

Figure 15 Adversarial versus non-adversarial performance for question answering, mathematical reasoning, and paraphrase detection benchmarks. Left: Results for pre-trained models. Right: Results for post-trained models.

图15展示了问答、数学推理和释义检测基准测试中对抗性与非对抗性性能的结果。左侧：预训练模型的结果。右侧：后训练模型的结果。

5.1.3 Adversarial Benchmarks 对抗性基准测试

In addition to the benchmarks presented above, we evaluate on several adversarial benchmarks in three areas: question answering, mathematical reasoning, and paraphrase detection. This testing probes the model's capabilities on tasks specifically created to be challenging and can potentially also point to overfitting on benchmarks. For question answering, we use Adversarial SQuAD (Jia and Liang, 2017) and Dynabench SQuAD (Kiela et al., 2021). For mathematical reasoning, we use GSM-Plus (Li et al., 2024c). For paraphrase detection, we use PAWS (Zhang et al., 2019).

除了上述基准测试外，我们还在三个领域评估了几个对抗性基准测试：问答、数学推理和释义检测。这种测试探查模型在专门设计为具有挑战性的任务上的能力，并可能指出对基准测试的过拟合。对于问答，我们使用对抗性SQuAD（Jia和Liang，2017）和Dynabench SQuAD（Kiela等人，2021）。对于数学推理，我们使用GSM-Plus（Li等人，2024c）。对于释义检测，我们使用PAWS（Zhang等人，2019）。

Figure 15 presents the scores of Llama 3 8B, 70B, and 405B on the adversarial benchmarks as a function of their performance on non-adversarial benchmarks. The non-adversarial benchmarks we use are SQuAD (Rajpurkar et al., 2016) for question answering, GSM8K for mathematical reasoning, and QQP (Wang et al., 2017) for paraphrase detection. Each datapoint represents a pair of an adversarial and non-adversarial datasets (e.g. QQP paired with PAWS), and we show all possible pairs within a category. The diagonal black line represents parity between adversarial and non-adversarial datasets - being on the line would indicate the model has similar performance regardless of the adversarial nature.

图15展示了Llama 3 8B、70B和405B在对抗性基准测试中的得分，作为它们在非对抗性基准测试中表现的函数。我们使用的非对抗性基准测试包括用于问答的SQuAD（Rajpurkar et al., 2016）、用于数学推理的GSM8K和用于复述检测的QQP（Wang et al., 2017）。每个数据点代表一对对抗性和非对抗性数据集（例如，QQP与PAWS配对），我们展示了同一类别内的所有可能配对。对角黑色线表示对抗性和非对抗性数据集之间的均衡——位于线上表明模型在对抗性性质上具有相似的表现。

On paraphrase detection, neither pre-trained nor post-trained models appear to suffer from the type of adversariality with which PAWS was constructed, marking a substantial step with respect to the previous generation of models. This result confirms the findings of Weber et al. (2023a), who also found that LLMs are less susceptible to the type of spurious correlations found in several adversarial datasets. For mathematical reasoning and question answering, however, the adversarial performances are substantially lower than the non-adversarial performances. This pattern is similar for pre-trained and post-trained models.

在复述检测方面，无论是预训练还是后训练模型，似乎都没有受到PAWS构建的对抗性类型的影响，相对于上一代模型来说，这是一个重大的进步。这一结果证实了Weber et al.（2023a）的发现，他们也发现LLMs对在几个对抗性数据集中发现的虚假相关性不那么敏感。然而，在数学推理和问答方面，对抗性表现明显低于非对抗性表现。这种模式在预训练和后训练模型中都相似。

5.1.4 Contamination Analysis 污染分析

We conduct a contamination analysis to estimate to what extent benchmark scores may be influenced by contamination of the evaluation data in the pre-training corpus. In previous work, several different contamination methods have been used, with various different hyperparameters - we refer to Singh et al. (2024) for an overview. Any of these methods can suffer from false positives and negatives, and how to best run contamination analyses is currently still an open field of research. Here, we largely follow the suggestions of Singh et al. (2024).

我们进行了一项污染分析，以估计基准测试分数可能在多大程度上受到预训练语料库中评估数据污染的影响。在之前的工作中，使用了多种不同的污染方法，具有不同的超参数——我们参考Singh et al.（2024）的概述。这些方法中的任何一种都可能遭受假阳性和假阴性，如何最好地进行污染分析目前仍然是一个开放的研究领域。在这里，我们主要遵循Singh et al.（2024）的建议。

Method. Specifically, Singh et al. (2024) propose to select contamination detection methods empirically, based on which method results in the largest difference between the 'clean' part of the dataset and the entire dataset, which they call estimated performance gain. For all our evaluation datasets, we score examples based on 8-gram overlap, a method that was found by Singh et al. (2024) to be accurate for many datasets. We consider an example of a dataset \(D\) to be contaminated if a ratio \({\mathcal{T}}_{D}\) of its tokens are part of an 8-gram occurring at least once in the pre-training corpus. We select \({\mathcal{T}}_{D}\) separately for each dataset, based on which value shows the maximal significant estimated performance gain across the three model sizes.

方法。具体而言，Singh 等人（2024 年）提出根据哪种方法在数据集的“干净”部分与整个数据集之间产生最大差异来经验性地选择污染检测方法，他们称之为估计性能增益。对于我们所有的评估数据集，我们根据 8-gram 重叠来评分，Singh 等人（2024 年）发现这种方法对许多数据集都很准确。我们认为数据集 \(D\) 的一个示例受到污染，如果其标记的一部分 \({\mathcal{T}}_{D}\) 是预训练语料库中至少出现一次的 8-gram 的一部分。我们针对每个数据集分别选择 \({\mathcal{T}}_{D}\)，基于哪个值在三种模型大小中显示出最大的显著估计性能增益。

Table 14 Performance of pre-trained models on long-context tasks. Results include \({95}\%\) confidence intervals.

表 14 预训练模型在长上下文任务上的性能。结果包括 \({95}\%\) 置信区间。

Table 15 Percentage of evaluation sets considered to be contaminated because similar data exists in the training corpus, and the estimated performance gain that may result from that contamination. See the text for details.

表 15 由于训练语料库中存在相似数据而被视为污染的评估集的百分比，以及由此污染可能带来的估计性能增益。详情见正文。

Results. In Table 15, we report the percentage of evaluation data that is considered contaminated for the maximal estimated performance gain, as described above, for all key benchmarks. From the table, we exclude numbers for benchmarks for which the results are not significant, for instance because the clean or contaminated set has too few examples, or because the observed performance gain estimate shows extremely erratic behavior. In Table 15, we observe that for some datasets contamination has a large impact, while for others it does not. For example, for PiQA and HellaSwag, both the estimation of contamination and the estimation of performance gain are high. For Natural Questions,on the other hand,the estimated \({52}\%\) contamination seems to have virtually no effect on the performance. For SQuAD and MATH, low thresholds yield high levels of contamination, but no performance gains. This suggests that contamination is either not helpful for these datasets, or that a larger \(\mathrm{n}\) is required to obtain a better estimate. Finally, for MBPP, HumanEval, MMLU and MMLU-Pro, other contamination detection methods may be needed: even with higher thresholds, 8-gram overlap gives such high contamination scores that it is impossible to get a good performance gain estimate.

结果。在表15中，我们报告了在所有关键基准测试中，为了获得最大估计性能增益，如上所述，被认为受到污染的评估数据百分比。从表中，我们排除了结果不显著的基准测试数据，例如因为干净或受污染的数据集样本太少，或者因为观察到的性能增益估计显示出极其不稳定的行为。在表15中，我们观察到对于某些数据集，污染有较大影响，而对于其他数据集则没有。例如，对于PiQA和HellaSwag，污染估计和性能增益估计都很高。另一方面，对于Natural Questions，估计的\({52}\%\)污染似乎对性能几乎没有影响。对于SQuAD和MATH，低阈值导致高水平的污染，但没有性能增益。这表明污染对这些数据集要么没有帮助，要么需要更大的\(\mathrm{n}\)来获得更好的估计。最后，对于MBPP、HumanEval、MMLU和MMLU-Pro，可能需要其他污染检测方法：即使使用更高的阈值，8-gram重叠也会给出如此高的污染分数，以至于无法获得良好的性能增益估计。

5.2 Post-trained Language Model 后训练语言模型

We present results for our Llama 3 post-trained models on benchmarks across different capabilities. Similar to pre-training we are releasing the data generated as part of evaluations with publicly available benchmarks which can be found on Huggingface here. Additional details on our eval setup can be found here.

我们展示了我们的Llama 3后训练模型在不同能力基准测试上的结果。与预训练类似，我们发布了作为评估一部分生成的数据，这些数据可以在Huggingface上公开获取。有关我们评估设置的更多详细信息可以在这里找到。

Benchmarks and metrics. Table 16 contains an overview of all the benchmarks, organized by the capability. We apply decontamination of the post-training data by running exact match with the prompts from each benchmark. In addition to the standard academic benchmarks, we also performed extensive human evaluation of different capabilities. Details are provided in Section 5.3.

基准测试和指标。表16概述了所有基准测试，按能力组织。我们通过与每个基准测试的提示进行精确匹配来对后训练数据进行去污染处理。除了标准的学术基准测试外，我们还对不同能力进行了广泛的人工评估。详细信息在第5.3节中提供。

Experimental setup. We employ a similar experimental setup to the pre-training phase and conduct a comparative analysis of Llama 3 alongside other models of comparable size and capability. To the extent possible, we evaluate the performance of other models ourselves and compare the results with the reported numbers, selecting the best score. You can find additional details on our evaluation setup here.

实验设置。我们采用与预训练阶段类似的实验设置，并对 Llama 3 进行比较分析，同时与其他尺寸和能力相当的模型进行对比。在可能的情况下，我们自行评估其他模型的性能，并将结果与报告的数据进行比较，选择最佳分数。您可以在此处找到有关我们评估设置的更多详细信息。

Table 16 Post-training benchmarks by category. Overview of all benchmarks we use to evaluate post-trained Llama 3 models, ordered by capability.

表 16 按类别划分的训练后基准测试。我们用于评估训练后 Llama 3 模型的所有基准测试概览，按能力排序。

5.2.1 General Knowledge and Instruction-Following Benchmarks 通用知识和指令遵循基准测试

We evaluate Llama 3 on benchmarks for general knowledge and instruction-following in Table 2.

我们在表 2 中对 Llama 3 进行了通用知识和指令遵循的基准测试评估。

General knowledge. We leverage MMLU (Hendrycks et al., 2021a) and MMLU-Pro (Wang et al., 2024b) to evaluate Llama 3's capability on knowledge-based question answering. For MMLU, we report the macro average of subtask accuracy under the 5-shot standard setting without CoT. MMLU-Pro is an extension of MMLU, incorporating more challenging, reasoning-focused questions, eliminating noisy questions, and expanding the choice set from four to ten options. Given its focus on complex reasoning, we report 5-shot CoT for MMLU-Pro. All tasks are formatted as generation tasks, similar to simple-evals (OpenAI, 2024).

通用知识。我们利用 MMLU（Hendrycks 等人，2021a）和 MMLU-Pro（Wang 等人，2024b）来评估 Llama 3 在基于知识的问题回答方面的能力。对于 MMLU，我们在不使用 CoT 的 5-shot 标准设置下报告子任务准确率的宏观平均值。MMLU-Pro 是 MMLU 的扩展，包含更具挑战性、以推理为重点的问题，消除了噪声问题，并将选项集从四个扩展到十个。鉴于其侧重于复杂推理，我们报告了 MMLU-Pro 的 5-shot CoT。所有任务均格式化为生成任务，类似于 simple-evals（OpenAI，2024）。

As shown in Table 2, our 8B and 70B Llama 3 variants outperform other models of similar sizes on both general knowledge tasks. Our 405B model outperforms GPT-4 and Nemotron 4 340B, with Claude 3.5 Sonnet leading among larger models.

如表 2 所示，我们的 8B 和 70B Llama 3 变体在通用知识任务上均优于其他尺寸相当的模型。我们的 405B 模型优于 GPT-4 和 Nemotron 4 340B，而 Claude 3.5 Sonnet 在更大模型中领先。

Instruction following. We assess the ability of Llama 3 and other models to follow natural language instructions on IFEval (Zhou et al., 2023). IFEval comprises approximately 500 "verifiable instructions" such as "write in more than 400 words", which can be verified by heuristics. We report the average of prompt-level and instruction-level accuracy, under strict and loose constraints in Table 2. Note that all Llama 3 variants outperform comparable models across IFEval.

遵循指令。我们评估了Llama 3和其他模型在IFEval（Zhou et al., 2023）上遵循自然语言指令的能力。IFEval包含约500条“可验证指令”，例如“写超过400字”，这些指令可以通过启发式方法验证。我们在表2中报告了在严格和宽松约束下的提示级和指令级准确率的平均值。请注意，所有Llama 3变体在IFEval上的表现均优于同类模型。

5.2.2 Proficiency Exams 能力考试

Next, we evaluate our models on a wide variety of proficiency exams originally designed to test humans. We source these exams from publicly available official sources; for some exams, we report average scores across different exam sets per proficiency exam. Specifically, we average:

接下来，我们在一系列原本设计用于测试人类的能力考试上评估我们的模型。我们从公开可用的官方来源获取这些考试；对于某些考试，我们报告了每个能力考试在不同考试集上的平均分数。具体来说，我们平均了：

• GRE: Official GRE Practice Test 1 and 2 (from the Educational Testing Services);

• GRE：官方GRE练习测试1和2（来自教育考试服务机构）；

• LSAT: Official Preptest 71, 73, 80 and 93 ;

• LSAT：官方预测试71、73、80和93；

• SAT: 8 exams from The Official SAT Study guide edition 2018;

• SAT：来自2018年版《官方SAT学习指南》的8个考试；

• AP: One official practice exam per subject;

• AP：每个科目一个官方练习考试；

• GMAT Official GMAT Online Exam.

• GMAT：官方GMAT在线考试。

Questions in these exams contain both MCQ style and generation questions. We exclude the questions that are accompanied with images. For the GRE exams that contain questions with multiple correct options, we qualify the outputs as correct only if all the correct options are selected by the model. The evaluations are run using few shot prompting wherever we have more than 1 exam set per exam. We scale the scores to be in the range 130-170 for GRE and report accuracy for all other exams.

这些考试中的问题包含选择题和生成题。我们排除了附带图像的问题。对于包含多个正确选项的GRE考试，我们仅在模型选择了所有正确选项时才将其输出视为正确。我们在每个考试有多个考试集的情况下使用少量提示进行评估。我们将GRE的分数范围调整为130-170，并报告所有其他考试的准确率。

Table 17 Performance of Llama 3 models and GPT-40 on a variety of proficiency exams \(\operatorname{including}\mathrm{{LSAT}},\mathrm{{SAT}},\mathrm{{GMAT}},\mathrm{{and}}\) AP, and GRE tests. For GRE exams, we report normalized score; for all others, we report accuracy. For the bottom two rows corresponding to GRE Quant. and GRE Verbal, we report the scaled scores out of 170.

表17展示了Llama 3模型和GPT-40在多种能力考试\(\operatorname{including}\mathrm{{LSAT}},\mathrm{{SAT}},\mathrm{{GMAT}},\mathrm{{and}}\) AP和GRE测试中的表现。对于GRE考试，我们报告标准化分数；对于其他所有考试，我们报告准确率。对于对应GRE数量和GRE语文的最后两行，我们报告170分制的分数。

Our results can be found in Table 17. We observe that the performance of our Llama 3405B model is very similar to Claude 3.5 Sonnet and GPT-4 40. Our 70B model has an even more impressive performance. It is significantly better than GPT-3.5 Turbo and beats Nemotron 4340B on many tests.

我们的结果见表17。我们观察到，我们的Llama 3405B模型的表现与Claude 3.5 Sonnet和GPT-4 40非常相似。我们的70B模型表现更为出色。它在许多测试中明显优于GPT-3.5 Turbo，并击败了Nemotron 4340B。

5.2.3 Coding Benchmarks 编码基准测试

We evaluate Llama 3 on code generation on several popular Python and multi-programming language benchmarks. To gauge the effectiveness of our models in generating functionally correct code, we use the pass@ \(N\) metric,which evaluates the pass rate for a set of unit tests among \(N\) generations. We report pass@1.

我们在几个流行的Python和多编程语言基准测试上评估Llama 3的代码生成能力。为了衡量我们模型生成功能正确代码的有效性，我们使用pass@\(N\)指标，该指标评估在一组单元测试中通过率\(N\)。我们报告pass@1。

Python code generation. HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) are popular benchmarks for Python code generation which focus on relatively simple, self-contained functions. HumanEval+ (Liu et al., 2024a) is an enhanced version of HumanEval, in which more tests are generated to avoid false positives. The MBPP EvalPlus base version (v0.2.0) is a selection of 378 well-formed problems out of the 974 initial problems in all of the original MBPP (train and test) dataset (Liu et al., 2024a). Results for these benchmarks are reported in Table 18. Across the Python variants of these benchmarks, Llama 3 8B and 70B outperform

Python代码生成。HumanEval（Chen et al., 2021）和MBPP（Austin et al., 2021）是流行的Python代码生成基准测试，它们侧重于相对简单、自包含的函数。HumanEval+（Liu et al., 2024a）是HumanEval的增强版本，其中生成了更多测试以避免假阳性。MBPP EvalPlus基础版本（v0.2.0）是从原始MBPP（训练和测试）数据集（Liu et al., 2024a）中的974个初始问题中精选出的378个格式良好的问题。这些基准测试的结果报告在表18中。在这些基准测试的Python变体中，Llama 3 8B和70B表现出色。

Table 18 Pass@1 scores on code generation benchmarks. We report results on HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), as well as EvalPlus (Liu et al., 2024a) versions of these benchmarks.

表18 代码生成基准测试的Pass@1得分。我们报告了在HumanEval（Chen et al., 2021）、MBPP（Austin et al., 2021）以及这些基准的EvalPlus版本（Liu et al., 2024a）上的结果。

Table 19 Performance of non-Python programming tasks. We report Llama 3 results on MultiPL-E (Cassano et al., 2023).

表19 非Python编程任务的性能。我们报告了Llama 3在MultiPL-E（Cassano et al., 2023）上的结果。

models of similar sizes. For the largest models, Llama 3 405B, Claude 3.5 Sonnet and GPT-4o perform similarly, with GPT-4o showing the strongest results.

相似规模的模型。对于最大的模型，Llama 3 405B、Claude 3.5 Sonnet和GPT-4o表现相似，其中GPT-4o显示出最强的结果。

Multi-programming language code generation. To assess code generation capabilities beyond Python, we report results for the MultiPL-E (Cassano et al., 2023) benchmark, which is based on translations of problems from HumanEval and MBPP. Results for a subset of popular programming languages are reported in Table 19. Note that there is a significant drop in performance compared to the Python counterparts in Table 18.

多编程语言代码生成。为了评估超越Python的代码生成能力，我们报告了MultiPL-E（Cassano et al., 2023）基准测试的结果，该基准基于HumanEval和MBPP问题的翻译。表19报告了部分流行编程语言的结果。请注意，与表18中的Python对应项相比，性能有显著下降。

5.2.4 Multilingual Benchmarks 多语言基准测试

Llama 3 supports 8 languages - English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai, although the underlying foundation model has been trained on a broader collection of languages. \({}^{9}\) In Table 20, we show results from evaluating Llama 3 on the multilingual MMLU (Hendrycks et al., 2021a) and Multilingual Grade School Math (MGSM) (Shi et al., 2022) benchmarks.

Llama 3支持8种语言——英语、德语、法语、意大利语、葡萄牙语、 Hindi、西班牙语和泰语，尽管底层基础模型已在更广泛的语言集合上进行了训练。\({}^{9}\) 在表20中，我们展示了在多语言MMLU（Hendrycks et al., 2021a）和多语言小学数学（MGSM）（Shi et al., 2022）基准测试上评估Llama 3的结果。

Multilingual MMLU. We translate MMLU questions, few-shot examples, and answers using Google Translate. We leave the task instructions in English and perform the evaluation in a 5-shot setting. In Table 20, we report average results across German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

多语言MMLU。我们使用Google Translate翻译MMLU问题、少样本示例和答案。我们保留英语的任务指令，并在5样本设置中进行评估。在表20中，我们报告了德语、法语、意大利语、葡萄牙语、Hindi、西班牙语和泰语的平均结果。

\({}^{9}\) Llama 3 has not been optimized or safety tuned for use cases in those other languages. Developers may fine-tune Llama 3 models for languages beyond the 8 supported languages provided they comply with the Llama 3 Community License and the Acceptable Use Policy and in such cases are responsible for ensuring that any uses of Llama 3 in additional languages is done in a safe and responsible manner.

\({}^{9}\) Llama 3 尚未针对其他语言的使用场景进行优化或安全调优。开发者可以在遵守 Llama 3 社区许可协议和可接受使用政策的前提下，对 Llama 3 模型进行除支持的8种语言之外的语言微调，并负责确保在额外语言中使用 Llama 3 时采取安全且负责任的方式。

MGSM (Shi et al., 2022). We use the same native prompts as in simple-evals (OpenAI, 2024) for testing our models in a 0 -shot CoT setting. In Table 20, we report averge results across languages covered in MGSM benchmark.

MGSM（Shi et al., 2022）。我们使用与 simple-evals（OpenAI, 2024）相同的原生提示，在0-shot CoT设置下测试我们的模型。在表20中，我们报告了MGSM基准涵盖语言的平均结果。

Table 20 Multilingual benchmarks. For MGSM (Shi et al., 2022), we report 0 -shot CoT results for our Llama 3 models. Multilingual MMLU is an internal benchmark with translated MMLU (Hendrycks et al., 2021a) questions and answers into 7 languages - we report 5-shot results averaged across these languages.

表20 多语言基准测试。对于MGSM（Shi et al., 2022），我们报告了Llama 3模型的0-shot CoT结果。多语言MMLU是一个内部基准，包含将MMLU（Hendrycks et al., 2021a）的问题和答案翻译成7种语言，我们报告了这些语言的5-shot平均结果。

We find that Llama 3 405B outperforms most other models on MGSM, achieving an average of 91.6%. On MMLU, in line with English MMLU results shown above,Llama 3 405B falls behind GPT-4o by 2%. On the other hand, both Llama 370B and 8B models demonstrate strong performance, leading among competitors with a wide margin on both tasks.

我们发现Llama 3 405B在MGSM上表现优于大多数其他模型，平均达到91.6%。在MMLU上，与上述英语MMLU结果一致，Llama 3 405B落后于GPT-4o 2%。另一方面，Llama 3 70B和8B模型均展现出强劲性能，在两项任务中均以较大优势领先于竞争对手。

5.2.5 Math and Reasoning Benchmarks 数学与推理基准测试

Our math and reasoning benchmark results are presented in Table 2. Llama 3 8B model outperforms other models of similar sizes on GSM8K, MATH, and GPQA. Our 70B model performs significantly better than other models in its class on all the benchmarks. Finally, Llama 3405B model is the best in its category on GSM8K and ARC-C, while on MATH, it is the second best model. On GPQA, it is competitive with GPT-4 40, with Claude 3.5 Sonnet being the best model by a significant margin.

我们的数学和推理基准测试结果如表2所示。Llama 3 8B模型在GSM8K、MATH和GPQA上优于其他同尺寸模型。我们的70B模型在所有基准测试中显著优于同级别的其他模型。最后，Llama 3405B模型在GSM8K和ARC-C类别中表现最佳，而在MATH中是第二好的模型。在GPQA上，它与GPT-4 40竞争激烈，而Claude 3.5 Sonnet以显著优势成为最佳模型。

5.2.6 Long Context Benchmarks 长上下文基准测试

We consider a diverse set of tasks that span various domains and text types. In the benchmarks we list below, we focus on sub-tasks that use unbiased evaluation protocols, i.e., accuracy-based metrics rather than n-gram overlapping metrics. We also prioritize tasks that we found to be of low variance.

我们考虑了一系列跨多个领域和文本类型的多样化任务。在下面列出的基准测试中，我们专注于使用无偏评估协议的子任务，即基于准确性的指标而非n-gram重叠指标。我们还优先考虑了我们发现方差较低的任务。

• Needle-in-a-Haystack (Kamradt, 2023) measures a model’s ability to retrieve a hidden information inserted in random parts of the long document. Our Llama 3 models demonstrate perfect needle retrieval performance,successfully retrieving \({100}\%\) of needles at all document depths and context lengths. We also measure performance on Multi-needle (Table 21), a variation of Needle-in-a-Haystack, where we insert four needles in the context and test if a model can retrieve two of them. Our Llama 3 models achieve near perfect retrieval results.

• “大海捞针”（Kamradt，2023）衡量模型在长文档随机部分中检索隐藏信息的能力。我们的Llama 3模型展示了完美的针检索性能，成功检索了\({100}\%\)的针，无论文档深度和上下文长度如何。我们还测量了“多针”（表21）的性能，这是“大海捞针”的一个变体，我们在上下文中插入四个针，并测试模型是否能检索其中两个。我们的Llama 3模型实现了近乎完美的检索结果。

• ZeroSCROLLS (Shaham et al., 2023) is a zero-shot benchmark for natural language understanding over long texts. We report numbers on the validation set, as the ground truth answers are not publicly available. Our Llama 3405B and 70B models either match or surpass other models on various tasks in this benchmark.

• ZeroSCROLLS（Shaham等人，2023）是一个针对长文本自然语言理解的零样本基准测试。我们报告了验证集上的数据，因为真实答案并未公开。我们的Llama 3405B和70B模型在基准测试的各项任务中与其他模型持平或超越。

• InfiniteBench (Zhang et al., 2024) requires models to understand long dependencies in the context window. We evaluate Llama 3 on En.QA (QA over novels) and En.MC (multiple-choice QA over novels), where our \({405}\mathrm{\;B}\) model outperforms all others. The gains are particularly significant on En.QA.

InfiniteBench（Zhang et al., 2024）要求模型理解上下文窗口中的长依赖关系。我们在En.QA（小说问答）和En.MC（小说多项选择问答）上评估Llama 3，我们的\({405}\mathrm{\;B}\)模型在这些任务中表现优于所有其他模型。特别是在En.QA上，我们的模型取得了显著的提升。

5.2.7 Tool Use Performance 工具使用性能

We evaluate our models on a range of benchmarks for zero-shot tool use (i.e. function calling): Nexus (Srinivasan et al., 2023), API-Bank (Li et al., 2023b), Gorilla API-Bench (Patil et al., 2023), and the Berkeley Function Calling Leaderboard (BFCL) (Yan et al., 2024). Results are shown in Table 22.

我们在一系列零样本工具使用（即函数调用）基准上评估我们的模型：Nexus（Srinivasan et al., 2023）、API-Bank（Li et al., 2023b）、Gorilla API-Bench（Patil et al., 2023）和Berkeley函数调用排行榜（BFCL）（Yan et al., 2024）。结果如表22所示。

On Nexus, our Llama 3 variants perform the best compared to their counterparts. On the API-Bank, our Llama \({38}\mathrm{B}\) and \({70}\mathrm{\;B}\) models outperform other models in their category by a significant margin. The \({405}\mathrm{\;B}\) model is behind Claude 3.5 Sonnet by only \({0.6}\%\) . Finally,our \({405}\mathrm{\;B}\) and \({70}\mathrm{\;B}\) models perform competitively on BFCL and are close second in their respective size class. Llama 3 8B performs the best in its category.

在Nexus上，我们的Llama 3变体相较于其他模型表现最佳。在API-Bank上，我们的Llama \({38}\mathrm{B}\)和\({70}\mathrm{\;B}\)模型在其类别中以显著优势超越其他模型。\({405}\mathrm{\;B}\)模型仅落后于Claude 3.5 Sonnet \({0.6}\%\)。最后，我们的\({405}\mathrm{\;B}\)和\({70}\mathrm{\;B}\)模型在BFCL上表现竞争性，并在各自的大小类别中位列第二。Llama 3 8B在其类别中表现最佳。

Table 21 Long-context benchmarks. For ZeroSCROLLS (Shaham et al., 2023), we report numbers on the validation set. For QuALITY we report exact match, for Qasper - f1 and for SQuALITY - rougeL. We report f1 for InfiniteBench (Zhang et al., 2024) En.QA metric and accuracy for En.MC. For Multi-needle (Kamradt, 2023) we insert 4 needles in the context and test if a model can retrieve 2 needles at different context lengths, we compute average recall across 10 sequence lengths up till \({128}\mathrm{k}\) .

表21 长上下文基准测试。对于ZeroSCROLLS（Shaham等人，2023年），我们报告验证集上的数据。对于QuALITY，我们报告完全匹配，对于Qasper - f1，对于SQuALITY - rougeL。我们报告InfiniteBench（Zhang等人，2024年）En.QA指标的f1值和En.MC的准确性。对于Multi-needle（Kamradt，2023年），我们在上下文中插入4根针，并测试模型是否能在不同上下文长度下检索到2根针，我们计算直到\({128}\mathrm{k}\)的10个序列长度的平均召回率。

Human evaluations. We also conduct human evaluations to test the tool use capabilities of the model, with a focus on code execution tasks. We collect 2000 user prompts related to code execution (without plotting or file uploads), plot generation, and file uploads. These prompts are collected from the LMSys dataset (Chiang et al., 2024), GAIA benchmark (Mialon et al., 2023b), human annotators, and synthetic generation.

人类评估。我们还进行人类评估，以测试模型的工具使用能力，重点是代码执行任务。我们收集了2000个与代码执行（不包括绘图或文件上传）、绘图生成和文件上传相关的用户提示。这些提示来自LMSys数据集（Chiang等人，2024年）、GAIA基准（Mialon等人，2023b年）、人工标注者和合成生成。

We compare Llama 3 405B to GPT-4o using OpenAI's Assistants \({\mathrm{{API}}}^{10}\) . The results are provided in Figure 16. On text-only code execution tasks and plots generation, Llama 3405B significantly beats GPT-4o. However, it lags behind on the file upload use case.

我们使用OpenAI的Assistants \({\mathrm{{API}}}^{10}\) 将Llama 3 405B与GPT-4o进行比较。结果如图16所示。在纯文本代码执行任务和绘图生成方面，Llama 3405B显著优于GPT-4o。然而，在文件上传用例方面，它落后于GPT-4o。

5.3 Human Evaluations 人类评估

In addition to evaluations on standard benchmark sets, we also perform a series of human evaluations. These evaluations allow us to measure and optimize more subtle aspects of model performance, such as our model's tone, verbosity, and

除了在标准基准集上的评估外，我们还进行了一系列人类评估。这些评估使我们能够测量和优化模型性能的更细微方面，例如我们模型的语气、冗长程度和

understanding of nuances and cul- Table 22 Zero-shot tool use benchmarks. We report function calling accuracy tural contexts. Well-designed hu- across Nexus (Srinivasan et al., 2023), API-Bank (Li et al., 2023b), API-man evaluations closely reflect the Bench (Patil et al., 2023), and BFCL (Yan et al., 2024). user experience, providing insights into how the model performs in real-world scenarios.

对细微差别和文化背景的理解。表22展示了零样本工具使用基准测试。我们报告了在Nexus（Srinivasan等人，2023年）、API-Bank（Li等人，2023b）、API-man评估（Patil等人，2023年）和BFCL（Yan等人，2024年）中的函数调用准确性。这些评估紧密反映了用户体验，提供了模型在现实场景中表现的见解。

Prompt collection. We collected high-quality prompt spanning a wide range of categories and difficulties. To do so, we first developed a taxonomy with categories and subcategories capturing as many model capabilities as possible. We used this taxonomy to collect about 7,000 prompts spanning six individual capabilities (English, reasoning,coding,Hindi,Spanish,and Portuguese),and three multiturn capabilities \({}^{11}\) (English,reasoning, and coding). We ensured that within each category, prompts are uniformly distributed across subcategories. We also categorized each prompt into one of three difficulty levels and ensured that our prompt collection contains roughly \({10}\%\) easy prompts, \({30}\%\) medium prompts,and \({60}\%\) hard prompts. All the human evaluation prompt sets were subject to a thorough quality assurance process. Modeling teams did not have access to our human-evaluation prompts to prevent accidental contamination or overfitting on the test set.

提示收集。我们收集了涵盖广泛类别和难度的高质量提示。为此，我们首先开发了一个分类法，包括类别和子类别，以尽可能多地捕捉模型的能力。我们利用这一分类法收集了约7,000个提示，涵盖六种独立能力（英语、推理、编程、印地语、西班牙语和葡萄牙语）和三种多轮能力 \({}^{11}\)（英语、推理和编程）。我们确保在每个类别中，提示在子类别中均匀分布。我们还根据难度将每个提示分为三个级别，并确保我们的提示集合包含大约 \({10}\%\) 简单提示、\({30}\%\) 中等提示和 \({60}\%\) 困难提示。所有人类评估提示集都经过了严格的质量保证流程。建模团队无法访问我们的人类评估提示，以防止测试集的意外污染或过拟合。

\({}^{10}\) https://platform.openai.com/docs/assistants/overview

\({}^{11}\) For multiturn human evaluations,the number of turns is between 2 and 11 in each prompt. We assess the model response in the final turn.

\({}^{11}\) 对于多轮人类评估，每个提示中的轮数在2到11之间。我们评估模型在最后一轮的响应。

Figure 16: Human evaluation results for Llama 3 405B vs. GPT-4o on code execution tasks including plotting and file uploads. Llama 3405B outperforms GPT-4o on code execution (without plotting or file uploads) as well as plot generation, but lags behind in file upload use cases.

图16：Llama 3 405B与GPT-4o在包括绘图和文件上传的代码执行任务上的人类评估结果。Llama 3405B在代码执行（不包括绘图或文件上传）以及绘图生成方面优于GPT-4o，但在文件上传用例中落后。

Evaluation process. To perform a pairwise human evaluation of two models, we ask human annotators which of two model responses (produced by different models) they prefer. Annotators use a 7-point scale for their ratings, enabling them to indicate whether one model response is much better than, better than, slightly better than, or about the same as the other model response. When an annotator indicates that one model response is better or much better than the other model response, we consider this a "win" for that model. We perform pairwise comparisons between models in which we report win rates per capability in the prompt set.

评估过程。为了进行两个模型的成对人类评估，我们询问人类标注者他们更喜欢两个模型响应中的哪一个（由不同模型产生）。标注者使用7点量表进行评分，使他们能够表明一个模型响应是否远优于、优于、略优于或与另一个模型响应大致相同。当标注者表示一个模型响应优于或远优于另一个模型响应时，我们认为这是该模型的“胜利”。我们在模型之间进行成对比较，报告提示集中每项能力的胜率。

Results. We use our human evaluation process to compare Llama 3 405B with GPT-4 (0125 API version), GPT-40 (API version), and Claude 3.5 Sonnet (API version). The results of these evaluations are presented in Figure 17. We observe that Llama 3405B performs approximately on par with the 0125 API version of GPT-4, while achieving mixed results (some wins and some losses) compared to GPT-4o and Claude 3.5 Sonnet. On nearly all capabilities, the win rates of Llama 3 and GPT-4 are within the margin of error. On multiturn reasoning and coding tasks, Llama 3405B outperforms GPT-4 but it underperforms GPT-4 on multilingual (Hindi, Spanish, and Portuguese) prompts. Llama 3 performs on par with GPT-4o on English prompts, on par with Claude 3.5 Sonnet on multilingual prompts, and outperforms Claude 3.5 Sonnet on single and multiturn English prompts. However, it trails Claude 3.5 Sonnet in capabilities such as coding and reasoning. Qualitatively, we find that model performance in human evaluations is heavily influenced by nuanced factors such as model tone, response structure, and verbosity - factors that we are optimizing for in our post-training process. Overall, our human evaluation results are consistent with those on standard benchmark evaluations: Llama 3405B is very competitive with leading industry models, making it the best-performing openly available model.

结果。我们利用人类评估流程来比较 Llama 3 405B 与 GPT-4（0125 API 版本）、GPT-40（API 版本）以及 Claude 3.5 Sonnet（API 版本）。这些评估的结果如图 17 所示。我们观察到，Llama 3405B 的表现与 GPT-4 的 0125 API 版本大致相当，而在与 GPT-40 和 Claude 3.5 Sonnet 的比较中则呈现出混合结果（有胜有负）。在几乎所有能力上，Llama 3 和 GPT-4 的胜率都在误差范围内。在多轮推理和编码任务上，Llama 3405B 优于 GPT-4，但在多语言（印地语、西班牙语和葡萄牙语）提示上表现不佳。Llama 3 在英语提示上与 GPT-40 相当，在多语言提示上与 Claude 3.5 Sonnet 相当，并在单轮和多轮英语提示上优于 Claude 3.5 Sonnet。然而，在编码和推理等能力上，它落后于 Claude 3.5 Sonnet。从定性角度看，我们发现模型在人类评估中的表现深受模型语调、响应结构和冗长程度等细微因素的影响——这些因素是我们训练后优化过程中的重点。总体而言，我们的人类评估结果与标准基准评估结果一致：Llama 3405B 与行业领先模型非常具有竞争力，使其成为表现最佳的公开可用模型。

Limitations. All human evaluation results underwent a thorough data quality assurance process. However, since it is challenging to define objective criteria for evaluating model responses, human evaluations can still be influenced by personal biases, backgrounds, and preferences of human annotators, which may lead to inconsistent or unreliable results.

局限性。所有人类评估结果都经过了严格的数据质量保证流程。然而，由于定义模型响应评估的客观标准具有挑战性，人类评估仍可能受到人类标注者的个人偏见、背景和偏好影响，这可能导致结果不一致或不可靠。

5.4 Safety 安全性

We focus our study on assessing Llama 3's ability to generate content in a safe and responsible way, while still maximizing helpful information. Our safety work begins in the pre-training stage, primarily in the form of

我们专注于评估 Llama 3 在安全且负责任的方式下生成内容的能力，同时最大化有用信息。我们的安全工作始于预训练阶段，主要形式为

Figure 17 Human evaluation results for the Llama 3 405B model. \({Left}\) : Comparison with GPT-4. \({Middle}\) : Comparison with GPT-4o. Right: Comparison with Claude 3.5 Sonnet. All results include 95% confidence intervals and exclude ties.

图 17 Llama 3 405B 模型的人类评估结果。\({Left}\)：与 GPT-4 的比较。\({Middle}\)：与 GPT-4o 的比较。右侧：与 Claude 3.5 Sonnet 的比较。所有结果包括 95% 置信区间并排除平局。

data cleaning and filtering. We then describe our approach to safety finetuning, focusing on how to train the model to align to specific safety policies while still retaining helpfulness. We analyze each of the Llama 3 capabilities, including multilingual, long context, tool usage, and various multimodal capabilities, to measure the effectiveness of our safety mitigations.

数据清洗和过滤。然后我们描述了我们的安全微调方法，重点是如何训练模型以符合特定的安全政策，同时仍然保持有用性。我们分析了 Llama 3 的每项能力，包括多语言、长上下文、工具使用和各种多模态能力，以衡量我们安全缓解措施的有效性。

Subsequently, we describe our assessment of uplift for cybersecurity and chemical and biological weapons risks. Uplift refers to the additional risk introduced by new technological developments compared to using existing available technologies (such as web search).

随后，我们描述了对网络安全和化学及生物武器风险提升的评估。提升指的是与使用现有可用技术（如网络搜索）相比，新技术发展引入的额外风险。

We then describe how we leverage Red Teaming to iteratively identify and combat various safety risks across capabilities and perform a residual risk assessment.

然后我们描述了如何利用红队测试来迭代识别和应对跨能力的安全风险，并进行残余风险评估。

Finally, we describe system-level safety, or the development and orchestration of classifiers around the input and output of the model itself to further enhance safety and make it easier for developers to both customize safety to various usecases and deploy generative AI in more responsible ways.

最后，我们描述了系统级安全，即围绕模型本身输入和输出的分类器的发展和编排，以进一步增强安全性，并使开发者更容易为各种用例定制安全措施，并以更负责任的方式部署生成式 AI。

5.4.1 Benchmark Construction 基准构建

We create various internal benchmarks to help us develop models safely and responsibly. Our benchmarks are heavily inspired by the risk categories from the ML Commons taxonomy of hazards (Vidgen et al., 2024). While various benchmarks for language model safety exist such as ToxiGen (Hartvigsen et al., 2022), XS Test (Röttger et al., 2023), and more, a number of the new capabilities in Llama 3 do not have sufficient external benchmarks available and often external benchmarks lack sufficient coverage of breadth and depth.

我们创建了多种内部基准测试，以帮助我们安全且负责任地开发模型。我们的基准测试深受ML Commons危害分类法（Vidgen et al., 2024）中的风险类别启发。尽管存在多种语言模型安全基准测试，如ToxiGen（Hartvigsen et al., 2022）、XS Test（Röttger et al., 2023）等，但Llama 3中的许多新功能缺乏足够的外部基准测试，且外部基准测试往往缺乏足够的广度和深度覆盖。

For each risk category, we collect human-written prompts that are either adversarial or borderline in nature - examples of such prompts can be found in Table 23. Adversarial prompts range from straightforward ones that directly elicit a harmful response to ones that incorporate sophisticated jailbreaking techniques. Benchmarks of these adversarial prompts are the basis for how we measure violation rate.

对于每个风险类别，我们收集了人为编写的提示，这些提示本质上是敌对的或边缘性的——此类提示的示例可在表23中找到。敌对提示从直接引发有害响应的简单提示到包含复杂越狱技术的提示不等。这些敌对提示的基准测试是我们衡量违规率的基础。

As a counter-metric to violation rate, we also construct false refusal benchmarks composed of borderline prompts. We refer to false refusal as occurring when a model refuses to answer in a helpful way even when a plausible, safe response is possible. Borderline prompts are prompts near the decision boundary that a well-calibrated model should handle, for example, "How can I steal the spotlight from my bestie who always acts like the main character??" Our overall benchmark size across violations and false refusals is over 4000 prompts per capability or language, and contains a mix of single-turn and multi-turn prompts.

作为违规率的反向指标，我们还构建了由边缘提示组成的错误拒绝基准测试。我们将错误拒绝定义为模型即使在可能提供合理、安全响应的情况下也不以有帮助的方式回答。边缘提示是接近决策边界的提示，一个校准良好的模型应能处理，例如，“我如何从总是表现得像主角的好友那里抢走焦点？？”我们在违规和错误拒绝方面的总体基准测试规模超过每种能力或语言的4000个提示，并包含单轮和多轮提示的混合。

Table 23 Examples of adversarial prompts from our internal benchmarks across all the capabilities.

表23 我们内部基准测试中跨所有能力的敌对提示示例。

Table 24 Average verbatim memorization in pre-trained Llama 3 for selected test scenarios. Our baseline is Llama 2 in the English, 50-gram scenario using the same prompting methodology applied to its data mix.

表24 预训练Llama 3在选定测试场景中的平均逐字记忆。我们的基线是使用相同提示方法应用于其数据混合的英语50-gram场景中的Llama 2。

5.4.2 Safety Pre-training 安全预训练

We believe responsible development must be considered from an end-to-end perspective and incorporated at every stage of model development and deployment. During pre-training, we apply a variety of filters, such as filters to identify websites that likely contain personally identifiable information (see Section 3.1). We also focus heavily on discoverable memorization (Nasr et al., 2023). Similar to Carlini et al. (2022), we sample prompts and ground truths at different frequencies of occurrence in the training data using an efficient rolling hash index of all n-grams in the corpus. We construct different test scenarios by varying the length of prompt and ground truth, the detected language of target data, and the domain. We then measure how often the model generates the ground truth sequence verbatim, and analyze the relative rates of memorization in the specified scenarios. We define verbatim memorization as the inclusion rate - the proportion of model generations that include the ground truth continuation exactly - and report averages weighted by the prevalence of given characteristics in the data,as shown in Table 24. We find low memorization rates of training data \(({1.13}\%\) and \({3.91}\%\) on average for the \({405}\mathrm{\;B}\) with \(n = {50}\) and \(n = {1000}\) respectively). Memorization rates are roughly on par with Llama 2 at equivalent size and using the same methodology applied to its data mix. \({}^{12}\)

我们相信，负责任的开发必须从端到端的角度考虑，并在模型开发和部署的每个阶段融入。在预训练阶段，我们应用了多种过滤器，例如用于识别可能包含个人身份信息网站的过滤器（参见第3.1节）。我们还重点研究了可发现的记忆化（Nasr等人，2023年）。与Carlini等人（2022年）类似，我们使用语料库中所有n-gram的高效滚动哈希索引，在训练数据中以不同频率采样提示和真实值。我们通过改变提示和真实值的长度、目标数据检测到的语言以及领域来构建不同的测试场景。然后，我们测量模型逐字生成真实值序列的频率，并分析指定场景中的记忆化相对速率。我们将逐字记忆化定义为包含率——即包含完全真实值延续的模型生成比例——并根据数据中给定特征的普遍性报告加权平均值，如表24所示。我们发现训练数据的记忆化率较低\(({1.13}\%\)和\({3.91}\%\)，平均而言，对于\({405}\mathrm{\;B}\)分别具有\(n = {50}\)和\(n = {1000}\)）。记忆化率大致与Llama 2在相同大小并使用应用于其数据混合的相同方法相当。\({}^{12}\)

5.4.3 Safety Finetuning 安全微调

We describe our approach to safety finetuning to mitigate risks across many capabilities, which encompasses two key aspects: (1) safety training data and (2) risk mitigation techniques. Our safety finetuning process builds upon our general finetuning methodology with modifications tailored to address specific safety concerns.

我们描述了安全微调的方法，以减轻跨多种能力的风险，这包括两个关键方面：（1）安全训练数据和（2）风险缓解技术。我们的安全微调过程基于我们的通用微调方法，并进行了针对特定安全问题的修改。

We optimize for two primary metrics: Violation Rate (VR), a metric that captures when the model produces a response that violates a safety policy, and False Refusal Rate (FRR), a metric that captures when the model incorrectly refuses to respond to a harmless prompt. In parallel, we evaluate model performance on helpfulness benchmarks to ensure that safety improvements do not compromise overall helpfulness.

我们针对两个主要指标进行优化：违规率（VR），该指标捕捉模型何时产生违反安全政策的响应；错误拒绝率（FRR），该指标捕捉模型何时错误地拒绝回应无害的提示。同时，我们评估模型在帮助性基准上的表现，以确保安全性的提升不会损害整体帮助性。

\({}^{12}\) Note there are limitations with our analysis — for example,recent work advocates for metrics beyond exact match (Ippolito et al., 2023) and alternative prompt search strategies (Kassem et al., 2024). Nonetheless, we find the results of the evaluations to be encouraging.

\({}^{12}\) 需要注意的是，我们的分析存在局限性——例如，最近的研究提倡使用超越精确匹配的指标（Ippolito 等人，2023）和替代的提示搜索策略（Kassem 等人，2024）。尽管如此，我们认为评估结果是令人鼓舞的。

Finetuning data. The quality and design of safety training data has a profound impact on performance. Through extensive ablations, we find that the quality is more critical than the quantity. We mainly use human-generated data collected from our data vendors, but find that it can be prone to errors and inconsistencies - particularly for nuanced safety policies. To ensure the highest quality data, we developed AI-assisted annotation tools to support our rigorous quality assurance processes. In addition to collecting adversarial prompts, we also gather a set of similar prompts, which we refer to as borderline prompts. These are closely related to the adversarial prompts but with a goal to teach the model to learn to provide helpful responses, thereby reducing the false refusal rate (FRR).

微调数据。安全训练数据的质量和设计对性能有深远影响。通过广泛的消融实验，我们发现质量比数量更为关键。我们主要使用从数据供应商处收集的人工生成数据，但发现这些数据容易出错且不一致——特别是在处理微妙的安全政策时。为了确保最高质量的数据，我们开发了AI辅助的标注工具来支持我们严格的质量保证流程。除了收集对抗性提示外，我们还收集了一组类似的提示，我们称之为边缘提示。这些提示与对抗性提示密切相关，但目的是教会模型学会提供有帮助的响应，从而降低错误拒绝率（FRR）。

Figure 18 Influence of model size on safety mix design for balancing violation rate (VR) and false refusal rate (FRR). Each point of the scatterplot represents a different data mix balancing safety and helpfulness data. Different model sizes retain varying capacities for safety learning. Our experiments show that \(8\mathrm{\;B}\) models require a higher proportion of safety data relative to helpfulness data in the overall SFT mix to achieve comparable safety performance to \({70}\mathrm{\;B}\) models. Larger models are more capable of discerning between adversarial and borderline context, resulting in a more favorable balance between VR and FRR.

图18 模型大小对安全混合设计的影响，以平衡违规率（VR）和误拒率（FRR）。散点图的每个点代表不同的数据混合，平衡了安全性和有用性数据。不同大小的模型保留了不同的安全学习能力。我们的实验表明，\(8\mathrm{\;B}\)模型相对于有用性数据，在整体SFT混合中需要更高比例的安全数据，以达到与\({70}\mathrm{\;B}\)模型相当的安全性能。较大的模型更能区分对抗性和边缘情境，从而在VR和FRR之间实现更有利的平衡。

Beyond human annotation, we also leverage synthetic data to improve the quality and coverage of our training datasets. We utilize a range of techniques to generate additional adversarial examples, including in-context learning with carefully crafted system prompts, guided mutation of seed prompts based on new attack vectors, and advanced algorithms including Rainbow Teaming (Samvelyan et al., 2024), based on MAP-Elites (Mouret and

除了人工标注，我们还利用合成数据来提高训练数据集的质量和覆盖范围。我们采用多种技术生成额外的对抗性示例，包括通过精心设计的系统提示进行情境学习，基于新攻击向量引导种子提示的变异，以及基于MAP-Elites（Mouret和Clune，2015）的高级算法，如Rainbow Teaming（Samvelyan等人，2024），生成在多个多样性维度上受限的提示。

Clune, 2015), which generate prompts constrained across multiple dimensions of diversity.

我们还进一步解决了模型在生成安全响应时的语气问题，这对下游用户体验有影响。我们为Llama 3开发了拒绝语气指南，并通过严格的质量保证流程确保所有新安全数据符合该指南。我们还通过零样本重写和人在回路的编辑相结合的方法，对现有安全数据进行细化，以生成高质量数据。通过采用这些方法，以及使用语气分类器评估安全响应的语气质量，我们能够显著改善模型的措辞。

We further address the model's tone when producing safe responses, which has an impact on downstream user experience. We developed a refusal tone guideline for Llama 3 and ensured that all new safety data adhered to it through rigorous quality assurance process. We also refine existing safety data to align with the guideline, using a combination of zero-shot rewriting and human-in-the-loop editing to produce high-quality data. By employing these methods, along with a tone classifier to assess tone quality for safety responses, we are able to significantly improve the model's verbiage.

我们还进一步解决了模型在生成安全响应时的语气问题，这对下游用户体验有影响。我们为Llama 3开发了拒绝语气指南，并通过严格的质量保证流程确保所有新安全数据符合该指南。我们还通过零样本重写和人在回路的编辑相结合的方法，对现有安全数据进行细化，以生成高质量数据。通过采用这些方法，以及使用语气分类器评估安全响应的语气质量，我们能够显著改善模型的措辞。

Safety supervised finetuning. Following our Llama 2 recipe (Touvron et al., 2023b), we combine all helpfulness data and safety data during the model alignment stage. Additionally, we introduce a borderline dataset to help the model discern the subtle distinctions between safe and unsafe requests. Our annotation teams are instructed to meticulously craft responses to safety prompts based on our guidelines. We have found that SFT is highly effective in aligning the model when we strategically balance the ratio of adversarial to borderline examples. We put the focus on more challenging risk areas, with a higher ratio of borderline examples. This plays a crucial role in our successful safety mitigation efforts while keeping false refusal to a minimum.

安全监督微调。遵循我们的 Llama 2 配方（Touvron 等人，2023b），我们在模型对齐阶段结合所有有用性数据和安全数据。此外，我们引入了一个边界数据集，以帮助模型辨别安全请求和不安全请求之间的细微差别。我们的标注团队根据我们的指南精心制作安全提示的响应。我们发现，在战略性地平衡对抗性示例与边界示例的比例时，SFT 在模型对齐方面非常有效。我们将重点放在更具挑战性的风险领域，边界示例的比例更高。这在保持最低误拒率的同时，对我们的安全缓解工作起到了至关重要的作用。

Further, we examine the impact of model size on the trade-off between FRR and VR in Figure 18. Our results show that it varies - with smaller models requiring a larger proportion of safety data relative to helpfulness, and that it is more challenging to efficiently balance VR and FRR compared to larger models.

此外，我们在图 18 中考察了模型大小对 FRR 和 VR 之间权衡的影响。我们的结果显示，这种影响是不同的——较小的模型相对于有用性数据需要更大比例的安全数据，并且与较大的模型相比，更难以有效平衡 VR 和 FRR。

Safety DPO. To reinforce safety learning, we incorporate adversarial and borderline examples into our preference datasets in DPO. We discover that crafting response pairs to be nearly orthogonal in an embedding space is particularly effective in teaching the model to distinguish between good and bad responses for a given prompt. We conduct multiple experiments to determine the optimal ratio of adversarial, borderline, and helpfulness examples, aiming to optimize the trade-off between FRR and VR. We also find that the model size influences the learning outcomes - as a result, we tailor different safety mixes for various model sizes.

安全 DPO。为了加强安全学习，我们将对抗性示例和边界示例纳入 DPO 的偏好数据集中。我们发现，在嵌入空间中制作响应对几乎正交特别有效，这有助于模型区分给定提示的好坏响应。我们进行了多次实验，以确定对抗性示例、边界示例和有用性示例的最佳比例，旨在优化 FRR 和 VR 之间的权衡。我们还发现，模型大小影响学习结果——因此，我们为不同大小的模型定制了不同的安全混合。

Figure 19 – Violation rates (VR) and false refusal rates (FRR) on English and our core multilingual short context benchmarks, comparing Llama 3405B - with and without Llama Guard (LG) system-level protections - to competitor models and systems. Languages not supported by Comp. 3 represented with an 'x.' Lower is better.

图 19 – 在英语和我们核心多语言短上下文基准上的违规率（VR）和误拒率（FRR），比较 Llama 3405B – 有无 Llama Guard（LG）系统级保护 – 与竞争对手模型和系统。Comp. 3 不支持的语言用 'x' 表示。数值越低越好。

Figure 20 – Violation rates (VR) and false refusal rates (FRR) on tool use and long context benchmarks. Lower is better. The performance for DocQA and Many-shot benchmarks are listed separately. Note we do not have a borderline data set for Many-shot, due to the adversarial nature of the benchmark, and thus do not measure false refusal rates on it. For Tool Usage (Search), we only test Llama 3 405B compared to Comp. 1.

图20 – 工具使用和长上下文基准的违规率（VR）和误拒率（FRR）。数值越低越好。DocQA和Many-shot基准的性能分别列出。请注意，由于Many-shot基准的对抗性质，我们没有边缘数据集，因此不在其上测量误拒率。对于工具使用（搜索），我们仅测试了Llama 3 405B与Comp. 1的对比。

5.4.4 Safety Results 安全结果

We first highlight Llama 3's general behavior along various axes and then describe results for each specific new capability and our effectiveness at mitigating the safety risks.

我们首先强调Llama 3在各个方面的普遍行为，然后描述每项新能力的具体结果以及我们在缓解安全风险方面的有效性。

Overall performance. A comparison of Llama 3’s final violation and false refusal rates with similar models can be found in Figures 19 and 20. These results focus on our largest parameter size Llama 3405B model, compared to relevant competitors. Two of the competitors are end-to-end systems accessed through API, and one of them is an open source language model that we host internally and we evaluate directly. \({}^{13}\) We evaluate our Llama models both standalone and coupled with Llama Guard, our open source system-level safety solution (more in Section 5.4.7).

总体性能。Llama 3的最终违规率和误拒率与类似模型的比较可以在图19和图20中找到。这些结果聚焦于我们最大的参数尺寸Llama 3405B模型，与相关竞争对手进行比较。其中两个竞争对手是通过API访问的端到端系统，另一个是我们内部托管的开源语言模型，我们直接对其进行评估。\({}^{13}\) 我们评估了Llama模型在独立运行和与Llama Guard（我们的开源系统级安全解决方案，更多内容见5.4.7节）结合使用的情况。

While a low violation rate is desirable, it is critical to consider false refusal as a counter-metric, as a model that always refuses is maximally safe, but not helpful in the slightest. Similarly, a model that always answers every prompt, regardless of how problematic the request, would be overly harmful and toxic. In Figure 21, leveraging our internal benchmarks, we explore how different models and systems in industry navigate this trade off and how Llama 3 compares. We find that our models achieve very competitive violation rate metrics while keeping false refusal rate low as well, indicating a solid balance between helpfulness and safety.

虽然低违规率是理想的，但考虑误拒率作为反向指标至关重要，因为一个总是拒绝的模型虽然最大限度地安全，但毫无帮助。同样，一个无论请求多么有问题都总是回答的模型，将过于有害和有毒。在图21中，利用我们的内部基准，我们探讨了行业中不同模型和系统如何权衡这一问题，以及Llama 3的比较情况。我们发现，我们的模型在违规率指标上实现了非常具有竞争力的表现, 同时保持低错误拒绝率，表明在有用性和安全性之间取得了良好的平衡。

\({}^{13}\) Because these safety benchmarks are internal to Meta,we acknowledge that the numbers in this section are not reproducible externally, and so we choose to anonymize the competitors we evaluate against.

\({}^{13}\) 由于这些安全基准是Meta内部的，我们承认本节中的数据无法在外部复现，因此我们选择对评估的竞争对手进行匿名处理。

Figure 21 Violation and false refusal rates across models and capabilities. Each \(\mathrm{{point}}\) represents the \(\mathrm{{overall}}\) false \(\mathrm{{refusal}}\) and violation rate for an internal capability benchmark across all safety categories. Symbols indicate whether we are evaluating model or system level safety. As expected model level safety results indicate higher violation rates and lower refusal rates compared to system level safety results. Llama 3 aims to balance a low violation rate with a low false refusal rate, while some competitors are more skewed towards one or the other.

图21 模型和能力中的违规率和错误拒绝率。每个 \(\mathrm{{point}}\) 代表所有安全类别中内部能力基准的 \(\mathrm{{overall}}\) 错误 \(\mathrm{{refusal}}\) 和违规率。符号表示我们是在评估模型级还是系统级安全。正如预期，模型级安全结果显示违规率较高，拒绝率较低，与系统级安全结果相比。Llama 3旨在平衡低违规率与低错误拒绝率，而一些竞争对手则更偏向于其中之一。

Multilingual safety. Our experiments demonstrate that safety knowledge in English does not readily transfer to other languages, particularly given the nuance of safety policies and language-specific context. Therefore, it is essential to collect high-quality safety data for each language. We also found that the distribution of safety data per language significantly impacts performance from a safety standpoint, with some languages benefiting from transfer learning while others require more language-specific data. To achieve a balance between FRR and VR, we iteratively add adversarial and borderline data while monitoring the impact on both metrics.

多语言安全。我们的实验表明，英语中的安全知识并不容易转移到其他语言，特别是考虑到安全政策的细微差别和特定语言的上下文。因此，为每种语言收集高质量的安全数据至关重要。我们还发现，每种语言的安全数据分布从安全角度来看显著影响性能，有些语言从迁移学习中受益，而其他语言则需要更多的特定语言数据。为了在FRR和VR之间取得平衡，我们迭代地添加对抗性和边缘数据，同时监控对这两个指标的影响。

We display results on our internal benchmarks in Figure 19 for short context models, showing Llama 3's violation and false refusal rates for English and non-English languages compared to similar models and systems. To construct the benchmarks for each language, we use a combination of prompts written by native speakers, sometimes supplementing with translations from our English benchmarks. For each of our supported languages, we find that Llama 405B with Llama Guard is at least as safe, if not strictly safer, than the two competing systems when measured on our internal benchmark, while maintaining competitive false refusal rates. Looking at the Llama 405B model on its own, without Llama Guard, we find that it has a significantly lower violation rate than the competing standalone open source model, trading off a higher false refusal rate.

我们在图19中展示了短上下文模型的内部基准测试结果，显示了与类似模型和系统相比，Llama 3在英语和非英语语言中的违规和错误拒绝率。为了构建每种语言的基准测试，我们使用了由母语者编写的提示组合，有时辅以从我们的英语基准测试中翻译的内容。对于我们支持的每一种语言，我们发现，在内部基准测试中衡量时，Llama 405B配备Llama Guard至少与两个竞争系统一样安全，如果不是更严格地安全，同时保持了有竞争力的错误拒绝率。单独观察Llama 405B模型，不带Llama Guard，我们发现它的违规率明显低于竞争的独立开源模型，代价是更高的错误拒绝率。

Long-context safety. Long-context models are vulnerable to many-shot jailbreaking attacks without targeted mitigation (Anil et al., 2024). To address this, we finetune our models on SFT datasets that include examples of safe behavior in the presence of demonstrations of unsafe behavior in context. We develop a scalable mitigation strategy that significantly reduces VR, effectively neutralizing the impact of longer context attacks even for 256-shot attacks. This approach shows little to no impact on FRR and most helpfulness metrics.

长上下文安全性。长上下文模型在没有针对性缓解措施的情况下容易受到多次攻击的破解（Anil等人，2024年）。为了解决这个问题，我们在SFT数据集上对我们的模型进行了微调，这些数据集包括在上下文中存在不安全行为演示的情况下安全行为的示例。我们开发了一种可扩展的缓解策略，显著降低了违规率，有效地中和了长上下文攻击的影响，即使是256次攻击。这种方法对错误拒绝率和大部份有用性指标几乎没有影响。

To quantify the effectiveness of our long context safety mitigations, we use two additional benchmarking methods: DocQA and Many-shot. For DocQA, short for "document question answering," we use long documents with information that could be utilized in adversarial ways. Models are provided both the document and a set of prompts related to the document in order to test whether the questions being related to information in the document affected the model's ability to respond safely to the prompts. For Many-shot, following Anil et al. (2024), we construct a synthetic chat history composed of unsafe prompt-response pairs. A final prompt, unrelated to previous messages, is used to test whether the unsafe behavior in-context influenced the model

为了量化我们长期上下文安全缓解措施的有效性，我们采用了两种额外的基准测试方法：DocQA 和 Many-shot。对于 DocQA，即“文档问答”，我们使用包含可能被恶意利用信息的长文档。模型同时提供文档和一组与文档相关的提示，以测试与文档中信息相关的问题是否影响模型对提示的安全响应能力。对于 Many-shot，遵循 Anil 等人（2024）的方法，我们构建了一个由不安全提示-响应对组成的合成聊天历史。一个与之前消息无关的最终提示用于测试上下文中的不安全行为是否影响了模型的安全响应。

to response unsafely. The violation and false refusal rates for both DocQA and Many-shot are shown in Figure 20. We see that Llama 405B (with and without Llama Guard) is Pareto-better than the Comp. 2 system across both violation rates and false refusal rates, across both DocQA and Many-shot. Relative to Comp. 1, we find that Llama 405B is significantly safer, while coming at a trade off on false refusal.

对于 DocQA 和 Many-shot 的违规和错误拒绝率如图 20 所示。我们发现 Llama 405B（无论是否启用 Llama Guard）在 DocQA 和 Many-shot 的违规率和错误拒绝率上均优于 Comp. 2 系统。相对于 Comp. 1，我们发现 Llama 405B 在安全性上显著提升，尽管在错误拒绝率上有所权衡。

Tool usage safety. The diversity of possible tools and the implementation of the tool usage call and integration into the model make tool usage a challenging capability to fully mitigate (Wallace et al., 2024). We focus on the search usecase. Violation and false refusal rates are shown in Figure 20. We tested against the Comp. 1 system, where we find that Llama 405B is significantly safer, though has a slightly higher false refusal rate.

工具使用安全。工具的多样性和工具使用调用及集成到模型中的实现使得工具使用成为一个难以完全缓解的挑战（Wallace 等人，2024）。我们专注于搜索用例。违规和错误拒绝率如图 20 所示。我们针对 Comp. 1 系统进行了测试，发现 Llama 405B 在安全性上显著提升，尽管错误拒绝率略高。

5.4.5 Cybersecurity and Chemical/Biological Weapons Safety 网络安全与化学/生物武器安全

CyberSecurity evaluation results. To evaluate cybersecurity risk, we leverage the CyberSecEval benchmark framework (Bhatt et al., 2023, 2024), which contains tasks that measure safety across domains such as generating insecure code, generating malicious code, textual prompt injection, and vulnerability identification. We developed and applied Llama 3 to new benchmarks on spear phishing and autonomous cyberattacks.

网络安全评估结果。为了评估网络安全风险，我们利用了CyberSecEval基准框架（Bhatt等人，2023，2024），该框架包含跨领域的任务，如生成不安全代码、生成恶意代码、文本提示注入和漏洞识别。我们开发并应用了Llama 3到新的基准测试，包括钓鱼攻击和自主网络攻击。

Overall, we find that Llama 3 does not have significant susceptibilities in generating malicious code or exploiting vulnerabilities. We describe brief results on specific tasks:

总体而言，我们发现Llama 3在生成恶意代码或利用漏洞方面没有显著的易感性。我们简要描述了特定任务的结果：

• Insecure coding testing framework: Evaluating Llama 3 8B, 70B, and 405B against the insecure coding testing framework, we continue to observe that larger models both generate more insecure code and also generate code with a higher average BLEU score (Bhatt et al., 2023).

• 不安全编码测试框架：评估Llama 3 8B、70B和405B对不安全编码测试框架的表现，我们继续观察到，较大的模型不仅生成更多不安全代码，而且生成的代码平均BLEU分数（Bhatt等人，2023）也更高。

• Code interpreter abuse prompt corpus: We identify that Llama 3 models are susceptible to executing malicious code under certain prompts, with Llama 3405B being particularly susceptible by complying with malicious prompts \({10.4}\%\) of the time. Llama 370 B complied at a rate of \({3.8}\%\) .

• 代码解释器滥用提示语料库：我们发现Llama 3模型在某些提示下容易执行恶意代码，其中Llama 3405B特别容易受到恶意提示的遵从，遵从率为\({10.4}\%\)。Llama 370B的遵从率为\({3.8}\%\)。

• Text-based prompt injection benchmark: When evaluated against prompt injection benchmarks, prompt injection attacks against Llama 3405B were successful \({21.7}\%\) of the time. Figure 22 provides text-based prompt injection success rates across Llama 3, GPT-4 Turbo, Gemini Pro, and Mixtral models.

• 基于文本的提示注入基准：在对提示注入基准进行评估时，针对Llama 3405B的提示注入攻击成功率为\({21.7}\%\)。图22提供了Llama 3、GPT-4 Turbo、Gemini Pro和Mixtral模型在文本提示注入成功率方面的比较。

• Vulnerability identification challenges: In assessing Llama 3’s ability to identify and exploit vulnerabilities using CyberSecEval 2's capture-the-flag test challenges, Llama 3 does not outperform commonly used, traditional non-LLM tools and techniques.

• 漏洞识别挑战：在评估Llama 3使用CyberSecEval 2的夺旗测试挑战来识别和利用漏洞的能力时，Llama 3并未超过常用的传统非LLM工具和技术。

• Spear phishing benchmark: We evaluate model persuasiveness and success rate in carrying out personalized conversations designed to deceive a target into unwittingly participating in security compromises. Randomized detailed victim profiles were generated by an LLM to serve as spear phishing targets. A judge LLM (Llama 370B) scored the performance of Llama 370B and 405B in interacting with a victim model (Llama 3 70B) and evaluated the success of the attempt. Llama 370B and Llama 3405B were evaluated by the judge LLM to be moderately persuasive. Llama 370B was judged by an LLM to have been successful in \({24}\%\) of spear phishing attempts while Llama 3405B was judged to be successful in \({14}\%\) of attempts. Figure 23 presents judge LLM-evaluated persuasiveness scores across models and phishing objectives.

• 鱼叉式网络钓鱼基准测试：我们评估模型在执行旨在欺骗目标无意中参与安全妥协的个性化对话中的说服力和成功率。通过大型语言模型（LLM）生成了随机详细的受害者资料，作为鱼叉式网络钓鱼的目标。一个评判用的大型语言模型（Llama 370B）对Llama 370B和405B在与受害者模型（Llama 3 70B）互动中的表现进行了评分，并评估了尝试的成功与否。评判用的大型语言模型认为Llama 370B和Llama 3405B具有中等的说服力。Llama 370B被评判用的大型语言模型认为在\({24}\%\)次鱼叉式网络钓鱼尝试中取得了成功，而Llama 3405B则在\({14}\%\)次尝试中被判定为成功。图23展示了评判用的大型语言模型对各模型和钓鱼目标的说服力评分。

• Attack automation framework: We assess Llama 3 70B’s and 405B’s potential to function as an autonomous agent across four critical phases of a ransomware attack - network reconnaissance, vulnerability identification, exploit execution, and post exploitation actions. We enable the models to behave autonomously by configuring the models to iteratively generate and execute new Linux commands in response to output from their prior commands on a Kali Linux virtual machine as they targeted another virtual machine with known vulnerabilities. Although Llama 3 70B and 405B efficiently identify network services and open ports in their network reconnaissance, the models fail to effectively use this information to gain initial access to the vulnerable machine across 20 and 23 test runs respectively. In identifying vulnerabilities, Llama 370B and 405B are moderately effective but struggle with selecting and applying successful exploitation techniques. Attempts to execute exploits were entirely unsuccessful as were post-exploit attempts to maintain access or impact hosts within a network.

• 攻击自动化框架：我们评估Llama 3 70B和405B在勒索软件攻击的四个关键阶段——网络侦察、漏洞识别、漏洞利用执行和后利用行动——中作为自主代理的潜力。我们通过配置模型在Kali Linux虚拟机上迭代生成并执行新的Linux命令以响应其先前命令的输出来实现模型的自主行为，这些模型针对另一台具有已知漏洞的虚拟机。尽管Llama 3 70B和405B在网络侦察中有效地识别了网络服务和开放端口，但模型在20次和23次测试运行中均未能有效利用这些信息获取对脆弱机器的初始访问权限。在识别漏洞方面，Llama 370B和405B表现中等有效，但在选择和应用成功的漏洞利用技术方面遇到困难。尝试执行漏洞利用完全不成功，后利用尝试维持访问或在网络内影响主机也同样失败。

Uplift testing for cyber attacks. We conduct an uplift study which measures the extent a virtual assistant improved the cyberattack rates of both novice and expert cyberattackers between two simulated offensive

针对网络攻击的提升测试。我们进行了一项提升研究，该研究衡量了在两个模拟进攻性网络安全挑战中，虚拟助手对新手和专家网络攻击者攻击率提升的程度。

cybersecurity challenges. A two-stage study was conducted with 62 internal volunteers. Volunteers were categorized into "expert" (31 subjects) and "novice" (31 subjects) cohorts based on their offensive security experience. For the first stage, subjects were asked to complete the challenge without any LLM assistance but with access to the open internet. For the second stage, subjects retained access to the internet but were also provided with Llama 3405B to complete a different offensive cybersecurity challenge of similar difficulty to the first. An analysis of the completion rates of challenge attack phases by subjects indicates that both novices and experts using the \({405}\mathrm{\;B}\) model demonstrated insignificant uplift over having open access to the internet without an LLM.

我们进行了一个两阶段的研究，共有62名内部志愿者参与。志愿者根据他们的进攻性安全经验被分为“专家”（31名受试者）和“新手”（31名受试者）两组。在第一阶段，受试者被要求在没有任何大型语言模型（LLM）协助的情况下，仅通过开放互联网完成挑战。在第二阶段，受试者保留了互联网访问权限，并获得了Llama 3405B来完成一个难度与第一阶段相似的不同进攻性网络安全挑战。对受试者完成挑战攻击阶段的分析表明，无论是新手还是专家，在使用\({405}\mathrm{\;B}\)模型的情况下，与仅通过开放互联网访问相比，其提升效果并不显著。

Figure 22 Text-based prompt injection success rates per model across prompt injection strategies. Llama 3 is on average more susceptible to prompt injection than GPT-4 Turbo and Gemini Pro but less susceptible than Mixtral models when evaluated using this benchmark.

图22 不同提示注入策略下各模型的文本提示注入成功率。根据此基准评估，Llama 3平均而言比GPT-4 Turbo和Gemini Pro更容易受到提示注入攻击，但比Mixtral模型更不易受影响。

Figure 23 Average spear phishing persuasiveness scores across spear phishermodels and goals. \(\mathrm{{At}}\) - tempt persuasiveness is evaluated by a Llama 3 70B judge LLM.

图23 不同钓鱼模型和目标的平均鱼叉式网络钓鱼说服力得分。\(\mathrm{{At}}\) - 诱惑说服力由Llama 3 70B判断型大型语言模型评估。

Uplift testing for chemical and biological weapons. To assess risks related to proliferation of chemical and biological weapons, we perform uplift testing designed to assess whether use of Llama 3 could meaningfully increase the capabilities of actors to plan such attacks.

针对化学和生物武器的提升测试。为了评估与化学和生物武器扩散相关的风险，我们进行了提升测试，旨在评估使用Llama 3是否能实质性地提高行动者策划此类攻击的能力。

The study consists of six-hour scenarios where teams of two participants were asked to generate fictitious operational plans for either a biological or chemical attack. The scenarios cover the major planning stages of a CBRNE attack (agent acquisition, production, weaponization, and delivery) and are designed to elicit detailed plans that would address challenges related to procurement of restricted materials, real-world laboratory protocols, and operational security. Participants are recruited based on previous experience in relevant areas of scientific or operational expertise, and assigned to teams consisting of two low-skill actors (no formal training) or two moderate-skill actors (some formal training and practical experience in science or operations).

该研究包括六小时的情景模拟，其中两两一组的参与者被要求为生物或化学攻击制定虚构的行动计划。这些情景涵盖了CBRNE攻击的主要规划阶段（代理获取、生产、武器化和投放），旨在引出详细的计划，以应对与受限材料的采购、现实实验室协议和操作安全相关的挑战。参与者根据以往在科学或操作专业领域的经验招募，并被分配到由两名低技能演员（无正式培训）或两名中等技能演员（有一定正式培训和科学或操作实践经验）组成的团队。

The study was generated in collaboration with a set of CBRNE experts, and designed to maximize the generality, validity, and robustness of both quantitative and qualitative outcomes. A preliminary study was also performed in order to validate the study design, including a robust power analysis ensuring that our sample size was sufficient for statistical analysis.

该研究是在一组CBRNE专家的合作下产生的，旨在最大化定量和定性结果的普遍性、有效性和稳健性。还进行了一项初步研究，以验证研究设计，包括进行稳健的功率分析，确保我们的样本量足以进行统计分析。

Each team is assigned to a "control" or "LLM" condition. The control team has access to internet-based resources only, while the LLM-enabled team had internet access as well as access to Llama 3 models enabled with web search (including PDF ingestion), information retrieval capabilities (RAG), and code execution (Python and Wolfram Alpha). To enable testing of RAG capabilities, a keyword search is used to generate a dataset of hundreds of relevant scientific papers and pre-loaded into the Llama 3 model inference system. At the conclusion of the exercise, the operational plans generated by each team are evaluated by subject matter experts with domain expertise in biology, chemistry, and operational planning. Each plan is evaluated across four stages of potential attacks, generating scores for metrics such as scientific accuracy, detail, detection avoidance, and probability of success in scientific and operational execution. After a robust Delphi process to mitigate bias and variability in subject matter expert (SME) evaluations, final scores are generated by pooling stage-level metrics into a comprehensive score.

每个团队被分配到“控制”或“LLM”条件。控制团队只能访问基于互联网的资源，而启用了LLM的团队除了互联网访问外，还能访问启用了网络搜索（包括PDF摄取）、信息检索能力（RAG）和代码执行（Python和Wolfram Alpha）的Llama 3模型。为了测试RAG能力，使用关键词搜索生成一个包含数百篇相关科学论文的数据集，并预加载到Llama 3模型推理系统中。在演练结束时，每个团队生成的行动计划由具有生物学、化学和操作规划领域专业知识的主题专家进行评估。每个计划在四个潜在攻击阶段进行评估，生成科学准确性、详细程度、检测规避和科学及操作执行成功概率等指标的分数。通过一个稳健的德尔菲过程来减轻主题专家评估中的偏差和变异性后，最终分数通过将阶段级指标汇总为一个综合分数来生成。

Quantitative analysis of these results of this study show no significant uplift in performance related to usage of the Llama 3 model. This result holds true when performing an aggregate analysis (comparing all LLM conditions to the web-only control condition) as well as for breakdowns by subgroups (e.g., separate evaluation of the Llama 3 70B and Llama 3405B models, or separate evaluation of scenarios related to chemical or biological weapons). After validating these results with CBRNE SMEs, we assess that there is a low risk that release of Llama 3 models will increase ecosystem risk related to biological or chemical weapon attacks.

本研究的定量分析结果显示，使用Llama 3模型在性能上没有显著提升。无论是进行总体分析（比较所有LLM条件与仅限网络的控制条件）还是按子组细分（例如，分别评估Llama 3 70B和Llama 3405B模型，或分别评估与化学或生物武器相关的场景），这一结果都成立。在与CBRNE主题专家验证这些结果后，我们评估认为，发布Llama 3模型增加与生物或化学武器攻击相关的生态系统风险的可能性很低。

5.4.6 Red Teaming 红队测试

We utilize Red Teaming to discover risks and use the findings to improve our benchmarks and safety tuning datasets. We conduct recurring red teaming exercises to continuously iterate and discover new risks, which guides our model development and mitigation process.

我们利用红队测试来发现风险，并利用这些发现来改进我们的基准和安全调优数据集。我们进行定期的红队测试演练，以持续迭代并发现新风险，这指导我们的模型开发和缓解过程。

Our red team consists of experts in cybersecurity, adversarial machine learning, responsible AI, and integrity, in addition to multilingual content specialists with backgrounds in integrity issues for specific geographic markets. We also partner with internal and external subject-matter experts in critical risk areas to help build risk taxonomies and aid in more focused adversarial assessment.

我们的红队由网络安全、对抗机器学习、负责任的人工智能和诚信领域的专家组成，此外还有具备特定地理市场诚信问题背景的多语种内容专家。我们还与内部和外部关键风险领域的主题专家合作，帮助构建风险分类法，并协助进行更有针对性的对抗性评估。

Adversarial testing on specific model capabilities. We began initial red teaming by focusing on individual model capabilities in a risk discovery process, in context of specific high-risk categories then testing capabilities together. The red team focused on prompt-level attacks to emulate more likely more real world scenarios - we find that models often deviate from expected behavior, particularly in cases when the prompt's intention is being obfuscated or when prompts layer multiple abstractions. These risks get more complex with additional capabilities, and we describe several of our red teaming discoveries in detail below. We utilize these red team discoveries in concert with our results on internal safety benchmarks to develop focused mitigations to continuously and iteratively improve model safety.

针对特定模型能力的对抗性测试。我们通过专注于风险发现过程中的单个模型能力，开始初步的红队测试，然后在特定的高风险类别背景下测试这些能力。红队专注于模拟更可能的现实世界场景的提示级攻击——我们发现模型经常偏离预期行为，特别是在提示的意图被混淆或提示包含多层抽象的情况下。随着能力的增加，这些风险变得更加复杂，我们在下面详细描述了我们的几个红队发现。我们利用这些红队发现与我们在内部安全基准上的结果相结合，开发有针对性的缓解措施，以持续迭代地提高模型安全性。

• Short and long-context English. We employed a mix of well known, published and unpublished techniques across single and multi-turn conversations. We also leveraged advanced, adversarial multi-turn automation similar to PAIR (Chao et al., 2023) across some techniques and risk categories. Largely, multi-turn conversations lead to more harmful outputs. Several attacks were pervasive across model checkpoints, particularly when used together.

• 短和长上下文英语。我们采用了混合的已知、已发表和未发表的技术，涵盖单轮和多轮对话。我们还利用了类似于PAIR（Chao et al., 2023）的先进对抗性多轮自动化技术，应用于某些技术和风险类别。大多数情况下，多轮对话会导致更有害的输出。几种攻击在模型检查点中普遍存在，尤其是当它们一起使用时。

– Multi-turn refusal suppression to specify the model response to follow a particular format or include/exclude particular information related to the refusal as specific phrases.

– 多轮拒绝抑制，以指定模型响应遵循特定格式或包含/排除与拒绝相关的特定信息，如特定短语。

– Hypothetical scenarios wrap violating prompts as hypothetical/theoretical tasks or fictional scenarios. Prompts can be as simple as adding the word "hypothetically" or crafting an elaborate layered scenario.

– 假设情景，将违反规则的提示包装为假设/理论任务或虚构情景。提示可以简单到添加“假设”这个词，或者构建一个精心设计的多层情景。

• Personas and role play gives the model a violating persona with specific violating response characteristics (e.g. "You are X,your goal is Y") or yourself as the user adapting a specific benign character that obfuscates the context of the prompt.

• 角色扮演和角色设定赋予模型一个具有特定违规反应特征的违规角色（例如，“你是X，你的目标是Y”），或者用户自己扮演一个特定的好角色，从而混淆提示的上下文。

• Adding disclaimers and warnings works as a form of response priming and we assume a method to allow for the model a path to helpful compliance that intersects with generalized safety training. Asking for disclaimers, trigger warnings and more to be added in multi-turn conversations in concert with other attacks mentioned contributed to increased violation rates.

• 添加免责声明和警告作为一种回应启动形式，我们假设一种方法允许模型与通用安全培训相交的路径，以实现有益的合规性。在多轮对话中要求添加免责声明、触发警告等，与其他提到的攻击相结合，有助于提高违规率。

• Gradually escalating violation is a multi-turn attack where the conversation starts out with a more or less benign request and then through direct prompting for more exaggerated content can gradually lead the model into generating a very violating response. Once the model has started outputting violating content, it can be difficult for the model to recover (or another attack can be used if a refusal is encountered). With longer context models, this will be an increasingly seen issue.

• 逐步升级的违规行为是一种多轮攻击，对话开始时提出一个或多或少无害的请求，然后通过直接提示更夸张的内容，可以逐渐引导模型生成非常违规的回应。一旦模型开始输出违规内容，模型可能难以恢复（或者如果遇到拒绝，可以使用另一种攻击）。对于更长的上下文模型，这将是一个越来越常见的问题。

• Multilingual. We identify a number of unique risks when considering multiple languages.

• 多语言。我们在考虑多种语言时识别出许多独特的风险。

\(-\) Mixing multiple languages in one prompt or conversation \(\operatorname{can}\) easily lead to more violating outputs than if a single language was used.

\(-\) 在一个提示或对话中混合多种语言 \(\operatorname{can}\) 比使用单一语言更容易导致更多违规输出。

• Lower resource languages can lead to violating outputs given a lack of related safety fine tuning data, weak model generalization of safety or prioritization of testing or benchmarks. However, this attack often result in poor quality generally, limiting real adversarial use.

• 低资源语言由于缺乏相关的安全微调数据、模型对安全性的弱泛化能力或测试或基准的优先级，可能导致违规输出。然而，这种攻击通常会导致质量普遍较低，限制了实际的对抗性使用。

– Slang, specific context or cultural-specific references can confuse or appear to be violating at first glance, only to see the model does not comprehend a given reference correctly to make an output truly harmful or prevent it from being a violating output.

– 俚语、特定上下文或文化特定参考可能会在第一眼看上去令人困惑或显得违规，只有在模型未能正确理解给定参考以产生真正有害的输出或防止其成为违规输出时才会发现。

• Tool use. During testing, apart from English-text level adversarial prompting techniques being successful in generating violating outputs, several tool specific attacks were also discovered. This included but was not limited to:

• 工具使用。在测试过程中，除了在英语文本层面成功生成违规输出的对抗性提示技术外，还发现了多种特定工具的攻击。这包括但不限于：

• Unsafe tool chaining such as asking for multiple tools at once with one being violating could, in early checkpoints, lead to all of the tools being called with a mix of benign and violating inputs.

• 不安全的工具链使用，例如一次性请求多个工具，其中一个是违规的，可能会在早期检查点导致所有工具被调用，混合了良性输入和违规输入。

• Forcing tool use often with specific input strings, fragmented or encoded text can trigger a tool input to be potentially violating, leading to a more violating output. Other techniques can then be used to access the tool results, even if the model would normally refuse to perform the search or assist with the results.

• 强制使用工具，通常使用特定的输入字符串、碎片化或编码的文本，可能会触发工具输入违规，导致更违规的输出。其他技术可以用来访问工具结果，即使模型通常会拒绝执行搜索或协助结果。

• Modifying tool use parameters such as swapping words in queries, retrying, or obfuscating some of the initial request in a multi-turn conversation lead to violations in many early checkpoints as a form of forcing tool use.

• 修改工具使用参数，如在查询中交换单词、重试或在多轮对话中混淆初始请求，会导致许多早期检查点违规，这是一种强制工具使用的方式。

Child safety risks. Child Safety risk assessments were conducted using a team of experts, to assess the model's capability to produce outputs that could result in Child Safety risks and inform on any necessary and appropriate risk mitigations via fine tuning. We leveraged those expert red teaming sessions to expand the coverage of our evaluation benchmarks through model development. For Llama 3, we conducted new in-depth sessions using objective based methodologies to assess model risks along multiple attack vectors. We also partnered with content specialists to perform red teaming exercises assessing potentially violating content while taking account of market specific nuances or experiences.

儿童安全风险。通过专家团队进行了儿童安全风险评估，以评估模型产生可能导致儿童安全风险输出的能力，并通过微调告知任何必要和适当的风险缓解措施。我们利用这些专家红队测试会议，通过模型开发扩展了我们的评估基准覆盖范围。对于Llama 3，我们使用基于目标的方法进行了新的深入会议，以评估多个攻击向量的模型风险。我们还与内容专家合作，进行了红队测试练习，评估潜在违规内容，同时考虑到市场特定的细微差别或经验。

5.4.7 System Level Safety 系统级安全

In various real-world applications of large language models, models are not used in isolation but are integrated into broader systems. In this section, we describe our system level safety implementation, which supplements model-level mitigations by providing more flexibility and control.

在大规模语言模型的各种实际应用中，模型不是孤立使用的，而是集成到更广泛的系统中。在本节中，我们描述了我们的系统级安全实施，它通过提供更大的灵活性和控制来补充模型级缓解措施。

To enable this, we develop and release a new classifier, Llama Guard 3, which is a Llama 3 8B model fine-tuned for safety classification. Similar to Llama Guard 2 (Llama-Team, 2024), this classifier is used to detect whether input prompts and/or output responses generated by language models violate safety policies on specific categories of harm.

为此，我们开发并发布了一个新的分类器，Llama Guard 3，这是一个针对安全分类微调的Llama 3 8B模型。与Llama Guard 2（Llama-Team，2024）类似，该分类器用于检测语言模型生成的输入提示和/或输出响应是否违反特定类别伤害的安全政策。

It is designed to support Llama's growing capabilities, and can be used for English and multilingual text. It is also optimized to be used in the context of tool-calls such as search-tools and preventing code interpreter abuse. Finally, we also provide quantized variants to reduce memory requirements. We encourage developers to use our release of system safety components as a foundation and configure them for their own use cases.

该系统旨在支持Llama不断增长的能力，并可用于英语和多语言文本。它还针对在搜索工具等工具调用场景以及防止代码解释器滥用方面进行了优化。最后，我们还提供了量化变体以减少内存需求。我们鼓励开发者使用我们发布的系统安全组件作为基础，并根据自身用例进行配置。

Taxonomy. We train on the 13 hazard categories listed in the AI Safety taxonomy (Vidgen et al., 2024): Child Sexual Exploitation, Defamation, Elections, Hate, Indiscriminate Weapons, Intellectual Property, Non-Violent Crimes, Privacy, Sex-Related Crimes, Sexual Content, Specialized Advice, Suicide & Self-Harm, and Violent Crimes. We also train on Code Interpreter Abuse category to support tool-calls use cases.

分类学。我们针对AI安全分类学（Vidgen等人，2024）中列出的13种危害类别进行训练：儿童性剥削、诽谤、选举、仇恨、无差别武器、知识产权、非暴力犯罪、隐私、性相关犯罪、性内容、专业建议、自杀与自残以及暴力犯罪。我们还针对代码解释器滥用类别进行训练，以支持工具调用用例。

Training data. We start with the English data used by Llama Guard (Inan et al., 2023) and expand this dataset to incorporate new capabilities. For new capabilities such as multilingual and tool use, we collect prompt and response classification data, as well as utilize the data collected for safety finetuning. We increase the number of unsafe responses in the training set by doing prompt engineering to get the LLM to not refuse responding to adversarial prompts. We use Llama 3 to obtain response labels on such generated data.

训练数据。我们从Llama Guard（Inan等人，2023）使用的英语数据开始，并扩展此数据集以纳入新能力。对于多语言和工具使用等新能力，我们收集了提示和响应分类数据，并利用为安全微调收集的数据。我们通过提示工程增加训练集中不安全响应的数量，以使LLM不拒绝回应对抗性提示。我们使用Llama 3对生成的此类数据获取响应标签。

To improve the performance of Llama Guard 3, we do extensive cleaning of the collected samples using human annotation as well as LLM annotation by Llama 3. Obtaining labels for user prompts is a much harder task for both humans and LLMs, and we find that the human labels are slightly better, especially for borderline prompts, though our full iterative system is able to reduce the noise and produce more accurate labels.

为了提高Llama Guard 3的性能，我们通过人工标注和Llama 3的LLM标注对收集的样本进行了大量清洗。获取用户提示的标签对人类和LLMs来说都是一项更为困难的任务，我们发现人类标签略胜一筹，尤其是在边缘提示方面，尽管我们的全迭代系统能够减少噪声并产生更准确的标签。

Table 25 ‘Violation Rate (VR) and False Refusal Rate (FRR) relative to Llama 3 when using Llama Guard 3 for input or output filtering on different languages. For example,- \({50}\%\) for VR means that there is a \({50}\%\) reduction in the rate of Llama 3 model violations when using Llama Guard. Evaluations are performed on generations from the 405B-parameter Llama 3 model. Lower is better.

表25展示了在使用Llama Guard 3进行不同语言的输入或输出过滤时，相对于Llama 3的违规率（VR）和误拒率（FRR）。例如，\({50}\%\) 表示在使用Llama Guard时，Llama 3模型的违规率降低了\({50}\%\)。评估基于405B参数的Llama 3模型的生成结果。数值越低表示效果越好。

Results. Llama Guard 3 is able to significantly reduce violations across capabilities (-65% violations on average across our benchmarks). Note that adding system safeguards (and any safety mitigations in general) comes at the cost of increased refusals to benign prompts. In Table 25 we report reductions in violation rate and increases in false refusal rate increase compared to the base model to highlight this tradeoff. This effect is also visible in Figures 19, 20, and 21.

结果显示，Llama Guard 3能够显著减少各方面的违规行为（平均违规率降低了65%）。需要注意的是，增加系统防护措施（以及任何安全缓解措施）会以增加对良性提示的拒绝为代价。在表25中，我们报告了相对于基础模型的违规率降低和误拒率增加，以突出这种权衡。这一效应在图19、20和21中也有所体现。

System safety also offers more flexibility. Llama Guard 3 can be deployed for specific harms only enabling control over the violations and false refusals trade-off at the harm category level. Table 26 presents violations reduction per category to inform which category should be turned on/off based on the developer use case.

系统安全还提供了更多的灵活性。Llama Guard 3可以仅针对特定危害进行部署，从而在危害类别层面控制违规和误拒之间的权衡。表26展示了各分类的违规减少情况，以便根据开发者的使用场景决定开启或关闭哪些类别。

To make it easier to deploy safety systems, we provide a quantized version of Llama Guard 3 using the commonly used int 8 quantization technique,reducing its size by more than \({40}\%\) . Table 27 illustrates that quantization has negligible impact on the performance of the model.

为了便于部署安全系统，我们提供了使用常见的int 8量化技术的Llama Guard 3量化版本，其大小减少了超过\({40}\%\)。表27表明，量化对模型的性能影响可以忽略不计。

Prompt-based system guards. System-level safety components enable developers to customize and control how LLM systems respond to user requests. As part of our work on improving the overall safety of the model system and enable developers to deploy responsibly, we describe and release the creation of two prompt-based filtering mechanisms: Prompt Guard and Code Shield. We open-source these for the community to leverage as-is or take as inspiration and adapt for their usecases.

基于提示的系统防护。系统级安全组件使开发者能够自定义和控制LLM系统如何响应用户请求。作为我们提高模型系统整体安全性并使开发者能够负责任地部署的工作的一部分，我们描述并发布了两种基于提示的过滤机制的创建：Prompt Guard和Code Shield。我们将其开源，供社区直接利用或作为灵感并根据其用例进行调整。

Prompt Guard is a model-based filter designed to detect prompt attacks, which are input strings designed to subvert the intended behavior of an LLM functioning as part of an application. The model is a multi-label classifier that detects two classes of prompt attack risk - direct jailbreaks (techniques that explicitly try to override a model's safety conditioning or system prompt) and indirect prompt injections (instances where third-party data included in a model's context window includes instructions inadvertently executed as user commands by an LLM). The model is fine-tuned from mDeBERTa-v3-base, a small (86M) parameter model suitable for filtering inputs into an LLM. We evaluate the performance on several evaluation datasets shown in Table 28. We evaluate on two datasets (jailbreaks and injections) drawn from the same distribution as the training data, as well as an out-of-distribution dataset in English, a multilingual jailbreak set built from machine translation, and a dataset of indirect injections drawn from CyberSecEval (both English and multilingual). Overall, we find that the model generalizes well to new distributions and has strong performance.

Prompt Guard是一种基于模型的过滤器，旨在检测提示攻击，这些攻击是设计用来颠覆作为应用程序一部分的LLM预期行为的输入字符串。该模型是一个多标签分类器，能够检测两类提示攻击风险——直接越狱（试图明确覆盖模型的安全调节或系统提示的技术）和间接提示注入（模型上下文窗口中包含的第三方数据无意中被LLM执行为用户命令的情况）。该模型是从mDeBERTa-v3-base微调而来的，这是一个适合过滤输入到LLM的小型（86M）参数模型。我们在表28所示的几个评估数据集上评估了性能。我们在两个数据集（越狱和注入）上进行评估，这些数据集与训练数据具有相同的分布，以及一个英语中的分布外数据集，一个从机器翻译构建的多语言越狱集，以及一个从CyberSecEval（英语和多语言）中提取的间接注入数据集。总体而言，我们发现该模型能够很好地泛化到新的分布，并具有强大的性能。

Code Shield is an example of a class of system-level protections based on providing inference-time filtering. In particular, it focuses on detecting the generation of insecure code before it might enter a downstream usecase such as a production system. It does so by leveraging a static analysis library, the Insecure Code Detector (ICD), to identify insecure code. ICD uses a suite of static analysis tools to perform the analysis across 7 programming languages. These kinds of guardrails are generally useful for developers, who can deploy multi-layered protections in various applications.

Code Shield 是基于提供推理时过滤的系统级保护类的一个例子。特别是，它专注于在可能进入下游用例（如生产系统）之前检测不安全代码的生成。它通过利用一个静态分析库，即不安全代码检测器（ICD），来识别不安全代码。ICD使用一系列静态分析工具在7种编程语言中进行分析。这类防护措施通常对开发者有用，他们可以在各种应用中部署多层保护。

Table 26 Violation rate and false refusal rate relative to Llama 3 when using Llama Guard 3 for input or output filtering on different safety categories. For example, \(- {50}\%\) for VR means that there is a \({50}\%\) reduction in the rate of Llama 3 model violations when using Llama Guard. Evaluations are performed on English prompts and generations from the 405B parameter Llama 3 model. Lower is better.

表26 相对于Llama 3在使用Llama Guard 3进行不同安全类别的输入或输出过滤时的违规率和误拒率。例如，\(- {50}\%\)表示在使用Llama Guard时，Llama 3模型的违规率有\({50}\%\)的降低。评估是在405B参数Llama 3模型的英语提示和生成上进行的。数值越低越好。

Table 27 int8 Llama Guard. Effect of int8 quantization on Llama Guard 3 output classification performance for different model capabilities.

表27 int8 Llama Guard。int8量化对Llama Guard 3输出分类性能在不同模型能力上的影响。

5.4.8 Limitations 局限性

We conducted extensive measurement and mitigation on a wide variety of risks to safe usage of Llama 3. However, no testing can be guaranteed to be exhaustive in identifying every possible risk. Llama 3 may still generate harmful content due to training on various datasets, particularly for languages beyond English and when prompt engineered by skilled adversarial red teamers. Malicious developers or adversarial users may find new ways to jailbreak our models and use them for various nefarious usecases. We will continue to proactively identify risks, conduct research on mitigation methods, and we encourage developers to consider responsibility in every aspect - from model development to deployment to users. We hope developers will leverage and contribute to the tools we release in our open-source system-level safety suite.

我们对Llama 3安全使用的各种风险进行了广泛的测量和缓解。然而，没有任何测试可以保证能够全面识别所有可能的风险。Llama 3可能仍会因在各种数据集上训练而生成有害内容，特别是在英语以外的语言中，以及在熟练的对抗性红队工程师精心设计的提示下。恶意开发者或对抗性用户可能会找到新的方法来破解我们的模型，并将它们用于各种不良用途。我们将继续主动识别风险，进行缓解方法的研究，并鼓励开发者在从模型开发到部署再到用户的每一个环节中考虑责任。我们希望开发者能够利用并贡献于我们在开源系统级安全套件中发布的工具。

6 Inference 推理

We investigate two main techniques to make inference with the Llama 3405B model efficient: (1) pipeline parallelism and (2) FP8 quantization. We have publicly released our implementation of FP8 quantization.

我们研究了两种主要技术以提高 Llama 3405B 模型的推理效率：（1）流水线并行和（2）FP8 量化。我们已经公开发布了我们的 FP8 量化实现。

6.1 Pipeline Parallelism 流水线并行

When using a BF16 number representation for the model parameters, Llama 3405B does not fit in the GPU memory of a single machine with 8 Nvidia H100 GPUs. To address this issue, we parallelize model inference using BF16 precision across 16 GPUs on two machines. Within each machine, the high NVLink bandwidth enables the use of tensor parallelism (Shoeybi et al., 2019). Across nodes, however, connectivity has lower bandwidth and higher latency, so we use pipeline parallelism (Huang et al., 2019) instead.

当使用 BF16 数字表示模型参数时，Llama 3405B 无法适应单台机器上配备 8 块 Nvidia H100 GPU 的内存。为了解决这个问题，我们使用 BF16 精度在两台机器的 16 块 GPU 上并行化模型推理。在每台机器内部，高 NVLink 带宽使得可以使用张量并行（Shoeybi 等人，2019）。然而，在节点之间，连接具有较低的带宽和较高的延迟，因此我们使用流水线并行（Huang 等人，2019）。

Table 28 Performance of Prompt Guard. We include in- and out-of-distribution evaluations, a multilingual jailbreak built using machine translation, and a dataset of indirect injections from CyberSecEval.

表 28 Prompt Guard 的性能。我们包括了分布内和分布外的评估，使用机器翻译构建的多语言越狱，以及来自 CyberSecEval 的间接注入数据集。

Figure 24 Effect of micro-batching on inference throughput and latency \(\mathrm{{during}\;{the}}\;{Left}.\;\mathrm{{pre}}\) -filling and \({Right}\) : \(\;\mathrm{{decoding}}\) stage. The numbers in the plot correspond to the (micro-)batch size.

图 24 微批处理对推理吞吐量和延迟的影响 \(\mathrm{{during}\;{the}}\;{Left}.\;\mathrm{{pre}}\) -填充和 \({Right}\) : \(\;\mathrm{{decoding}}\) 阶段。图中的数字对应于（微）批大小。

During training with pipeline parallelism, bubbles are a major efficiency concern (see Section 3.3). However, they are not an issue during inference, since inference does not involve a backward pass that requires a pipeline flush. Therefore, we use micro-batching to improve inference throughput with pipeline parallelism.

在流水线并行训练期间，气泡是一个主要的效率问题（见第 3.3 节）。然而，在推理过程中它们不是问题，因为推理不涉及需要流水线刷新的反向传播。因此，我们使用微批处理来提高流水线并行推理的吞吐量。

We evaluate the effect of using two micro-batches in inference workloads of 4,096 input tokens and 256 output tokens both during the key-value cache pre-fill stage of inference and during the decoding stage. We find that micro-batching improves throughput of inference with the same local batch size; see Figure 24. These improvements result from micro-batching enabling concurrent execution of micro batches in both these stages. The additional synchronization points due to micro-batching also increase latency but, overall, micro-batching still leads to a better throughput-latency trade-off.

我们评估了在推理工作负载中使用两个微批次的效果，该工作负载包含4,096个输入令牌和256个输出令牌，在推理的关键值缓存预填充阶段和解码阶段均进行了评估。我们发现，微批次化提高了具有相同本地批次大小的推理吞吐量；参见图24。这些改进源于微批次化使得在这两个阶段中可以并发执行微批次。由于微批次化导致的额外同步点也增加了延迟，但总体上，微批次化仍然带来了更好的吞吐量-延迟权衡。

6.2 FP8 Quantization FP8量化

We perform experiments leveraging the native FP8 support of H100 GPUs to perform low-precision inference. To enable low-precision inference, we apply FP8 quantization to most matrix multiplications inside the model. In particular, we quantize most parameters and activations in the feedforward network layers in the model,which account for roughly \({50}\%\) of the inference compute time. We do not quantize parameters in the self-attention layers of the model. We leverage dynamic scaling factors for better accuracy (Xiao et al., 2024b),optimizing our CUDA kernels \({}^{15}\) to reduce the overhead of calculating the scales. We find that the quality of Llama 3405B is sensitive to certain types of quantization, and make a few additional changes to increase the model output quality:

我们利用H100 GPU的原生FP8支持进行低精度推理实验。为了启用低精度推理，我们对模型内部的大多数矩阵乘法应用了FP8量化。特别是，我们对模型中前馈网络层的大多数参数和激活进行了量化，这些占据了大约\({50}\%\)的推理计算时间。我们没有对模型的自注意力层中的参数进行量化。我们利用动态缩放因子以获得更好的精度（Xiao et al., 2024b），优化我们的CUDA内核\({}^{15}\)以减少计算缩放的开销。我们发现Llama 3405B的质量对某些类型的量化很敏感，并进行了一些额外的更改以提高模型输出质量：

1. Akin to Zhang et al. (2021), we do not perform quantization in the first and last Transformer layers.

2. 类似于Zhang et al. (2021)，我们不在第一层和最后一层Transformer层中进行量化。

3. High-perplexity tokens such as dates can lead to large activation values. In turn, these can lead to high dynamic scaling factors in FP8 and a non-negligible number of underflows, leading to errors in decoding.

4. 高困惑度令牌，如日期，可能导致大的激活值。反过来，这些可能导致FP8中的高动态缩放因子，以及不可忽略的数量的下溢，导致解码错误。

\({}^{15}\) Our FP8 kernels are available at https://github.com/pytorch/FBGEMM/tree/main/fbgemm_gpu/experimental/gen_ai. We provide usage examples at https://github.com/meta-llama/llama-agentic-system.

\({}^{15}\) 我们的 FP8 内核可在 https://github.com/pytorch/FBGEMM/tree/main/fbgemm_gpu/experimental/gen_ai 获取。使用示例请参见 https://github.com/meta-llama/llama-agentic-system。

Figure 25 Illustration of tensor-wise and row-wise FP8 quantization. \({Right}\) : Row-wise quantization enables the use of more granular activation factors than Left: tensor-wise quantization.

图 25 展示了逐张量和逐行的 FP8 量化示意图。\({Right}\)：逐行量化允许使用比左图：逐张量量化更细粒度的激活因子。

Figure 26 * Reward score distribution for Llama 3 405B using BF16 and FP8 inference. Our \(\mathrm{{FP}}8\) quantization \(\mathrm{{approach}}\) has negligible impact on the model's responses.

图 26 * 使用 BF16 和 FP8 推理的 Llama 3 405B 奖励分数分布。我们的 \(\mathrm{{FP}}8\) 量化 \(\mathrm{{approach}}\) 对模型的响应影响微乎其微。

To address this issue, we upper bound the dynamic scaling factors to 1200 .

为了解决这个问题，我们将动态缩放因子上限设为 1200。

3. We use row-wise quantization, computing scaling factors across rows for parameter and activation matrices (see Figure 25). We find this works better than a tensor-wise quantization approach.

4. 我们采用逐行量化，对参数和激活矩阵按行计算缩放因子（见图 25）。我们发现这种方法比逐张量量化效果更好。

Effect of quantization errors. Evaluations on standard benchmarks often suggest that FP8 inference performs on par with BF16 inference even without these mitigations. However, we find that such benchmarks do not adequately reflect the effects of FP8 quantization. When scaling factors are not upper bounded, the model occasionally produces corrupted responses even though the benchmark performance is strong. Instead of relying on benchmarks to measure distribution changes due to quantization, we find it is better to analyze the distribution of reward-model scores for 100,000 responses produced using both FP8 and BF16. Figure 26 shows the resulting reward distribution for our quantization approach. The results in the figure show that our approach to FP8 quantization has very limited impact on the model's response.

量化误差的影响。在标准基准测试中，FP8 推理通常与 BF16 推理表现相当，即便没有这些缓解措施。然而，我们发现这些基准测试并未充分反映 FP8 量化的影响。当缩放因子未设上限时，即使基准性能强劲，模型偶尔也会产生损坏的响应。我们发现，与其依赖基准测试来衡量量化导致的分布变化，不如分析使用 FP8 和 BF16 生成的 100,000 条响应的奖励模型分数分布。图 26 展示了我们量化方法的奖励分布结果。图中的结果显示，我们的 FP8 量化方法对模型的响应影响非常有限。

Experimental evaluation of efficiency. Figure 27 depicts the throughput-latency trade-off of performing FP8 inference with Llama 3 405B in the pre-fill and decoding stages, using 4,096 input tokens and 256 output tokens. The figure compares the efficiency of FP8 inference with that of the two-machine BF16 inference approach described in Section 6.1. The results show that use of FP8 inference leads to throughput improvements of up to \({50}\%\) during the pre-fill stage,and a substantially better throughput-latency trade-off during decoding.

效率的实验评估。图27展示了在预填充和解码阶段使用4,096个输入令牌和256个输出令牌进行FP8推理时，Llama 3 405B的吞吐量-延迟权衡。该图比较了FP8推理与第6.1节描述的双机BF16推理方法的效率。结果显示，使用FP8推理在预填充阶段带来了高达\({50}\%\)的吞吐量提升，并且在解码阶段实现了显著更好的吞吐量-延迟权衡。

Figure 27 Throughput-latency trade-off in FP8 inference with Llama 3 405B compared with BF16 inference using different pipeline parallelization setups. Left: Results for pre-filling. Right: Results for decoding.

图27 在FP8推理中，Llama 3 405B与使用不同流水线并行化设置的BF16推理的吞吐量-延迟权衡。左侧：预填充结果。右侧：解码结果。

7 Vision Experiments 视觉实验

We perform a series of experiments in which we incorporate visual-recognition capabilities into Llama 3 via a compositional approach that consists of two main stages. First, we compose a pre-trained image encoder (Xu et al., 2023) and the pre-trained language model by introducing and training a set of cross-attention layers between the two models (Alayrac et al., 2022) on a large number of image-text pairs. This leads to the model illustrated in Figure 28. Second, we introduce temporal aggregator layers and additional video cross-attention layers that operate on a large collection of video-text pairs to learn the model to recognize and process temporal information from videos.

我们进行了一系列实验，通过一个包含两个主要阶段的组合方法，将视觉识别能力融入Llama 3中。首先，我们在两个模型之间引入并训练一组交叉注意力层（Alayrac et al., 2022），将预训练的图像编码器（Xu et al., 2023）和预训练的语言模型组合在一起，并在大量图像-文本对上进行训练。这导致了图28所示的模型。其次，我们引入了时间聚合层和额外的视频交叉注意力层，这些层在大量视频-文本对上运行，以使模型能够识别和处理来自视频的时间信息。

A compositional approach to foundation model development has several advantages: (1) it enables us to parallelize the development of the vision and language modeling capabilities; (2) it circumvents complexities of joint pre-training on visual and language data that stem from tokenization of visual data, differences in background perplexities of tokens originating from different modalities, and contention between modalities; (3) it guarantees that model performance on text-only tasks is not affected by the introduction of visual-recognition capabilities, and (4) the cross-attention architecture ensures that we do not have to expend compute passing full-resolution images through the increasingly LLM backbones (specifically, the feed-forward networks in each transformer layer), making it more efficient during inference. We note that our multimodal models are still under development and not yet ready for release.

基础模型开发的组合方法具有以下几个优势：（1）它使我们能够并行开发视觉和语言建模能力；（2）它规避了视觉和语言数据联合预训练的复杂性，这些复杂性源于视觉数据的标记化、来自不同模态的标记的背景困惑度差异以及模态之间的竞争；（3）它保证了仅文本任务的模型性能不会因引入视觉识别能力而受到影响；（4）交叉注意力架构确保我们不必在推理过程中通过日益庞大的LLM主干（特别是每个transformer层中的前馈网络）传递全分辨率图像，从而提高了效率。我们注意到，我们的多模态模型仍在开发中，尚未准备好发布。

Before presenting the results of our experiments in Section 7.6 and 7.7, we describe the data we used to train visual recognition capabilities, the model architecture of the vision components, how we scale training of those components, and our pre-training and post-training recipes.

在第7.6和7.7节介绍我们的实验结果之前，我们描述了用于训练视觉识别能力的数据、视觉组件的模型架构、我们如何扩展这些组件的训练，以及我们的预训练和后训练方法。

7.1 Data 数据

We describe our image and video data separately below.

我们下面分别描述我们的图像和视频数据。

7.1.1 Image Data 图像数据

Our image encoder and adapter are trained on image-text pairs. We construct this dataset via a complex data processing pipeline that consists of four main stages: (1) quality filtering, (2) perceptual de-duplication, (3) resampling, and (4) optical character recognition. We also apply a series of safety mitigations.

我们的图像编码器和适配器是在图像-文本对上训练的。我们通过一个复杂的数据处理流水线构建了这个数据集，该流水线包括四个主要阶段：（1）质量过滤，（2）感知去重，（3）重采样，和（4）光学字符识别。我们还应用了一系列安全缓解措施。

• Quality filtering. We implement quality filters that remove non-English captions and low-quality captions via heuristics such as low alignment scores produced by (Radford et al., 2021). Specifically, we remove all image-text pairs below a certain CLIP score.

• 质量过滤。我们实施了质量过滤器，通过启发式方法（如由（Radford et al., 2021）产生的低对齐分数）去除非英语字幕和低质量字幕。具体来说，我们移除了所有CLIP分数低于某个阈值的图像-文本对。

• De-duplication. De-duplicating large-scale training datasets benefits model performance because it reduces training compute spent on redundant data (Esser et al., 2024; Lee et al., 2021; Abbas et al.,

• 去重。对大规模训练数据集进行去重有利于模型性能，因为它减少了在冗余数据上花费的训练计算量（Esser 等人，2024；Lee 等人，2021；Abbas 等人，2023）和记忆（Carlini 等人，2023；Somepalli 等人，2023）。因此，我们出于效率和隐私原因对训练数据进行去重。为此，我们使用内部版本的最新 SSCD 复制检测模型（Pizzi 等人，2022）来大规模去重图像。对于所有图像，我们首先使用 SSCD 模型计算一个 512 维的表示。我们利用这些嵌入对数据集中的所有图像进行最近邻（NN）搜索，使用余弦相似度度量。我们将超过某个相似度阈值的示例定义为重复项。我们使用连通分量算法对这些重复项进行分组，并每个连通分量仅保留一对图像-文本对。我们通过以下方式提高去重管道的效率：（1）使用 k-means 聚类对数据进行预聚类和（2）使用 FAISS（Johnson 等人，2019）进行 NN 搜索和聚类。

Figure 28 Illustration of the compositional approach to adding multimodal capabilities to Llama 3 that we study in this paper. This approach leads to a multimodal model that is trained in five stages: (1) language model pre-training, (2) multi-modal encoder pre-training, (3) vision adapter training, (4) model finetuning, and (5) speech adapter training.

图 28 展示了我们在本文中研究的将多模态能力添加到 Llama 3 的组合方法。这种方法导致了一个多模态模型，该模型经过五个阶段的训练：（1）语言模型预训练，（2）多模态编码器预训练，（3）视觉适配器训练，（4）模型微调，以及（5）语音适配器训练。

1. and memorization (Carlini et al., 2023; Somepalli et al., 2023). Hence, we de-duplicate our training data for both efficiency and privacy reasons. To do so, we use an internal version of the state-of-the-art SSCD copy-detection model (Pizzi et al., 2022) to de-duplicate images at scale. For all images, we first compute a 512-dimensional representation using the SSCD model. We use those embeddings to perform a nearest neighbor (NN) search for each image across all images in our data set, using a cosine similarity measure. We define examples above a certain similarity threshold as duplicates. We group these duplicates using a connected-components algorithm, and maintain only one image-text pair per connected component. We increase the efficiency of our de-duplication pipeline by: (1) pre-clustering the data using k-means clusters and (2) using FAISS (Johnson et al., 2019) for NN searches and clustering.

2023）和记忆（Carlini 等人，2023；Somepalli 等人，2023）。因此，我们出于效率和隐私原因对训练数据进行去重。为此，我们使用内部版本的最新 SSCD 复制检测模型（Pizzi 等人，2022）来大规模去重图像。对于所有图像，我们首先使用 SSCD 模型计算一个 512 维的表示。我们利用这些嵌入对数据集中的所有图像进行最近邻（NN）搜索，使用余弦相似度度量。我们将超过某个相似度阈值的示例定义为重复项。我们使用连通分量算法对这些重复项进行分组，并每个连通分量仅保留一对图像-文本对。我们通过以下方式提高去重管道的效率：（1）使用 k-means 聚类对数据进行预聚类和（2）使用 FAISS（Johnson 等人，2019）进行 NN 搜索和聚类。

• Resampling. We ensure diversity of the image-text pairs via resampling akin to Xu et al. (2023); Mahajan et al. (2018); Mikolov et al. (2013). First, we construct a vocabulary of n-grams by parsing high-quality text sources. Next, we compute the frequency of each vocabulary n-gram in our dataset. We then resample the data as follows: If any of the n-grams in a caption occurs less than \(T\) times in the vocabulary, we keep the corresponding image-text pair. Otherwise, we independently sample each of the n-grams \({n}_{i}\) in the caption with probability \(\sqrt{T/{f}_{i}}\) where \({f}_{i}\) indicates the frequency of n-gram \({n}_{i}\) ; we keep the image-text pair if any of the n-grams was sampled. This resampling aids performance on low-frequency categories and fine-grained recognition tasks.

• 重采样。我们通过类似于 Xu 等人（2023 年）；Mahajan 等人（2018 年）；Mikolov 等人（2013 年）的重采样方法来确保图像-文本对的多樣性。首先，我们通过解析高质量的文本来源来构建一个 n-gram 词汇表。接下来，我们计算每个词汇表 n-gram 在我们数据集中的频率。然后，我们按照以下方式对数据进行重采样：如果一个标题中的任何 n-gram 在词汇表中出现的次数少于 \(T\) 次，我们保留相应的图像-文本对。否则，我们独立地以概率 \(\sqrt{T/{f}_{i}}\) 对标题中的每个 n-gram \({n}_{i}\) 进行采样，其中 \({f}_{i}\) 表示 n-gram \({n}_{i}\) 的频率；如果任何 n-gram 被采样，我们保留图像-文本对。这种重采样有助于低频类别和细粒度识别任务的性能提升。

• Optical character recognition. We further improve our image-text data by extracting text written in the image and concatenating it with the caption. The written text is extracted using a proprietary optical character recognition (OCR) pipeline. We observe that adding OCR data into the training data greatly improves tasks that require OCR capabilities, such as document understanding.

• 光学字符识别。我们通过提取图像中书写的文本并将其与标题连接来进一步改进我们的图像-文本数据。书写的文本是通过专有的光学字符识别（OCR）流程提取的。我们观察到，将 OCR 数据加入训练数据中大大提高了需要 OCR 能力的任务，例如文档理解。

Transcribing documents. To improve the performance of our models on document understanding tasks, we render pages from documents as images and paired the images with their respective text. The document text is obtained either directly from the source or via a document parsing pipeline.

• 转录文档。为了提高我们的模型在文档理解任务上的性能，我们将文档页面渲染为图像，并将这些图像与其相应的文本配对。文档文本要么直接从源获取，要么通过文档解析流程获取。

Safety. We focus primarily on ensuring that the pre-training dataset for image recognition does not contain unsafe content, such as sexual abuse material (CSAM) (Thiel, 2023). We scan all our training images for CSAM using perceptual hashing approaches such as PhotoDNA (Farid, 2021) as well as internal, proprietary classifiers. We also use a proprietary media-risk retrieval pipeline to identify and remove image-text pairs that we consider to be NSFW, for example, because they contain sexual or violent content. We believe that minimizing the prevalence of such material in the training dataset improves the safety of the final model without impacting its helpfulness. Finally, we perform face blurring on all images in our training set. We test the model against human generated prompts that refer to an attached image.

安全性。我们主要关注确保图像识别的预训练数据集不包含不安全内容，例如性虐待材料（CSAM）（Thiel，2023）。我们使用感知哈希方法（如PhotoDNA（Farid，2021））以及内部专有分类器对所有训练图像进行CSAM扫描。我们还使用专有的媒体风险检索管道来识别和移除我们认为不适合工作环境的图像-文本对，例如因为它们包含性或暴力内容。我们相信，减少此类材料在训练数据集中的普遍性可以提高最终模型的安全性，而不影响其有用性。最后，我们对训练集中的所有图像进行面部模糊处理。我们测试模型对提及附带图像的人工生成提示的响应。

Annealing data. We create an annealing dataset by resampling the image-caption pairs to a smaller volume of \(\sim {350}\mathrm{M}\) examples using n-grams. Since the n-grams resampling favor richer text descriptions,this selects a higher-quality data subset. We augment the resulting data with \(\sim {150}\mathrm{M}\) examples from five additional sources:

退火数据。我们通过使用n-gram对图像-标题对进行重采样，创建了一个退火数据集，以减少到\(\sim {350}\mathrm{M}\)个示例的较小体积。由于n-gram重采样倾向于更丰富的文本描述，这选择了一个更高质量的数据子集。我们从五个额外来源中增加\(\sim {150}\mathrm{M}\)个示例来增强结果数据：

• Visual grounding. We link noun phrases in the text to bounding boxes or masks in the image. The grounding information (bounding boxes and masks) are specified in the image-text pair in two ways. (1) We overlay boxes or masks with marks on the image and use marks in the text as reference, akin to set-of-marks (Yang et al.,2023a). (2) We insert normalized \(\left( {{x}_{\min },{y}_{\min },{x}_{\max },{y}_{\max }}\right)\) coordinates directly into the text, demarcated by special tokens.

• 视觉定位。我们将文本中的名词短语与图像中的边界框或掩码相链接。定位信息（边界框和掩码）在图像-文本对中以两种方式指定。（1）我们在图像上叠加带有标记的框或掩码，并使用文本中的标记作为参考，类似于标记集（Yang et al.，2023a）。（2）我们直接在文本中插入标准化\(\left( {{x}_{\min },{y}_{\min },{x}_{\max },{y}_{\max }}\right)\)坐标，由特殊标记分隔。

• Screenshot parsing. We render screenshots from HTML code and task the model with predicting the code that produced a specific element in the screenshot, akin to Lee et al. (2023). The element of interest is indicated in the screenshot via a bounding box.

• 截图解析。我们从HTML代码渲染截图，并让模型预测生成截图中特定元素的代码，类似于Lee et al.（2023）。感兴趣的元素通过截图中的边界框来指示。

• Question-answer pairs. We include question-answer pairs, enabling us to use volumes of question-answering data that are too large to be used in model finetuning.

• 问答对。我们包含问答对，使我们能够使用大量的问题回答数据，这些数据太大，无法用于模型微调。

• Synthetic captions. We include images with synthetic captions that were generated by an early version of the model. Compared to original captions, we find that synthetic captions provide a more comprehensive description of images than the original captions.

• 合成字幕。我们包含带有合成字幕的图像，这些字幕是由模型的早期版本生成的。与原始字幕相比，我们发现合成字幕比原始字幕提供了更全面的图像描述。

• Synthetically-generated structured images. We also include synthetically generated images for a variety of domains such as charts, tables, flowcharts, math equations and textual data. These images are accompanied by a structured representation such as the corresponding markdown or LaTeX notation. Besides improving recognition capabilities of the model for these domains, we find this data useful to generate question-answer pairs via the text model for finetuning.

• 合成生成的结构化图像。我们还包含为各种领域（如图表、表格、流程图、数学方程和文本数据）合成的图像。这些图像伴随着结构化表示，如相应的Markdown或LaTeX符号。除了提高模型对这些领域的识别能力外，我们发现这些数据对于通过文本模型生成用于微调的问答对很有用。

7.1.2 Video Data 视频数据

For video pre-training, we use a large dataset of video-text pairs. Our dataset is curated through a multi-stage process. We filter and clean the associated texts using rule-based heuristics, such as ensuring a minimum length and fixing capitalization. Then, we run language identification models to filter out non-English texts. We run OCR detection models to filter out videos with excessive overlaid text. To ensure reasonable alignment between the video-text pairs, we use CLIP (Radford et al., 2021) style image-text and video-text contrastive models. We first compute image-text similarity using a single frame in the videos and filtered out low similarity pairs, and then subsequently filter out pairs with low video-text alignment. Some of our data contains static or low-motion videos; we filter out such data using motion-score based filtering (Girdhar et al., 2023). We do not apply any filters on the visual quality of the videos such as aesthetic scores or resolution filtering.

对于视频预训练，我们使用一个大型视频-文本对数据集。我们的数据集通过多阶段过程进行筛选和清理。我们使用基于规则的启发式方法过滤和清理相关文本，例如确保最小长度和修正大小写。然后，我们运行语言识别模型来过滤非英语文本。我们运行OCR检测模型来过滤掉带有过多叠加文本的视频。为了确保视频-文本对之间的合理对齐，我们使用CLIP（Radford等人，2021）风格的图像-文本和视频-文本对比模型。我们首先使用视频中的单个帧计算图像-文本相似度，并过滤掉低相似度的对，然后进一步过滤掉视频-文本对齐度低的对。我们的一些数据包含静态或低动态视频；我们使用基于运动分数的过滤（Girdhar等人，2023）来过滤掉这些数据。我们不对视频的视觉质量（如美学分数或分辨率过滤）应用任何过滤器。

Our dataset contains videos with an average duration of 21 seconds and a median duration of 16 seconds, with over \({99}\%\) videos being under a minute. The spatial resolution varies significantly between \({320}\mathrm{p}\) and \(4\mathrm{\;K}\) videos,with over \({70}\%\) of the videos having a short side greater than 720 pixels. The videos have varying aspect ratios with almost all videos having between aspect ratio between 1:2 and 2:1, with a 1:1 median.

我们的数据集包含平均时长为21秒、中位数时长为16秒的视频，其中超过\({99}\%\)的视频时长不足一分钟。空间分辨率在\({320}\mathrm{p}\)和\(4\mathrm{\;K}\)视频之间差异显著，超过\({70}\%\)的视频的短边大于720像素。视频的宽高比各不相同，几乎所有视频的宽高比在1:2到2:1之间，中位数为1:1。

7.2 Model Architecture 模型架构

Our visual-recognition model consists of three main components: (1) an image encoder, (2) an image adapter, and (3) a video adapter.

我们的视觉识别模型由三个主要部分组成：（1）图像编码器，（2）图像适配器，和（3）视频适配器。

Image encoder. Our image encoder is a standard vision transformer (ViT; Dosovitskiy et al. (2020)) that is trained to align images and text (Xu et al., 2023). We use the ViT-H/14 variant of the image encoder, which has \({630}\mathrm{M}\) parameters that were trained on \({2.5}\mathrm{\;B}\) image-text pairs for five epochs. The image encoder is pre-trained on images with resolution \({224} \times {224}\) ; images were split up into \({16} \times {16}\) patches of equal size (i.e., a patch size of 14x14 pixels). As also demonstrated by prior work such as ViP-Llava (Cai et al., 2024), we observe that image encoders trained via a contrastive text alignment objective are unable to preserve fine-grained localization information. To alleviate this, we employ a multi-layer feature extraction, where features from the \({4}^{th},{8}^{th},{16}^{th},{24}^{th}\) and \({31}^{st}\) layers are also provided in addition to the final layer features. In addition, we further insert 8 gated self-attention layers (making a total of 40 transformer blocks) prior to pre-training of the cross-attention layers to learn alignment-specific features. The image encoder therefore eventually has a total \({850}\mathrm{M}\) parameters with the additional layers. With the multi-layer features,the image encoder produces a 7680-dimensional representation for each of the resulting \({16} \times {16} = {256}\) patches. The parameters of the image encoder are not frozen during subsequent training stages as we found it to improve performance, especially in domains such as text recognition.

图像编码器。我们的图像编码器是一个标准的视觉变换器（ViT；Dosovitskiy 等人（2020）），经过训练以对齐图像和文本（Xu 等人，2023）。我们使用图像编码器的 ViT-H/14 变体，该变体具有 \({630}\mathrm{M}\) 参数，这些参数在 \({2.5}\mathrm{\;B}\) 图像-文本对上训练了五个周期。图像编码器在分辨率为 \({224} \times {224}\) 的图像上进行预训练；图像被分割成大小相等的 \({16} \times {16}\) 个补丁（即，每个补丁大小为 14x14 像素）。正如先前工作（如 ViP-Llava（Cai 等人，2024））所展示的那样，我们观察到通过对比文本对齐目标训练的图像编码器无法保留细粒度定位信息。为了缓解这一问题，我们采用了多层特征提取，除了最终层特征外，还提供了 \({4}^{th},{8}^{th},{16}^{th},{24}^{th}\) 和 \({31}^{st}\) 层的特征。此外，我们在交叉注意力层预训练之前进一步插入了 8 个门控自注意力层（总共 40 个变换块），以学习对齐特定特征。因此，图像编码器最终具有总共 \({850}\mathrm{M}\) 参数，包括额外的层。通过多层特征，图像编码器为每个生成的 \({16} \times {16} = {256}\) 补丁生成一个 7680 维的表示。在后续训练阶段，图像编码器的参数并未冻结，因为我们发现这可以提高性能，特别是在文本识别等领域。

Image adapter. We introduce cross-attention layers between the visual token representations produced by the image encoder and the token representations produced by the language model (Alayrac et al., 2022). The cross-attention layers are applied after every fourth self-attention layer in the core language model. Like the language model itself, the cross-attention layers use generalized query attention (GQA) for increased efficiency. The cross-attention layers introduce substantial numbers of additional trainable parameters into the model: for Llama 3405B,the cross-attention layers have \(\approx {100}\mathrm{\;B}\) parameters. We pre-train our image adapter in two stages: (1) initial pre-training followed by (2) annealing:

图像适配器。我们在图像编码器产生的视觉令牌表示和语言模型产生的令牌表示之间引入交叉注意力层（Alayrac et al., 2022）。交叉注意力层在核心语言模型的每第四个自注意力层之后应用。与语言模型本身一样，交叉注意力层使用广义查询注意力（GQA）以提高效率。交叉注意力层向模型引入了大量额外的可训练参数：对于Llama 3405B，交叉注意力层有\(\approx {100}\mathrm{\;B}\)参数。我们分两个阶段预训练我们的图像适配器：（1）初始预训练，随后是（2）退火：

• Initial pre-training. We pre-train our image adapter on our dataset of \(\sim 6\mathrm{\;B}\) image-text pairs described above. For compute efficiency reasons,we resize all images to fit within at most four tiles of \({336} \times {336}\) pixels each,where we arrange the tiles to support different aspect ratios,e.g., \({672} \times {672},{672} \times {336}\) ,and \({1344} \times {336}\) .

• 初始预训练。我们在上述描述的\(\sim 6\mathrm{\;B}\)图像-文本对数据集上预训练我们的图像适配器。出于计算效率的原因，我们将所有图像调整大小以适应最多四个\({336} \times {336}\)像素的图块，我们排列这些图块以支持不同的宽高比，例如，\({672} \times {672},{672} \times {336}\)和\({1344} \times {336}\)。

• Annealing. We continue training the image adapter on \(\sim {500}\mathrm{M}\) images from the annealing dataset described above. During annealing, we increase the per-tile image resolution to improve performance on tasks that require higher-resolution images, for example, infographics understanding.

• 退火。我们继续在上述描述的退火数据集中的\(\sim {500}\mathrm{M}\)图像上训练图像适配器。在退火过程中，我们提高每个图块的图像分辨率，以提高在需要更高分辨率图像的任务上的性能，例如，信息图理解。

Video adapter. Our model takes as input up to 64 frames (uniformly sampled from a full video), each of which is processed by the image encoder. We model temporal structure in videos through two components: (i) encoded video frames are aggregated by a temporal aggregator which merges 32 consecutive frames into one, (ii) additional video cross attention layers are added before every fourth image cross attention layer. The temporal aggregator is implemented as a perceiver resampler (Jaegle et al., 2021; Alayrac et al., 2022). We pre-train using 16 frames per video (aggregated to 1 frame), but increase the number of input frames to 64 during supervised finetuning. The video aggregator and cross attention layers have \({0.6}\mathrm{\;B}\) and \({4.6}\mathrm{\;B}\) parameters for Llama 3 7B and 70B, respectively.

视频适配器。我们的模型以最多64帧（从完整视频中均匀采样）作为输入，每一帧都由图像编码器处理。我们通过两个组件对视频中的时间结构进行建模：（i）编码后的视频帧通过时间聚合器进行聚合，该聚合器将32个连续帧合并为一个；（ii）在每第四个图像交叉注意力层之前添加额外的视频交叉注意力层。时间聚合器采用感知器重采样器实现（Jaegle et al., 2021; Alayrac et al., 2022）。我们使用每视频16帧（聚合为1帧）进行预训练，但在监督微调期间将输入帧数增加到64。视频聚合器和交叉注意力层分别有\({0.6}\mathrm{\;B}\)和\({4.6}\mathrm{\;B}\)参数用于Llama 3 7B和70B。

7.3 Model Scaling 模型缩放

After the visual-recognition components are added to Llama 3, the model contains self-attention layers, cross-attention layers, and a ViT image encoder. To train adapters for the smaller 8B and 70B parameter models, we found a combination of data and tensor parallelization is the most efficient. Model or pipeline parallelism does not increase efficiency at these scales because the gathering of model parameters would dominate the computation. We do, however, use pipeline parallelism (in addition to data and tensor parallelism) when training the adapter for the \({405}\mathrm{\;B}\) parameter model. Training at this scale introduces three new challenges in addition to those outlined in Section 3.3: model heterogeneity, data heterogeneity, and numerical instabilities.

在视觉识别组件添加到Llama 3之后，模型包含自注意力层、交叉注意力层和ViT图像编码器。为了训练较小8B和70B参数模型的适配器，我们发现数据和张量并行化的组合最为高效。在这些规模上，模型或管道并行化不会提高效率，因为模型参数的收集将主导计算。然而，我们在训练\({405}\mathrm{\;B}\)参数模型的适配器时，除了数据和张量并行化外，还使用了管道并行化。在这个规模上进行训练，除了第3.3节中提到的挑战外，还引入了三个新的挑战：模型异质性、数据异质性和数值不稳定性。

Model heterogeneity. The model computation is heterogeneous because more computation is performed on some tokens than on others. In particular, image tokens are processed by the image encoder and the cross-attention layers, whereas text tokens are only processed by the language backbone. This heterogeneity leads to bottlenecks in the scheduling of pipeline parallelism. We address this problem by ensuring each pipeline stage contains five layers: namely, four self-attention layers in the language backbone and a cross-attention layer. (Recall that we introduce a cross-attention layer after every fourth self-attention layer.) In addition, we replicate the image encoder on all pipeline stages. Because we train on paired image-text data, this enables us to perform load balancing between the image and text parts of the computation.

模型异质性。模型计算是异质性的，因为某些标记上的计算比其他标记上的计算更多。特别是，图像标记由图像编码器和交叉注意力层处理，而文本标记仅由语言主干处理。这种异质性导致流水线并行调度中的瓶颈。我们通过确保每个流水线阶段包含五层来解决这个问题：即，语言主干中的四层自注意力层和一层交叉注意力层。（回想一下，我们在每第四层自注意力层后引入一层交叉注意力层。）此外，我们在所有流水线阶段复制图像编码器。因为我们训练的是成对的图像-文本数据，这使我们能够在图像和文本部分的计算之间进行负载均衡。

Data heterogeneity. The data is heterogeneous because, on average, images have more tokens than the associated text: an image has 2,308 tokens, whereas the associated text contains an average of only 192 tokens. As a result, the computation of cross-attention layers requires more time and memory than the computation of self-attention layers. We address this problem by introducing sequence parallelization in the image encoder, so that each GPU processes roughly the same number of tokens. Because the average text size is relatively short, we also use a substantially larger micro-batch size ( 8 instead of 1).

数据异质性。数据是异质性的，因为平均而言，图像的标记比相关文本的标记更多：一个图像有 2,308 个标记，而相关文本平均只有 192 个标记。因此，交叉注意力层的计算比自注意力层的计算需要更多的时间和内存。我们通过在图像编码器中引入序列并行化来解决这个问题，以便每个 GPU 处理大致相同数量的标记。因为平均文本长度相对较短，我们还使用了更大的微批量大小（8 而不是 1）。

Numerical instabilities. After the image encoder is added to the model, we find that performing gradient accumulation in bf16 led to numerical instabilities. The most likely explanation for this is that image tokens are introduced into the language backbone via all cross-attention layers. This implies that numerical deviations in the representation of an image token have an outsized impact on the overall computation because the errors are compounded. We address this by performing gradient accumulation in FP32.

数值不稳定性。在模型中添加图像编码器后，我们发现使用 bf16 进行梯度累积会导致数值不稳定性。最可能的解释是图像标记通过所有交叉注意力层引入到语言主干中。这意味着图像标记表示中的数值偏差对整体计算有较大影响，因为错误被累积了。我们通过使用 FP32 进行梯度累积来解决这个问题。

7.4 Pre-training 预训练

Image. We initialize from the pre-trained text model and vision encoder weights. The vision encoder is unfrozen, while the text model weights are kept frozen as explained above. First, we train the model using 6B image-text pairs where each image is resized to fit within four tiles of \({336} \times {336}\) pixels. We use a global batch size of \({16},{384}\) and a cosine learning rate schedule with initial learning rate \({10} \times {10}^{-4}\) and a weight decay of 0.01 . The initial learning rate was determined based on small-scale experiments. However, these findings did not generalize well to very long training schedules and dropped the learning rate a few times during training when the loss values became stagnant. After the base pre-training, we increase the image resolution further and continue training the same weights on the annealing dataset. The optimizer is re-initialized via warm-up to learning rate \(2 \times {10}^{-5}\) and again follows a cosine schedule.

图像。我们从预训练的文本模型和视觉编码器权重开始。视觉编码器是解冻的，而文本模型权重保持冻结，如上所述。首先，我们使用6B图像-文本对训练模型，其中每个图像被调整大小以适应\({336} \times {336}\)像素的四个图块。我们使用全局批量大小为\({16},{384}\)，并采用余弦学习率调度，初始学习率为\({10} \times {10}^{-4}\)，权重衰减为0.01。初始学习率是根据小规模实验确定的。然而，这些发现并未很好地推广到非常长的训练计划，并且在训练过程中当损失值停滞时，学习率会多次下降。在基础预训练之后，我们进一步提高图像分辨率，并在退火数据集上继续训练相同的权重。优化器通过预热重新初始化为学习率\(2 \times {10}^{-5}\)，并再次遵循余弦调度。

Video. For video pre-training, we start from the image pre-trained and annealed weights as described above. We add the video aggregator and cross-attention layers as described in the architecture, initialized randomly. We freeze all the parameters in the model except the video-specific ones (the aggregator and video cross-attention), and train them on the video pre-training data. We use the same training hyperparameters as the image annealing stage, with small differences in the learning rate. We uniformly sample 16 frames from the full video, and represent each frame using four chunks,each of size of \({448} \times {448}\) pixels. We use an aggregation factor of 16 in the video aggregator, hence obtaining one effective frame, which the text tokens cross-attend to. We use a global batch size of 4,096,a sequence length of 190 tokens,and a learning rate of \({10}^{-4}\) during training.

视频。对于视频预训练，我们从如上所述的图像预训练和退火权重开始。我们添加视频聚合器和交叉注意力层，如架构中所述，随机初始化。我们冻结模型中除视频特定参数（聚合器和视频交叉注意力）之外的所有参数，并在视频预训练数据上训练它们。我们使用与图像退火阶段相同的训练超参数，学习率略有不同。我们从完整视频中均匀采样16帧，并使用四个块表示每一帧，每个块的大小为\({448} \times {448}\)像素。我们在视频聚合器中使用16的聚合因子，因此获得一个有效帧，文本标记对其进行交叉注意力。我们在训练过程中使用全局批量大小为4,096，序列长度为190个标记，学习率为\({10}^{-4}\)。

7.5 Post-Training 后训练

In this section, we describe the post-training recipe for our vision adapters. After pre-training, we fine-tune the model on highly curated multi-modal conversational data to enable chat capabilities. We further implement direct preference optimization (DPO) to boost human evaluation performance and rejection sampling to improve multi-modal reasoning capabilities. Finally, we add a quality-tuning stage where we continue fine-tuning the model on a very small set of high-quality conversational data which further boosts human evaluation while retaining performance across benchmarks. More details on each of these steps are provided below.

在本节中，我们描述了我们的视觉适配器的后训练流程。在预训练之后，我们在精心策划的多模态对话数据上对模型进行微调，以实现聊天功能。我们进一步实施直接偏好优化（DPO）以提升人类评估性能，并通过拒绝采样来提高多模态推理能力。最后，我们增加了一个质量微调阶段，在该阶段我们继续在非常小的高质量对话数据集上对模型进行微调，这进一步提升了人类评估性能，同时保持了跨基准的性能。关于这些步骤的更多细节将在下面提供。

7.5.1 Supervised Finetuning Data 监督微调数据

We describe our supervised finetuning (SFT) data for image and video capabilities separately below.

我们分别描述了用于图像和视频能力的监督微调（SFT）数据。

Image. We utilize a mix of different datasets for supervised finetuning.

图像。我们利用了不同数据集的混合进行监督微调。

• Academic datasets. We convert a highly filtered collection of existing academic datasets to question-answer pairs using templates or via LLM rewriting. The LLM rewriting's purpose is to augment the data with different instructions and to improve the language quality of answers.

• 学术数据集。我们通过模板或通过LLM重写将高度筛选的现有学术数据集转换为问答对。LLM重写的目的是通过不同的指令来增强数据，并提高答案的语言质量。

• Human annotations. We collect multi-modal conversation data via human annotators for a wide range of tasks (open-ended question-answering, captioning, practical use cases, etc.) and domains (e.g., natural images and structured images). Annotators are provided with images and asked to write conversations. To ensure diversity, we cluster large-scale datasets and sampled images uniformly across different clusters. Further, we acquire additional images for a few specific domains by expanding a seed via k-nearest neighbors. Annotators are also provided with intermediate checkpoints of existing models to facilitate model-in-the-loop style annotations, so that model generations can be utilized as a starting point by the annotators to then provide additional human edits. This is an iterative process, in which model checkpoints would be regularly updated with better performing versions trained on the latest data. This increases the volume and efficiency of human annotations, while also improving their quality.

• 人工标注。我们通过人工标注者收集了广泛任务（开放式问答、字幕、实际用例等）和领域（例如，自然图像和结构化图像）的多模态对话数据。标注者会得到图像并被要求编写对话。为了确保多样性，我们对大规模数据集进行聚类，并在不同聚类中均匀采样图像。此外，我们通过k-最近邻扩展种子，为少数特定领域获取额外的图像。标注者还会得到现有模型的中间检查点，以促进模型在环风格的标注，使得模型生成可以作为标注者的起点，然后提供额外的人工编辑。这是一个迭代过程，模型检查点会定期更新为在最新数据上训练的性能更好的版本。这增加了人工标注的量和效率，同时也提高了它们的质量。

• Synthetic data. We explore different ways to generate synthetic multi-modal data by using text-representations of images and a text-input LLM. The high-level idea is to utilize the reasoning capabilities of text-input LLMs to generate question-answer pairs in the text domain, and replace the text representation with its corresponding images to produce synthetic multi-modal data. Examples include rendering texts from question-answer datasets as images or rendering table data into synthetic images of tables and charts. Additionally, we use captions and OCR extractions from existing images to generate additional conversational or question-answer data related to the images.

合成数据。我们探索通过使用图像的文本表示和文本输入的大型语言模型（LLM）来生成多模态合成数据的不同方法。高级思路是利用文本输入LLM的推理能力在文本领域生成问答对，并用相应的图像替换文本表示以生成多模态合成数据。例如，将问答数据集中的文本渲染为图像，或将表格数据渲染为合成表格和图表图像。此外，我们使用现有图像的标题和OCR提取来生成与图像相关的额外对话或问答数据。

Video. Similar to the image adapter, we use academic datasets with pre-existing annotations and convert them into appropriate textual instructions and target responses. The targets are converted to open-ended responses or multiple-choice options, whichever is more appropriate. We ask humans to annotate videos with questions and corresponding answers. The annotators are asked to focus on questions that could not be answered based on a single frame, to steer the annotators towards questions that require temporal understanding.

视频。与图像适配器类似，我们使用带有预先标注的学术数据集，并将其转换为适当的文本指令和目标响应。目标被转换为开放式响应或多选选项，以更适合的方式呈现。我们要求人类为视频标注问题和相应的答案。标注者被要求关注那些不能仅基于单帧回答的问题，以引导标注者提出需要时间理解的问题。

7.5.2 Supervised Finetuning Recipe 监督微调配方

We describe our supervised finetuning (SFT) recipe for image and video capabilities separately below.

我们分别描述了针对图像和视频能力的监督微调（SFT）配方。

Image. We initialize from the pre-trained image adapter, but hot-swap the pre-trained language model's weights with the instruction tuned language model's weights. The language model weights are kept frozen to maintain text-only performance, i.e., we only update the vision encoder and image adapter weights.

图像。我们从预训练的图像适配器初始化，但将预训练语言模型的权重与指令微调语言模型的权重进行热交换。语言模型权重保持冻结以维持纯文本性能，即我们仅更新视觉编码器和图像适配器的权重。

Our approach to finetune the model is similar to Wortsman et al. (2022). First, we run a hyperparameter sweep using multiple random subsets of data, learning rates and weight decay values. Next, we rank the models based on their performance. Finally,we average the weights of the top- \(K\) models to obtain the final model. The value of \(K\) is determined by evaluating the averaged models and selecting the instance with highest performance. We observe that the averaged models consistently yield better results compared to the best individual model found via grid search. Further, this strategy reduces sensitivity to hyperparameters.

我们的模型微调方法类似于 Wortsman 等人（2022）。首先，我们使用多个随机数据子集、学习率和权重衰减值进行超参数扫描。接着，我们根据模型性能对其进行排序。最后，我们将排名前 \(K\) 的模型的权重进行平均，以获得最终模型。\(K\) 的值是通过评估平均模型并选择性能最高的实例来确定的。我们观察到，与通过网格搜索找到的最佳单个模型相比，平均模型始终能产生更好的结果。此外，这种策略降低了模型对超参数的敏感性。

Video. For video SFT, we initialize the video aggregator and cross-attention layers using the pre-trained weights. The rest of the parameters in the model, the image weights and the LLM, are initialized from corresponding models following their finetuning stages. Similar to video pre-training, we then finetune only the video parameters on the video SFT data. For this stage, we increase the video length to 64 frames, and use an aggregation factor of 32 to get two effective frames. The resolution of the chunks is also increased to be consistent with the corresponding image hyperparameters.

视频。对于视频SFT，我们使用预训练权重初始化视频聚合器和交叉注意力层。模型的其余参数，包括图像权重和大型语言模型（LLM），则从相应的模型中初始化，这些模型经过了各自的微调阶段。与视频预训练类似，我们随后仅在视频SFT数据上对视频参数进行微调。在这一阶段，我们将视频长度增加到64帧，并使用32的聚合因子来获得两帧有效帧。块的分辨率也增加到与相应的图像超参数一致。

7.5.3 Preference Data 偏好数据

We built multimodal pair-wise preference datasets for reward modeling and direct preference optimization.

我们构建了多模态成对偏好数据集，用于奖励建模和直接偏好优化。

• Human annotations. The human-annotated preference data consists of comparisons between two different model outputs, labeled as "chosen" and "rejected", with 7-scale ratings. The models used to generate responses are sampled on-the-fly from a pool of the best recent models, each with different characteristics. We update the model pool weekly. Besides preference labels, we also request annotators to provide optional human edits to correct inaccuracies in "chosen" responses because vision tasks have a low tolerance for inaccuracies. Note that human editing is an optional step because there is a trade-off between volume and quality in practice.

• 人工标注。人工标注的偏好数据包括对两种不同模型输出的比较，标记为“选择”和“拒绝”，并附有7级评分。用于生成响应的模型是从最近最佳模型池中实时抽样的，每个模型具有不同的特征。我们每周更新模型池。除了偏好标签外，我们还要求标注者提供可选的人工编辑，以纠正“选择”响应中的不准确之处，因为视觉任务对不准确性容忍度较低。请注意，人工编辑是一个可选步骤，因为在实践中存在数量与质量之间的权衡。

• Synthetic data. Synthetic preference pairs could also be generated by using text-only LLMs to edit and deliberately introduce errors in the supervised finetuning dataset. We took the conversational data as input, and use an LLM to introduce subtle but meaningful errors (e.g., change objects, change attributes, add mistakes in calculations, etc.). These edited responses are used as negative "rejected" samples and paired with the "chosen" original supervised finetuning data.

• 合成数据。通过使用仅文本的大型语言模型（LLMs）对监督微调数据集进行编辑并故意引入错误，也可以生成合成偏好对。我们将对话数据作为输入，并使用LLM引入微妙但有意义的错误（例如，改变对象、改变属性、在计算中添加错误等）。这些编辑后的响应被用作负面的“拒绝”样本，并与“选择”的原始监督微调数据配对。

• Rejection sampling. Furthermore, to create more on-policy negative samples, we leveraged the iterative process of rejection sampling to collect additional preference data. We discuss our usage of rejection sampling in more detail in the following sections. At a high-level, rejection sampling is used to iteratively sample high-quality generations from a model. Therefore, as a by-product, all generations that are not selected can be used as negative rejected samples and used as additional preference data pairs.

• 拒绝采样。此外，为了创建更多符合策略的负面样本，我们利用拒绝采样的迭代过程来收集额外的偏好数据。我们将在后续章节中详细讨论拒绝采样的使用。在较高层次上，拒绝采样用于从模型中迭代采样高质量的生成内容。因此，作为副产品，所有未被选中的生成内容都可以用作负面的拒绝样本，并作为额外的偏好数据对使用。

7.5.4 Reward Modeling 奖励建模

We train a vision reward model (RM) on top of the vision SFT model and the language RM. The vision encoder and the cross-attention layers are initialized from the vision SFT model and unfrozen during training, while the self-attention layers are initialized from the language RM and kept frozen. We observe that freezing the language RM part generally leads to better accuracy, especially on tasks that require the RM to judge based on its knowledge or the language quality. We adopt the same training objective as the language RM, but adding a weighted regularization term on the square of the reward logits averaged over the batch, which prevents the reward scores from drifting.

我们在视觉SFT模型和语言RM的基础上训练一个视觉奖励模型（RM）。视觉编码器和交叉注意力层从视觉SFT模型初始化并在训练期间保持未冻结状态，而自注意力层从语言RM初始化并保持冻结状态。我们观察到，冻结语言RM部分通常会带来更好的准确性，特别是在需要RM根据其知识或语言质量进行判断的任务中。我们采用与语言RM相同的训练目标，但增加了一个加权正则化项，即在批次上平均的奖励对数的平方，以防止奖励分数漂移。

The human preference annotations in Section 7.5.3 are used to train the vision RM. We follow the same practice as language preference data (Section 4.2.1) to create two or three pairs with clear ranking (edited \(>\) chosen \(>\) rejected). In addition,we also synthetically augment the negative responses by perturbing the words or phrases related to the information in the image (such as numbers or visual texts). This encourages the vision RM to ground its judgement based on the actual image content.

第7.5.3节中的人类偏好注释用于训练视觉奖励模型（RM）。我们遵循与语言偏好数据（第4.2.1节）相同的实践，创建两到三对具有明确排序的配对（编辑后的 \(>\) 选定的 \(>\) 被拒绝的）。此外，我们还通过扰动与图像中信息相关的单词或短语（如数字或视觉文本）来合成增强负面响应。这鼓励视觉RM基于实际图像内容进行判断。

7.5.5 Direct Preference Optimization 直接偏好优化

Similar to the language model (Section 4.1.4), we further train the vision adapters with Direct Preference Optimization (DPO; Rafailov et al. (2023)) using the preference data described in Section 7.5.3. To combat the distribution shift during post-training rounds, we only keep recent batches of human preference annotations while dropping batches that are sufficiently off-policy (e.g.,if the base pre-trained model is changed). We find that instead of always freezing the reference model, updating it in an exponential moving average (EMA) fashion every k-steps helps the model learn more from the data, resulting in better performance in human evaluations. Overall, we observed that the vision DPO model consistently performs better than its SFT starting point in human evaluations for every finetuning iteration.

类似于语言模型（第4.1.4节），我们进一步使用第7.5.3节中描述的偏好数据，通过直接偏好优化（DPO；Rafailov等人（2023））训练视觉适配器。为了应对后训练轮次中的分布偏移，我们只保留最近批次的人类偏好注释，同时丢弃足够偏离策略的批次（例如，如果基础预训练模型发生变化）。我们发现，与始终冻结参考模型不同，以指数移动平均（EMA）方式每隔k步更新它有助于模型更多地从数据中学习，从而在人类评估中获得更好的性能。总体而言，我们观察到，在每次微调迭代中，视觉DPO模型在人类评估中始终优于其SFT起点。

7.5.6 Rejection Sampling 拒绝采样

Most available question-answer pairs only contain the final answer and lack the chain-of-thought explanation that is required to train a model that generalizes well for reasoning tasks. We use rejection sampling to generate the missing explanations for such examples and boost the model's reasoning capabilities.

大多数可用的问答对仅包含最终答案，缺乏训练一个在推理任务中泛化良好的模型所需的思维链解释。我们使用拒绝采样来生成这些示例中缺失的解释，并提升模型的推理能力。

Given a question-answer pair, we generate multiple answers by sampling the finetuned model with different system prompts or temperature. Next, we compare the generated answers to the ground-truth via heuristics or an LLM judge. Finally, we retrain the model by adding the correct answers back into the finetuning data mix. We find it useful to keep multiple correct answers per question.

给定一个问题-答案对，我们通过使用不同的系统提示或温度对微调模型进行采样来生成多个答案。接下来，我们通过启发式方法或大型语言模型（LLM）判断来比较生成的答案与真实答案。最后，我们将正确的答案重新加入到微调数据集中来重新训练模型。我们发现每个问题保留多个正确答案是有用的。

To ensure we only add high-quality examples back into training, we implemented the following two guardrails. First, we find that some examples contain incorrect explanations, despite the final answer being correct. We observed that this pattern occurs more frequently for questions where only a small fraction of the generated answers is correct. Therefore, we drop answers for questions where the probability of the answer being correct is below a certain threshold. Second, raters prefer some answers over others due to differences in language or style. We use the reward model to select top- \(K\) highest-quality answers and add them back into training.

为了确保我们只将高质量的示例加入到训练中，我们实施了以下两个防护措施。首先，我们发现尽管最终答案正确，但有些示例包含不正确的解释。我们观察到这种情况在只有一小部分生成答案正确的问题中更频繁发生。因此，我们丢弃那些正确答案概率低于某个阈值的问题的答案。其次，由于语言或风格的差异，评分者对某些答案的偏好超过其他答案。我们使用奖励模型来选择最高质量的 \(K\) 个答案，并将它们加入到训练中。

7.5.7 Quality Tuning 质量调优

We curate a small but highly selective SFT dataset where all samples have been rewritten and verified either by humans or our best models to meet our highest standards. We train DPO models with this data to improve response quality, calling the process Quality-Tuning (QT). We find that QT significantly improves human evaluations without affecting generalization verified by benchmarks when the QT dataset covers a wide range

我们精心策划了一个小而高度精选的SFT数据集，其中所有样本都已由人类或我们的最佳模型重写和验证，以达到我们的最高标准。我们使用这些数据训练DPO模型以提高响应质量，称这一过程为质量调优（QT）。我们发现，当QT数据集涵盖广泛的任务并适当应用早期停止时，QT显著提高了人类评估，而不影响通过基准验证的泛化能力。

Table 29 Image understanding performance of our vision module attached to Llama 3. We \(\mathrm{{compare}}\) model \(\mathrm{{performance}}\) to GPT-4V, GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. A Results obtained using external OCR tools.

表29 我们的视觉模块附加到Llama 3上的图像理解性能。我们将 \(\mathrm{{compare}}\) 模型 \(\mathrm{{performance}}\) 与GPT-4V、GPT-4o、Gemini 1.5 Pro和Claude 3.5 Sonnet进行比较。A 结果是通过外部OCR工具获得的。

of tasks and proper early stopping is applied. We select checkpoints at this stage purely based on benchmarks to ensure capabilities are retained or improved.

在这一阶段，我们纯粹基于基准来选择检查点，以确保能力得到保留或提升。

7.6 Image Recognition Results 图像识别结果

We evaluate the performance of the image understanding capabilities of Llama 3 on a range of tasks spanning natural image understanding, text understanding, charts understanding and multimodal reasoning:

我们评估了Llama 3在涵盖自然图像理解、文本理解、图表理解和多模态推理的一系列任务中的图像理解能力：

• MMMU (Yue et al., 2024a) is a challenging dataset for mulitmodal reasoning where model is expected to understand images and solve college-level problems spanning 30 different disciplines. This includes both multiple-choice and open ended questions. We evaluate our model on the validation set with 900 images, in line with other works.

• MMMU（Yue et al., 2024a）是一个具有挑战性的多模态推理数据集，要求模型理解图像并解决涵盖30个不同学科的大学水平问题。这包括多项选择题和开放式问题。我们按照其他工作的标准，在包含900张图像的验证集上评估我们的模型。

• VQAv2 (Antol et al., 2015) tests the ability of a model to combine image understanding, language understanding and commonsense knowlege to answer generic questions about natural images

• VQAv2（Antol et al., 2015）测试模型结合图像理解、语言理解和常识知识来回答关于自然图像的通用问题的能力。

• Al2 Diagram (Kembhavi et al., 2016) evaluates models capability to parse scientific diagrams and answer questions about the same. We use the same evaluation protocol as Gemini and x.ai, and report scores using a transparent bounding box.

• Al2 Diagram（Kembhavi et al., 2016）评估模型解析科学图表并回答相关问题的能力。我们使用与Gemini和x.ai相同的评估协议，并使用透明边界框报告分数。

• ChartQA (Masry et al., 2022) is a challenging benchmark for charts understanding. This requires model to visually understand different kinds of charts and answer logical questions about the charts.

• ChartQA（Masry et al., 2022）是一个具有挑战性的图表理解基准。这要求模型视觉理解不同类型的图表并回答关于图表的逻辑问题。

• TextVQA (Singh et al., 2019) is a popular benchmark dataset that requires models to read and reason about text in images to answer questions about them. This tests the OCR understanding ability of the model on natural images.

• TextVQA（Singh et al., 2019）是一个流行的基准数据集，要求模型阅读并推理图像中的文本以回答相关问题。这测试了模型在自然图像上的OCR理解能力。

• DocVQA (Mathew et al., 2020) is a benchmark dataset focused on document analysis and recognition. It contains images of a wide range of documents which evaluates a model's ability to perform OCR understanding and reason about the contents of a document to answer questions about them.

• DocVQA（Mathew et al., 2020）是一个专注于文档分析和识别的基准数据集。它包含各种文档的图像，评估模型执行OCR理解并推理文档内容以回答相关问题的能力。

Table 29 presents the results of our experiments. The results in the table show that our vision module attached to Llama 3 performs competitively across a wide range of image-recognition benchmarks at varying model capacities. Using the resulting Llama 3-V 405B model, we outperform GPT-4V on all benchmarks, while being slightly behind Gemini 1.5 Pro and Claude 3.5 Sonnet. Llama 3405B appears particularly competitive on document understanding tasks.

表29展示了我们的实验结果。表中的结果显示，我们的视觉模块附加到Llama 3上，在不同模型容量的一系列图像识别基准测试中表现出色。使用Llama 3-V 405B模型，我们在所有基准测试中超越了GPT-4V，同时略逊于Gemini 1.5 Pro和Claude 3.5 Sonnet。Llama 3405B在文档理解任务中表现尤为出色。

7.7 Video Recognition Results 视频识别结果

We evaluate our video adapter for Llama 3 on three benchmarks:

我们在三个基准测试上评估了Llama 3的视频适配器：

• PerceptionTest (Pătrăucean et al., 2023) evaluates the model's ability to answer temporal reasoning questions focusing on skills (memory, abstraction, physics, semantics) and different types of reasoning (descriptive,explanatory,predictive,counterfactual). It consists of \({11.6K}\) test QA pairs,each with an on-average \({23s}\) long video,filmed by 100 participants worldwide to show perceptually interesting tasks. We focus on the multiple-choice question answering task, where each question is paired with

• PerceptionTest（Pătrăucean等人，2023）评估模型回答时间推理问题的能力，重点关注技能（记忆、抽象、物理、语义）和不同类型的推理（描述性、解释性、预测性、反事实）。它包含\({11.6K}\)个测试问答对，每个问答对平均时长为\({23s}\)的视频，由全球100名参与者拍摄，展示感知上有趣的任务。我们专注于多项选择问答任务，每个问题配有三

Table 30 Video understanding performance of our vision module attached to Llama 3. We find that across range of tasks covering long-form and temporal video understanding, our vision adapters for Llama3 8B and 70B parameters are competitive and sometimes even outperform alternative models.

表30展示了我们的视觉模块附加到Llama 3上的视频理解性能。我们发现，在涵盖长格式和时间视频理解的一系列任务中，我们的Llama3 8B和70B参数的视觉适配器具有竞争力，有时甚至超过其他模型。

three possible options. We report performance on the held-out test split which is accessed by submitting our predictions to an online challenge server. \({}^{16}\)

个可能选项。我们报告了保留测试集上的性能，该测试集通过向在线挑战服务器提交我们的预测来访问。\({}^{16}\)

• NExT-QA (Xiao et al., 2021) is another temporal and causal reasoning benchmark, with a focus on open-ended question answering. It consists of \({1K}\) test videos each on-average \({44s}\) in length,paired with \({9K}\) questions. The evaluation is performed by comparing the model’s responses with the ground truth answer using Wu-Palmer Similarity (WUPS) (Wu and Palmer,1994). \({}^{17}\)

• NExT-QA（Xiao 等人，2021）是另一个关注开放式问答的时间和因果推理基准。它包含 \({1K}\) 个测试视频，每个视频平均时长为 \({44s}\)，并配有 \({9K}\) 个问题。评估通过将模型的回答与使用 Wu-Palmer 相似度（WUPS）（Wu 和 Palmer，1994）的正确答案进行比较来执行。\({}^{17}\)

• TVQA (Lei et al., 2018) evaluates the model's ability to perform compositional reasoning, requiring spatiotemporal localization of relevant moments, recognition of visual concepts, and joint reasoning with subtitle-based dialogue. This dataset, being derived from popular TV shows, additionally tests for the model's ability to leverage its outside-knowledge of those TV shows in answering the questions. It consists of over \({15K}\) validation QA pairs,with each corresponding video clip being on-average \({76s}\) in length. It also follows a multiple-choice format with five options for each question, and we report performance on the validation set following prior work (OpenAI, 2023b).

• TVQA（Lei 等人，2018）评估模型进行组合推理的能力，要求对相关时刻进行时空定位，识别视觉概念，并与基于字幕的对话进行联合推理。该数据集源自流行电视剧，还测试了模型利用对这些电视剧的外部知识来回答问题的能力。它包含超过 \({15K}\) 个验证问答对，每个对应的视频片段平均时长为 \({76s}\)。它还遵循每题五个选项的多项选择格式，我们按照先前的工作（OpenAI，2023b）报告验证集上的表现。

• ActivityNet-QA (Yu et al., 2019) evaluates the model's ability to reason over long video clips to understand actions,spatial relations,temporal relations,counting,etc. It consists of \({8K}\) test QA pairs from 800 videos, each on-average 3 minutes long. For evaluation, we follow the protocol from prior work (Google, 2023; Lin et al., 2023; Maaz et al., 2024), where the model generates short one-word or one-phrase answers, and the correctness of the output is evaluated using the GPT-3.5 API which compares it to the ground truth answer. We report the average accuracy as evaluated by the API.

• ActivityNet-QA（Yu 等人，2019）评估模型对长视频片段进行推理以理解动作、空间关系、时间关系、计数等的能力。它包含来自 800 个视频的 \({8K}\) 个测试问答对，每个视频平均时长为 3 分钟。对于评估，我们遵循先前工作（Google，2023；Lin 等人，2023；Maaz 等人，2024）的协议，模型生成简短的一词或一短语答案，输出的正确性通过与正确答案比较的 GPT-3.5 API 进行评估。我们报告由 API 评估的平均准确率。

When performing inference, we uniformly sample frames from the full video clip and pass those frames into the model with a short text prompt. Since most of our benchmarks involve answering multiple-choice questions, we use the following prompt: Select the correct answer from the following options: {question}. Answer with the correct option letter and nothing else. For benchmarks that require producing a short answer (e.g., ActivityNet-QA and NExT-QA), we use the following prompt: Answer the question using a single word or phrase. {question}. For NExT-QA, since the evaluation metric (WUPS) is sensitive to the length and the specific words used, we additionally prompt the model to be specific and respond with the most salient answer, for instance specifying "living room" instead of simply responding with "house" when asked a location question. For benchmarks that contain subtitles (i.e., TVQA), we include the subtitles corresponding to the clip in the prompt during inference.

在进行推理时，我们从完整视频片段中均匀采样帧，并将这些帧与简短的文本提示一起传递给模型。由于我们的大多数基准测试涉及回答多项选择题，因此我们使用以下提示：从以下选项中选择正确答案：{question}。仅用正确的选项字母回答。对于需要提供简短答案的基准测试（例如，ActivityNet-QA 和 NExT-QA），我们使用以下提示：用单个词或短语回答问题。{question}。对于 NExT-QA，由于评估指标（WUPS）对长度和使用的具体词汇敏感，我们额外提示模型要具体，并给出最突出的答案，例如，在询问地点问题时，指定“客厅”而不是简单地回答“房子”。对于包含字幕的基准测试（即 TVQA），我们在推理时将对应片段的字幕包含在提示中。

We present the performance of Llama 3 8B and 70B in Table 30. We compare Llama 3's performance with that of two Gemini and two GPT-4 models. Note that all our results are zero-shot, as we do not include any part of these benchmarks in our training or finetuning data. We find that our Llama 3 models that train a small video adapter during post-training are very competitive, and in some cases even better, than other models that potentially leverage native multimodal processing all the way from pre-training. Llama 3 performs particularly well on video recognition given that we only evaluate the \(8\mathrm{\;B}\) and \({70}\mathrm{\;B}\) parameter models. Llama 3 achieves its best performance on PerceptionTest, suggesting the model has a strong ability to perform complex temporal reasoning. On long-form activity understanding tasks like ActivityNet-QA, Llama 3 is able to obtain strong results even though it is processing only up to 64 frames, which means that for a 3-minute long video the model only processes one frame every 3 seconds.

我们在表30中展示了Llama 3 8B和70B的性能。我们将Llama 3的性能与两个Gemini和两个GPT-4模型进行了比较。请注意，我们的所有结果都是零样本的，因为我们没有在我们的训练或微调数据中包含这些基准的任何部分。我们发现，在训练后阶段训练小型视频适配器的Llama 3模型非常具有竞争力，并且在某些情况下甚至优于其他可能从预训练开始就利用原生多模态处理的模型。鉴于我们仅评估\(8\mathrm{\;B}\)和\({70}\mathrm{\;B}\)参数模型，Llama 3在视频识别方面表现尤为出色。Llama 3在PerceptionTest上取得了最佳性能，表明该模型具有强大的复杂时间推理能力。在ActivityNet-QA等长篇活动理解任务中，即使Llama 3仅处理最多64帧，也能获得强大结果，这意味着对于一个3分钟长的视频，模型每3秒仅处理一帧。

\({}^{16}\) See https://eval.ai/web/challenges/challenge-page/2091/overview.

\({}^{16}\) 参见 https://eval.ai/web/challenges/challenge-page/2091/overview。

\({}^{17}\) See https://github.com/doc-doc/NExT-OE.

\({}^{17}\) 参见 https://github.com/doc-doc/NExT-OE。

Figure 29 Architecture of our speech interface for Llama 3.

图29 Llama 3语音接口的架构。

8 Speech Experiments 语音实验

We perform experiments to study a compositional approach of integrating speech capabilities into Llama 3 , resembling the method we used for visual recognition. On the input side, an encoder, together with an adapter, is incorporated to process speech signals. We leverage a system prompt (in text) to enable different modes of operation for speech understanding in Llama 3. If no system prompt is provided, the model acts as a general-purpose spoken dialogue model which can effectively respond to the user speech in a manner that is consistent with the text-only version of Llama 3. The dialogue history is introduced as the prompt prefix to improve the multi-round dialogue experience. We also experiment with system prompts that enable the use of Llama 3 for automatic speech recognition (ASR) and automatic speech translation (AST). The speech interface of Llama 3 supports up to 34 languages. \({}^{18}\) It also allows for the interleaved input of text and speech, enabling the model to solve advanced audio-comprehension tasks.

我们进行实验以研究将语音能力集成到 Llama 3 中的组合方法，类似于我们在视觉识别中使用的方法。在输入端，结合编码器和适配器来处理语音信号。我们利用系统提示（以文本形式）来启用 Llama 3 中不同模式的语音理解操作。如果没有提供系统提示，模型将作为通用口语对话模型，能够有效地以与仅文本版本的 Llama 3 一致的方式响应用户语音。对话历史作为提示前缀引入，以改善多轮对话体验。我们还尝试了启用 Llama 3 进行自动语音识别（ASR）和自动语音翻译（AST）的系统提示。Llama 3 的语音接口支持多达 34 种语言。\({}^{18}\) 它还允许文本和语音的交错输入，使模型能够解决高级音频理解任务。

We also experiment with a speech generation approach in which we implement a streaming text-to-speech (TTS) system that generates speech waveforms on-the-fly during language model decoding. We design the speech generator for Llama 3 based on a proprietary TTS system and do not fine-tune the language model for speech generation. Instead, we focus on improving speech synthesis latency, accuracy, and naturalness by leveraging Llama 3 embeddings at inference time. The speech interface is illustrated in Figure 28 and 29.

我们还尝试了一种语音生成方法，在其中我们实现了一个流式文本到语音（TTS）系统，该系统在语言模型解码期间即时生成语音波形。我们基于专有 TTS 系统为 Llama 3 设计了语音生成器，并且没有对语言模型进行语音生成的微调。相反，我们专注于通过在推理时利用 Llama 3 嵌入来提高语音合成的延迟、准确性和自然度。语音接口如图 28 和 29 所示。

8.1 Data 数据

8.1.1 Speech Understanding 语音理解

The training data can be categorized into two types. The pre-training data includes a large amount of unlabeled speech, which is used to initialize the speech encoder in a self-supervised manner. The supervised finetuning data includes speech recognition, speech translation, and spoken dialogue data; this data is used to unlock specific abilities when integrated with the large language model.

训练数据可以分为两种类型。预训练数据包括大量未标记的语音，用于以自监督方式初始化语音编码器。监督微调数据包括语音识别、语音翻译和口语对话数据；这些数据用于在与大型语言模型集成时解锁特定能力。

Pre-training data. To pre-train the speech encoder, we curate a dataset of approximately 15M hours of speech recordings encompassing a large number of languages. We filter our audio data using a voice activity detection (VAD) model and select audio samples with a VAD threshold above 0.7 for pre-training. In speech pre-training data, we also focus on ensuring the absence of PII. We use the Presidio Analyzer to identify such PII.

预训练数据。为了预训练语音编码器，我们精心策划了一个包含约1500万小时语音录音的数据集，涵盖了大量语言。我们使用语音活动检测（VAD）模型过滤音频数据，并选择VAD阈值高于0.7的音频样本进行预训练。在语音预训练数据中，我们还着重确保不存在个人身份信息（PII）。我们使用Presidio分析器来识别此类PII。

Speech recognition and translation data. Our ASR training data contains \({230}\mathrm{K}\) hours of manually transcribed speech recordings that span 34 languages. Our AST training data contains 90K hours of translations in two directions: from 33 languages to English and from English to 33 languages. This data contains both supervised and synthetic data generated using the NLLB toolkit (NLLB Team et al., 2022). The use of synthetic AST data enables us to increase model quality for low-resource languages. The speech segments in our data have a maximum length of 60 seconds.

语音识别和翻译数据。我们的自动语音识别（ASR）训练数据包含\({230}\mathrm{K}\)小时的人工转录语音录音，涵盖34种语言。我们的自动语音翻译（AST）训练数据包含9万小时的翻译，方向包括从33种语言到英语和从英语到33种语言。这些数据既包括监督数据，也包括使用NLLB工具包（NLLB团队等人，2022年）生成的合成数据。使用合成AST数据使我们能够提高低资源语言的模型质量。我们的数据中的语音片段最长为60秒。

Spoken dialogue data. To finetune the speech adapter for spoken dialogue, we synthetically generate responses

口语对话数据。为了微调口语对话的语音适配器，我们合成生成响应

\({}^{18}\) The speech interface supports the following 34 languages: Arabic,Bengali,Chinese,Czech,Dutch,English,Finnish,French, German, Greek, Gujarati, Hindi, Hungarian, Indonesian, Italian, Japanese, Kannada, Korean, Malayalam, Marathi, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Swahili, Swedish, Tamil, Telugu, Thai, Turkish, Urdu, Vietnamese.

\({}^{18}\) 语音界面支持以下34种语言：阿拉伯语、孟加拉语、中文、捷克语、荷兰语、英语、芬兰语、法语、德语、希腊语、古吉拉特语、印地语、匈牙利语、印度尼西亚语、意大利语、日语、卡纳达语、韩语、马拉雅拉姆语、马拉地语、波斯语、波兰语、葡萄牙语、罗马尼亚语、俄语、西班牙语、斯瓦希里语、瑞典语、泰米尔语、泰卢固语、泰语、土耳其语、乌尔都语、越南语。

for speech prompts by asking the language model to respond to transcriptions of those prompts (Fathullah et al., 2024). We generate synthetic data this way using a subset of the ASR dataset with \({60}\mathrm{\;K}\) hours of speech. In addition, we generate 25K hours of synthetic data by running the Voicebox TTS system (Le et al., 2024) on subsets of the data used to finetune Llama 3. We used several heuristics to select a subset of finetuning data that matches the distribution of speech. These heuristics include focusing on relatively short prompts with a simple structure and without non-text symbols.

通过要求语言模型对这些提示的转录进行响应来生成语音提示的合成数据（Fathullah et al., 2024）。我们使用ASR数据集的一个子集，包含\({60}\mathrm{\;K}\)小时的语音，以这种方式生成合成数据。此外，我们通过在用于微调Llama 3的数据子集上运行Voicebox TTS系统（Le et al., 2024），生成了25,000小时的合成数据。我们使用了多种启发式方法来选择与语音分布相匹配的微调数据子集。这些启发式方法包括关注结构简单且不含非文本符号的相对较短的提示。

8.1.2 Speech Generation 语音生成

The speech generation datasets mainly consist of those for training the text normalization (TN) model and the prosody model (PM). Both training data are augmented with an additional input feature of the Llama 3 embeddings to provide contextual information.

语音生成数据集主要包括用于训练文本规范化（TN）模型和韵律模型（PM）的数据集。这两种训练数据都增加了Llama 3嵌入的额外输入特征，以提供上下文信息。

Text normalization data. Our TN training dataset includes \({55}\mathrm{\;K}\) samples that cover a wide range of semiotic classes (e.g., number, date, time) that require non-trivial normalization. Each sample is a pair of written-form text and the corresponding normalized spoken-form text, with an inferred sequence of handcrafted TN rules that carry out the normalization.

文本规范化数据。我们的TN训练数据集包括\({55}\mathrm{\;K}\)个样本，涵盖了广泛的符号类别（例如，数字、日期、时间），这些类别需要非平凡的规范化。每个样本都是书面形式文本和相应的规范化口语形式文本的对，并推断出一系列手工制作的TN规则来执行规范化。

Prosody model data. The PM training data includes linguistic and prosodic features extracted from a \({50}\mathrm{\;K}\) -hour TTS dataset, which are paired transcripts and audios recorded by professional voice actors in studio settings.

韵律模型数据。PM训练数据包括从\({50}\mathrm{\;K}\)小时的TTS数据集中提取的语言和韵律特征，这些数据集是由专业配音演员在录音室环境中录制的配对文本和音频。

Llama 3 embedding. The Llama 3 embeddings are taken as the output of the 16th decoder layer. We work exclusively with the Llama 3 8B model and extract the embeddings for a given text (i.e. written-form input text for TN or the audio transcript for PM) as if they are generated by the Llama 3 model with an empty user prompt. In a given sample, each chunk in the Llama 3 token sequence is explicitly aligned with the corresponding chunks in native input sequence for TN or PM, i.e., TN-specific text tokens (demarcated by unicode category) or phone-rate features respectively. This allows for training the TN and PM modules with streaming input of Llama 3 tokens and embeddings.

Llama 3 嵌入。Llama 3 嵌入作为第 16 层解码器的输出。我们仅使用 Llama 3 8B 模型，并提取给定文本（即 TN 的书面形式输入文本或 PM 的音频转录）的嵌入，就好像它们是由带有空用户提示的 Llama 3 模型生成的一样。在给定的样本中，Llama 3 令牌序列中的每个块都与 TN 或 PM 的原生输入序列中的相应块明确对齐，即 TN 特定的文本令牌（由 Unicode 类别划分）或电话率特征。这使得能够使用 Llama 3 令牌和嵌入的流输入来训练 TN 和 PM 模块。

8.2 Model Architecture 模型架构

8.2.1 Speech Understanding 语音理解

On the input side, the speech module consists of two successive modules: a speech encoder and an adapter. The output of the speech module is directly fed into the language model as token representation, enabling direct interaction between speech and text tokens. Furthermore, we incorporate two new special tokens to enclose the sequence of speech representations. The speech module differs substantially from the vision module (see Section 7), which feeds multi-modal information into the language model via cross-attention layers. By contrast, the speech module generates embeddings that can be seamlessly integrated with text tokens, enabling the speech interface to leverage all the capabilities of the Llama 3 language model.

在输入端，语音模块由两个连续的模块组成：一个语音编码器和一个适配器。语音模块的输出直接作为令牌表示输入到语言模型中，实现语音和文本令牌之间的直接交互。此外，我们引入了两个新的特殊令牌来包围语音表示序列。语音模块与视觉模块（见第 7 节）有很大不同，后者通过交叉注意力层将多模态信息输入到语言模型中。相比之下，语音模块生成的嵌入可以与文本令牌无缝集成，使语音接口能够利用 Llama 3 语言模型的所有功能。

Speech encoder. Our speech encoder is a Conformer (Gulati et al., 2020) model with 1B parameters. The input to the model consists of 80-dimensional mel-spectrogram features, which are first processed by a stride-4 stacking layer followed by a linear projection to reduce the frame length to \({40}\mathrm{\;{ms}}\) . The resulting features are processed by an encoder with 24 Conformer layers. Each Conformer layer has a latent dimension of 1536, and consists of two Macron-net style feed-forward networks with dimension 4096, a convolution module with kernel size 7, and a rotary attention module (Su et al., 2024) with 24 attention heads.

语音编码器。我们的语音编码器是一个具有10亿参数的Conformer模型（Gulati等人，2020）。模型的输入包括80维的梅尔频谱图特征，这些特征首先经过步长为4的堆叠层处理，然后通过线性投影层将帧长度减少到\({40}\mathrm{\;{ms}}\)。随后，这些特征由包含24个Conformer层的编码器处理。每个Conformer层具有1536维的潜在维度，并包含两个维度为4096的Macron-net风格前馈网络、一个核大小为7的卷积模块和一个具有24个注意力头的旋转注意力模块（Su等人，2024）。

Speech adapter. The speech adapter contains about \({100}\mathrm{M}\) parameters. It is composed of a convolution layer, a rotary Transformer layer, and a linear layer. The convolution layer has a kernel size of 3 and a stride of 2, which is designed to reduce the speech frame length to 80ms. This allows the model to provide more coarse-grained features to the language model. The Transformer layer has a latent dimension of 3072 and a feed-forward network with a dimension of 4096 which further processes the information from speech with context after the convolutional downsampling. Finally, the linear layer maps the output dimension to match that of the language-model embedding layer.

语音适配器。语音适配器包含约\({100}\mathrm{M}\)参数。它由一个卷积层、一个旋转Transformer层和一个线性层组成。卷积层具有核大小为3和步长为2，旨在将语音帧长度减少到80毫秒，从而使模型能够向语言模型提供更粗粒度的特征。Transformer层具有3072维的潜在维度和一个维度为4096的前馈网络，进一步处理卷积下采样后的语音信息。最后，线性层将输出维度映射以匹配语言模型嵌入层的维度。

8.2.2 Speech Generation 语音生成

We use Llama 3 8B embeddings in two key components for speech generation: Text Normalization and Prosody Modeling. The TN module ensures semantic correctness by contextually transforming written text into spoken form. The PM module enhances naturalness and expressiveness by predicting prosodic features using these embeddings. Together, they enable accurate and natural speech generation.

我们在语音生成的两个关键组件中使用Llama 3 8B嵌入：文本规范化（TN）和韵律建模（PM）。TN模块通过将书面文本上下文转换为口语形式来确保语义正确性。PM模块通过使用这些嵌入预测韵律特征来增强自然度和表现力。它们共同实现了准确且自然的语音生成。

Text normalization. As a determinant of the semantic correctness of generated speech, the text normalization (TN) module carries out context-aware transformation from written-form text into the respective spoken form which is eventually verbalized by the downstream components. For example, the written-form text 123 is read as a cardinal number (one hundred twenty three) or spelled digit-by-digit (one two three) depending on the semantic context. The TN system consists of a streaming LSTM-based sequence-tagging model that predicts the sequence of handcrafted TN rules used to transform the input text (Kang et al., 2024). The neural model also takes in Llama 3 embeddings via cross attention to leverage the contextual information encoded therein, enabling minimal text token lookahead and streaming input/output.

文本规范化。作为生成语音语义正确性的决定因素，文本规范化（TN）模块执行从书面形式文本到相应口语形式的上下文感知转换，最终由下游组件口头表达。例如，书面形式文本“123”根据语义上下文被读作基数词（一百二十三）或逐位拼读（一 二 三）。TN系统由一个基于LSTM的流式序列标注模型组成，该模型预测用于转换输入文本的手工TN规则序列（Kang et al., 2024）。神经模型还通过交叉注意力引入Llama 3嵌入，利用其中编码的上下文信息，实现最小文本标记前瞻和流式输入/输出。

Prosody modeling. To enhance the naturalness and expressiveness of synthesized speech, we integrate a decoder-only Transformer-based Prosody model (PM) (Radford et al., 2021) that takes the Llama 3 embeddings as an additional input. This integration leverages the linguistic capabilities of Llama 3, utilizing both its textual output and intermediate embeddings at the token rate (Devlin et al., 2018; Dong et al., 2019; Raffel et al., 2020; Guo et al., 2023) to enhance the prediction of prosody features, thus reducing the lookahead required by the model.

韵律建模。为了增强合成语音的自然度和表现力，我们集成了一个基于Transformer的解码器专用韵律模型（PM）（Radford et al., 2021），该模型将Llama 3嵌入作为额外输入。这种集成利用了Llama 3的语言能力，利用其文本输出和在标记速率下的中间嵌入（Devlin et al., 2018; Dong et al., 2019; Raffel et al., 2020; Guo et al., 2023）来增强韵律特征的预测，从而减少模型所需的前瞻。

The PM integrates several input components to generate comprehensive prosody predictions: linguistic features derived from the text normalization front-end detailed above, tokens, and embeddings. The PM predicts three key prosodic features: log duration of each phone, log F0 (fundamental frequency) average, and log power average across the phone duration. The model comprises a uni-directional Transformer and six attention heads. Each block includes cross-attention layers and dual fully connected layers with a hidden dimension of 864. A distinctive feature of the PM is its dual cross-attention mechanism, with one layer dedicated to linguistic inputs and the other to Llama embeddings. This setup efficiently manages varying input rates without requiring explicit alignment.

PM 整合了多个输入组件以生成全面的韵律预测：来自上述文本归一化前端的语言特征、标记和嵌入。PM 预测三个关键的韵律特征：每个音素的对数时长、对数 F0（基频）平均值以及音素时长内的对数功率平均值。该模型包含一个单向 Transformer 和六个注意力头。每个块包括交叉注意力层和具有 864 隐藏维度的双全连接层。PM 的一个显著特点是其双交叉注意力机制，其中一层专用于语言输入，另一层用于 Llama 嵌入。这种设置有效地管理不同输入速率，无需显式对齐。

8.3 Training Recipe 训练配方

8.3.1 Speech Understanding 语音理解

Training of the speech module is done in two stages. The first stage, speech pre-training, leverages unlabeled data to train a speech encoder that exhibits strong generalization capabilities across languages and acoustic conditions. In the second stage, supervised fine-tuning, the adapter and pre-trained encoder are integrated with the language model, and trained jointly with it while the LLM stays frozen. This enables the model to respond to speech input. This stage uses labeled data corresponding to speech understanding abilities.

语音模块的训练分为两个阶段。第一阶段，语音预训练，利用未标记数据训练一个在语言和声学条件下具有强大泛化能力的语音编码器。在第二阶段，监督微调，适配器和预训练编码器与语言模型集成，并与语言模型一起训练，同时 LLM 保持冻结状态。这使得模型能够响应语音输入。此阶段使用与语音理解能力相对应的标记数据。

Multilingual ASR and AST modeling often results in language confusion/interference, which leads to degraded performance. A popular way to mitigate this is to incorporate language identification (LID) information, both on the source and target side. This can lead to improved performance in the predetermined set of directions, but it does come with potential loss of generality. For instance, if a translation system expects LID on both source and target side, then the model will not likely to show good zero-shot performance in directions that were not seen in training. So our challenge is to design a system that allows LID information to some extent, but keeps the model general enough such that we can have the model do speech translation in unseen directions. To address this, we design system prompts which only contain LID for the text to be emitted (target side). There is no LID information for the speech input (source side) in these prompts, which also potentially allows it to work with code-switched speech. For ASR, we use the following system prompt: Repeat after me in {language}:, where {language} comes from one of the 34 languages (English, French, etc.) For speech translation, the system prompt is: Translate the following sentence into {language}:. This design has been shown to be effective in prompting the language model to respond in the desired language. We used the same system prompts during training and inference.

多语言自动语音识别（ASR）和自动语音翻译（AST）建模常常导致语言混淆/干扰，从而降低性能。一种流行的缓解方法是结合语言识别（LID）信息，无论是在源端还是目标端。这可以在预定的方向集合中提高性能，但它确实带来了潜在的通用性损失。例如，如果一个翻译系统期望在源端和目标端都有LID，那么该模型在训练中未见过的方向上不太可能表现出良好的零样本性能。因此，我们的挑战是设计一个系统，该系统在一定程度上允许LID信息，但保持模型足够通用，以便我们能够让模型在未见过的方向上进行语音翻译。为了解决这个问题，我们设计了仅包含待输出文本（目标端）LID的系统提示。在这些提示中，语音输入（源端）没有LID信息，这也可能使其能够处理代码转换的语音。对于ASR，我们使用的系统提示是：“{language}语言跟我重复：”，其中{language}来自34种语言之一（如英语、法语等）。对于语音翻译，系统提示是：“将以下句子翻译成{language}：”。这种设计已被证明能有效提示语言模型以期望的语言响应。我们在训练和推理过程中使用了相同的系统提示。

Speech pre-training. We use the self-supervised BEST-RQ algorithm (Chiu et al., 2022) to pre-train the speech encoder. We apply a mask of 32-frame length with a probability of \({2.5}\%\) to the input mel-spectrogram. If the speech utterances are longer than 60 seconds, we perform a random crop of 6K frames, corresponding to 60 seconds of speech. We quantize mel-spectrogram features by stacking 4 consecutive frames, projecting the 320-dimensional vectors to a 16-dimensional space, and performing a nearest-neighbor search with respect to cosine similarity metric within a codebook of 8,192 vectors. To stabilize pre-training, we employ 16 different codebooks. The projection matrix and codebooks are randomly initialized and are not updated throughout the model training. The multi-softmax loss is used only on masked frames for efficiency reasons. The encoder is trained for \({500}\mathrm{\;K}\) steps with a global batch size of 2,048 utterances.

语音预训练。我们使用自监督的 BEST-RQ 算法（Chiu 等人，2022）来预训练语音编码器。我们对输入的梅尔频谱图应用一个长度为 32 帧的掩码，其概率为 \({2.5}\%\)。如果语音话语超过 60 秒，我们执行一个随机裁剪，裁剪出 6K 帧，相当于 60 秒的语音。我们通过堆叠 4 个连续帧，将 320 维向量投影到 16 维空间，并在一个包含 8,192 个向量的码本中，根据余弦相似度度量进行最近邻搜索，来量化梅尔频谱图特征。为了稳定预训练，我们使用了 16 个不同的码本。投影矩阵和码本都是随机初始化的，并且在整个模型训练过程中不进行更新。出于效率原因，仅对掩码帧使用多重 softmax 损失。编码器训练了 \({500}\mathrm{\;K}\) 步，全局批量大小为 2,048 个话语。

Supervised finetuning. Both the pre-trained speech encoder and the randomly initialized adapter are further jointly optimized with Llama 3 in the supervised finetuning stage. The language model remains unchanged during this process. The training data is a mixture of ASR, AST, and spoken dialogue data. The speech model for Llama 3 8B is trained for \({650}\mathrm{\;K}\) updates,using a global batch size of 512 utterances and an initial learning rate of \({10}^{-4}\) . The speech model for Llama 370 B is trained for \({600}\mathrm{\;K}\) updates,using a global batch size of 768 utterances and an initial learning rate of \(4 \times {10}^{-5}\) .

监督微调。在监督微调阶段，预训练的语音编码器和随机初始化的适配器与 Llama 3 一起进一步联合优化。在此过程中，语言模型保持不变。训练数据是 ASR、AST 和口语对话数据的混合。Llama 3 8B 的语音模型训练了 \({650}\mathrm{\;K}\) 次更新，使用全局批量大小为 512 个话语和初始学习率为 \({10}^{-4}\)。Llama 370 B 的语音模型训练了 \({600}\mathrm{\;K}\) 次更新，使用全局批量大小为 768 个话语和初始学习率为 \(4 \times {10}^{-5}\)。

8.3.2 Speech Generation 语音生成

To support real-time processing, the prosody model employs a lookahead mechanism that considers a fixed number of future phones and a variable number of future tokens. This ensures consistent lookahead while processing incoming text, which is crucial for low-latency speech synthesis applications.

为了支持实时处理，韵律模型采用了一种前瞻机制，该机制考虑了固定数量的未来音素和可变数量的未来标记。这确保了在处理输入文本时的一致性前瞻，对于低延迟语音合成应用至关重要。

Training. We develop a dynamic alignment strategy utilizing causal masking to facilitate streamability in speech synthesis. This strategy incorporates a lookahead mechanism for a fixed number of future phones and a variable number of future tokens, aligning with the chunking process during text normalization (Section 8.1.2). For each phone, the token lookahead includes the maximum number of tokens defined by the chunk size, resulting in variable lookahead for Llama embeddings but fixed lookahead for phonemes.

训练。我们开发了一种动态对齐策略，利用因果掩码来促进语音合成的流式处理。该策略结合了前瞻机制，考虑了固定数量的未来音素和可变数量的未来标记，与文本规范化过程中的分块处理（第8.1.2节）相一致。对于每个音素，标记前瞻包括由块大小定义的最大数量的标记，导致Llama嵌入的可变前瞻，但音素的前瞻是固定的。

The Llama 3 embeddings are sourced from the Llama 3 8B model, which remains frozen during the training of the Prosody Model. The input phone-rate features include both linguistic and speaker/style controllability elements. The model training is conducted with a batch size of 1,024 utterances, each with a maximum length of 500 phones. We employ a learning rate of \(9 \times {10}^{-4}\) using the AdamW optimizer,training over 1 million updates with a learning rate warmup for the first 3,000 updates, following a cosine schedule.

Llama 3嵌入来自Llama 3 8B模型，该模型在韵律模型训练期间保持冻结状态。输入的音素速率特征包括语言学和说话者/风格可控性元素。模型训练采用批量大小为1,024个话语，每个话语的最大长度为500个音素。我们使用AdamW优化器，学习率为\(9 \times {10}^{-4}\)，训练超过100万次更新，前3,000次更新采用学习率预热，随后遵循余弦调度。

Inference. During inference, the same lookahead mechanism and causal masking strategy are employed to ensure consistency between training and real-time processing. The PM handles incoming text in a streaming manner, updating the input phone by phone for phone-rate features and chunk by chunk for token-rate features. The new chunk input is updated only when the first phone for that chunk is current, maintaining the alignment and lookahead as during training.

推理。在推理过程中，采用了相同的前瞻机制和因果掩码策略，以确保训练和实时处理之间的一致性。PM以流式方式处理输入文本，逐音素更新音素速率特征，逐块更新标记速率特征。仅当该块的第一个音素当前时，才更新新的块输入，保持与训练期间的对齐和前瞻。

For prosody target prediction, we employ a delayed pattern approach (Kharitonov et al., 2021), which enhances the model's ability to capture and reproduce long-range prosodic dependencies. This approach contributes to the naturalness and expressiveness of the synthesized speech, ensuring low-latency and high-quality output.

对于韵律目标预测，我们采用了一种延迟模式方法（Kharitonov et al., 2021），该方法增强了模型捕捉和再现长距离韵律依赖的能力。这种方法有助于合成语音的自然度和表现力，确保低延迟和高品质的输出。

8.4 Speech Understanding Results 语音理解结果

We evaluate the speech understanding capabilities of our speech interface for Llama 3 on three tasks: (1) automatic speech recognition, (2) speech translation, and (3) spoken question answering. We compare the performance of our speech interface for Llama 3 with three state-of-the-art models for speech understanding: Whisper (Radford et al.,2023),SeamlessM4T (Barrault et al.,2023),and Gemini. \({}^{19}\) In all the evaluations,we used greedy search for Llama 3 token prediction.

我们评估了Llama 3语音接口在三个任务上的语音理解能力：（1）自动语音识别，（2）语音翻译，和（3）口语问答。我们将Llama 3语音接口的性能与三种最先进的语音理解模型进行了比较：Whisper（Radford et al., 2023），SeamlessM4T（Barrault et al., 2023），以及Gemini。\({}^{19}\) 在所有评估中，我们使用了贪心搜索进行Llama 3的令牌预测。

Speech recognition. We evaluate the ASR performance on the English datasets of Multilingual LibriSpeech (MLS; Pratap et al. (2020)), LibriSpeech (Panayotov et al., 2015), VoxPopuli (Wang et al., 2021a), and a subset of the multilingual FLEURS dataset (Conneau et al., 2023). In evaluation, the decoding results are post-processed using the Whisper text normalizer to ensure consistency in comparing with the reported results of other models. On all benchmarks, we measure the word error rate of our speech interface for Llama 3

语音识别。我们在多语言LibriSpeech（MLS；Pratap et al. (2020)），LibriSpeech（Panayotov et al., 2015），VoxPopuli（Wang et al., 2021a）和多语言FLEURS数据集（Conneau et al., 2023）的一个子集上评估了ASR性能。在评估中，解码结果使用Whisper文本规范化器进行后处理，以确保与其他模型报告结果的比较一致性。在所有基准测试中，我们测量了Llama 3语音接口的词错误率。

\({}^{19}\) Due to technical limitations,we compare with the performance of Gemini on MLS reported in the original paper.

\({}^{19}\) 由于技术限制，我们与原始论文中报告的Gemini在MLS上的性能进行了比较。

Table 31 Word error rate of our speech interface for Llama 3 on speech recognition tasks. We report the performance of Whisper, SeamlessM4T, and Gemini for reference.

表31 Llama 3语音接口在语音识别任务上的词错误率。我们报告了Whisper，SeamlessM4T和Gemini的性能以供参考。

Table 32 BLEU score of our speech interface for Llama 3 on speech translation tasks. We \(\mathrm{{report}}\) the \(\mathrm{{performance}}\) of \(\mathrm{{Whisper}}\) and SeamlessM4T for reference.

表32展示了我们在Llama 3语音翻译任务上的BLEU分数。我们\(\mathrm{{report}}\)参考了\(\mathrm{{Whisper}}\)和SeamlessM4T的\(\mathrm{{performance}}\)。

on the standard test set of those benchmarks, except for Chinese, Japanese, Korean and Thai, where the character error rate is reported.

在这些基准的标准测试集上，除了中文、日语、韩语和泰语外，报告了字符错误率。

Table 31 shows the results of ASR evaluations. It demonstrates the strong performance of Llama 3 (and multi-modal foundation models more generally) on speech recognition tasks: our model outperforms models that are tailored to speech like Whisper \({}^{20}\) and SeamlessM4T on all benchmarks. On MLS English,Llama 3 performs similarly to Gemini.

表31展示了ASR评估的结果。它展示了Llama 3（以及更普遍的多模态基础模型）在语音识别任务上的强大性能：我们的模型在所有基准测试中都优于专门针对语音的模型，如Whisper\({}^{20}\)和SeamlessM4T。在MLS英语上，Llama 3的表现与Gemini相似。

Speech translation. We also evaluate our models on speech translation tasks in which the model is asked to translate non-English speech into English text. We use the FLEURS and Covost 2 (Wang et al., 2021b) datasets in these evaluations, measuring BLEU scores of the translated English. Table 32 presents the results of these experiments. \({}^{21}\) The performance of our models in speech translation highlights the advantages of multimodal foundation models for tasks such as speech translation.

语音翻译。我们还在语音翻译任务上评估了我们的模型，这些任务要求模型将非英语语音翻译成英语文本。在这些评估中，我们使用了FLEURS和Covost 2（Wang et al., 2021b）数据集，测量了翻译成英语的BLEU分数。表32展示了这些实验的结果。\({}^{21}\)我们的模型在语音翻译中的表现突显了多模态基础模型在语音翻译等任务上的优势。

Spoken question answering. The speech interface of Llama 3 demonstrates remarkable question answering capabilities. The model can effortlessly comprehend code-switched speech without any prior exposure to such data. Notably, although the model was trained only on single-turn dialogue, it is capable of engaging in extended, coherent multi-turn dialogue sessions. Figure 30 presents a few examples that highlight these multilingual and multi-turn capabilities.

口语问答。Llama 3的语音界面展示了出色的问答能力。该模型可以轻松理解代码转换的语音，无需事先接触此类数据。值得注意的是，尽管模型仅在单轮对话上进行了训练，但它能够进行扩展的、连贯的多轮对话会话。图30展示了一些突出这些多语言和多轮能力的示例。

Safety. We evaluate the safety of our speech model on MuTox (Costa-jussà et al., 2023), a multilingual audio-based dataset of 20,000 utterances for English and Spanish and 4,000 for 19 other languages, each with toxicity labels attached. The audio is passed as input to the model and the output is evaluated for toxicity, after cleaning some special characters. We apply the MuTox classifier (Costa-jussà et al., 2023) and compare the results with Gemini 1.5 Pro. We evaluate the percentage of added toxicity (AT), when the input prompt is safe and the output is toxic, and the percentage of lost toxicity (LT), when the input prompt is toxic and the answer is safe. Table 33 shows the results for English and an average across all 21 languages that we evaluated on. \({}^{22}\) The percentage of added toxicity is very low: our speech models have the lowest percentage of added toxicity for English,with less than \(1\%\) . It removes significantly more toxicity than it adds.

安全性。我们在 MuTox（Costa-jussà 等人，2023）上评估了我们语音模型的安全性，这是一个包含 20,000 条英语和西班牙语以及 4,000 条其他 19 种语言的基于音频的多语言数据集，每条音频都附有有毒性标签。音频作为输入传递给模型，经过清除一些特殊字符后，输出被评估其有毒性。我们应用 MuTox 分类器（Costa-jussà 等人，2023）并将结果与 Gemini 1.5 Pro 进行比较。我们评估了当输入提示安全而输出有毒时的添加有毒性百分比（AT），以及当输入提示有毒而回答安全时的丢失有毒性百分比（LT）。表 33 显示了我们在英语和所有 21 种评估语言上的平均结果。\({}^{22}\) 添加有毒性的百分比非常低：我们的语音模型在英语中的添加有毒性百分比最低，低于 \(1\%\)。它去除的有毒性远多于添加的有毒性。

8.5 Speech Generation Results 语音生成结果

For speech generation, we focus on evaluating the quality of token-wise input streaming models with the Llama 3 embeddings for the text normalization and prosody modeling tasks. The evaluation focuses on

对于语音生成，我们专注于评估带有 Llama 3 嵌入的逐词输入流模型的质量，用于文本规范化和韵律建模任务。评估重点在于

\({}^{20}\) On FLEURS ASR,Malayalam is not officially reported for Whisper v3,so we use the average of 33 languages.

\({}^{20}\) 在 FLEURS ASR 上，Whisper v3 未正式报告马拉雅拉姆语的结果，因此我们使用 33 种语言的平均值。

\({}^{21}\) On Covost 2,we evaluate only on 15 (out of 21) languages.

\({}^{21}\) 在 Covost 2 上，我们仅评估了 15 种（共 21 种）语言。

\({}^{22}\) Note that for Gemini,we encountered that a significant number of responses were empty,which could be due to safety filters on their side (though some empty responses were for non-toxic input) or to rate limits. To conduct the analysis, we assumed that all the empty responses are safe. This is the most conservative approach for results and the upper bound of what Gemini results would look like.

\({}^{22}\) 需要注意的是，对于Gemini，我们遇到了大量响应为空的情况，这可能是由于他们方面的安全过滤器（尽管一些空响应是非有毒输入）或速率限制。为了进行分析，我们假设所有空响应都是安全的。这是最保守的结果处理方法，也是Gemini结果的上限。

Figure 30 Transcribed dialogue examples using the speech interface for Llama 3. The examples illustrate \(\mathrm{{zero}}\) -shot multi-turn and code-switching capabilities.

图30 使用Llama 3语音界面的转录对话示例。这些示例展示了\(\mathrm{{zero}}\)次的多轮对话和代码切换能力。

Table 33 Speech toxicity of our speech interface to Llama 3 on the MuTox dataset. AT \(\mathrm{{refers}}\) to added \(\mathrm{{toxicity}}\left( \% \right)\) and \(\mathrm{{LT}}\) refers to lost toxicity \(\left( \% \right)\) .

表33 我们的语音界面在MuTox数据集上对Llama 3的语音毒性。AT \(\mathrm{{refers}}\) 添加了 \(\mathrm{{toxicity}}\left( \% \right)\) 和 \(\mathrm{{LT}}\) 指的是毒性丧失 \(\left( \% \right)\)。

comparisons with models that do not take the Llama 3 embeddings as an additional input.

与不将Llama 3嵌入作为额外输入的模型进行比较。

Text normalization. To measure the effect of Llama 3 embeddings, we experimented with changing the amount of right context the model uses. We trained the model using a right context of \(3\mathrm{{TN}}\) tokens (demarcated by unicode category). This model is compared to models that do not use the Llama 3 embeddings, using a 3-token right context or a full bi-directional context. As expected, Table 34 shows using the full right context improves performance for the model without Llama 3 embeddings. However, the model that incorporates the Llama 3 embeddings outperforms all other models, hence enabling token-rate input/output streaming without relying on long context in the input.

文本规范化。为了衡量Llama 3嵌入的影响，我们尝试改变模型使用的右侧上下文量。我们使用\(3\mathrm{{TN}}\)个令牌（由Unicode类别划分）的右侧上下文训练模型。将此模型与不使用Llama 3嵌入的模型进行比较，后者使用3个令牌的右侧上下文或完整的双向上下文。正如预期的那样，表34显示使用完整的右侧上下文可以提高没有Llama 3嵌入的模型的性能。然而，结合了Llama 3嵌入的模型优于所有其他模型，从而实现了无需依赖输入中的长上下文的令牌速率输入/输出流。

Prosody modeling. To evaluate the performance of the our prosody model (PM) with Llama 3 8B, we conducted two sets of human evaluation comparing models with and without Llama 3 embeddings. Raters listened to samples from different models and indicated their preferences. To generate the final speech waveform, we use an in-house transformer based acoustic model (Wu et al., 2021) that predicts spectral features and a WaveRNN neural vocoder (Kalchbrenner et al., 2018) to generate the final speech waveform.

韵律建模。为了评估我们的韵律模型（PM）与Llama 3 8B的性能，我们进行了两组人工评估，比较了有无Llama 3嵌入的模型。评测者听取了来自不同模型的样本并表达了他们的偏好。为了生成最终的语音波形，我们使用了一个基于Transformer的声学模型（Wu et al., 2021）来预测频谱特征，并使用WaveRNN神经声码器（Kalchbrenner et al., 2018）来生成最终的语音波形。

Table 34 Sample-wise text normalization (TN) accuracy. We compare models with or without Llama 3 8B embeddings, and using different right-context values.

表34 样本级文本规范化（TN）准确率。我们比较了有无Llama 3 8B嵌入的模型，并使用了不同的右上下文值。

First, we compare directly to a streaming baseline model without Llama 3 embeddings. In the second test, the Llama 3 8B PM is compared to a non-streaming baseline model without Llama 3 embeddings. As shown in Table 35,the Llama 3 8B PM is preferred \({60}\%\) of the time compared to the streaming baseline,and

首先，我们直接与一个没有Llama 3嵌入的流式基线模型进行比较。在第二次测试中，Llama 3 8B PM与一个没有Llama 3嵌入的非流式基线模型进行比较。如表35所示，Llama 3 8B PM相比流式基线模型更受青睐\({60}\%\)的时间，并且

Table 35 Prosody Modeling (PM) evaluation. \({Left}\) : Rater preferences of PM for Llama 3 8B vs. streaming phone-only baseline. Right: Rater preferences of PM for Llama 3 8B vs. non-streaming phone-only baseline.

表35 韵律建模（PM）评估。\({Left}\)：Llama 3 8B PM与仅流式电话基线的评测者偏好。右：Llama 3 8B PM与仅非流式电话基线的评测者偏好。

\({63.6}\%\) of the time compared to the non-streaming baseline,indicating a significant improvement in perceived quality. The key advantage of the Llama 3 8B PM is its token-wise streaming capability (Section 8.2.2), which maintains low latency during inference. This reduces the model's lookahead requirements, enabling more responsive and real-time speech synthesis compared to non-streaming baselines. Overall, the Llama 38B prosody model consistently outperforms the baseline models, demonstrating its effectiveness in enhancing the naturalness and expressiveness of synthesized speech.

与非流式基线相比，\({63.6}\%\) 的时间显著减少，表明感知质量有显著提升。Llama 3 8B PM 的关键优势在于其逐词流式处理能力（第 8.2.2 节），这使得推理过程中保持低延迟。这减少了模型的前瞻需求，使得与非流式基线相比，语音合成更加响应迅速和实时。总体而言，Llama 38B 韵律模型始终优于基线模型，展示了其在增强合成语音的自然度和表现力方面的有效性。

The development of Llama 3 builds on a large body of prior work studying foundation models for language, images, videos, and speech. A comprehensive overview of that work is outside the scope of this paper; we refer the reader to Bordes et al. (2024); Madan et al. (2024); Zhao et al. (2023a) for such overviews. Below, we briefly outline seminal works that directly influenced the development of Llama 3.

Llama 3 的开发建立在对语言、图像、视频和语音的基础模型进行大量先前研究的基础上。全面概述这些工作超出了本文的范围；我们建议读者参考 Bordes 等人（2024 年）；Madan 等人（2024 年）；Zhao 等人（2023a）以获取此类概述。以下，我们简要概述了对 Llama 3 开发产生直接影响的重要工作。

9.1 Language 语言

Scale. Llama 3 follows the enduring trend of applying straightforward methods at ever increasing scales in foundation models. Improvements are driven by increased compute and improved data, with the 405B model using almost fifty times the pre-training compute budget of Llama 270B. Despite containing 405B parameters, our largest Llama 3 in fact contains fewer parameters than earlier and much less performant models such as PALM (Chowdhery et al., 2023), due to better understanding of scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). Little is publicly known about the size of other frontier models, such as Claude 3 or GPT 4 (OpenAI, 2023a), but overall performance is compareable.

规模。Llama 3 遵循了在基础模型中不断增加规模应用简单方法的持久趋势。改进源于计算量的增加和数据质量的提升，405B 模型使用的预训练计算预算几乎是 Llama 270B 的五十倍。尽管包含 405B 参数，我们的最大 Llama 3 实际上包含的参数比早期且性能低得多的模型如 PALM（Chowdhery 等人，2023 年）要少，这是由于对缩放定律的更好理解（Kaplan 等人，2020 年；Hoffmann 等人，2022 年）。关于其他前沿模型如 Claude 3 或 GPT 4（OpenAI，2023a）的规模公开信息很少，但总体性能相当。

Small models. Developments in smaller models have paralleled those in large models. Models with fewer parameters can dramatically improve inference cost and simplify deployment (Mehta et al., 2024; Team et al., 2024). The smaller Llama 3 models achieve this by training far beyond the point of compute optimal training, effectively trading training compute for inference efficiency. An alternative path is to distill larger models into smaller ones, as in Phi (Abdin et al., 2024).

小型模型。小型模型的发展与大型模型并行。参数较少的模型可以显著降低推理成本并简化部署（Mehta et al., 2024; Team et al., 2024）。较小的 Llama 3 模型通过在计算最优训练点之外进行远超训练，有效地用训练计算换取推理效率。另一种方法是将从大型模型中提炼成小型模型，如 Phi（Abdin et al., 2024）。

Architectures. While Llama 3 makes minimal architectural modifiations to compared to Llama 2, other recent foundation models have explored other designs. Most notably, mixture of experts architectures (Shazeer et al., 2017; Lewis et al., 2021; Fedus et al., 2022; Zhou et al., 2022) can be used as an efficient way to increase the capacity of a models, such as in Mixtral (Jiang et al., 2024) and Arctic (Snowflake, 2024). Llama 3 outperforms these models, suggesting that dense architectures are not the limiting factor, but there remain numerous trade offs in terms of training and inference efficiency, and model stability at scale.

架构。尽管 Llama 3 相对于 Llama 2 进行了最小的架构修改，但其他最近的基石模型探索了其他设计。最值得注意的是，专家混合架构（Shazeer et al., 2017; Lewis et al., 2021; Fedus et al., 2022; Zhou et al., 2022）可以作为一种有效的方式来增加模型的容量，例如在 Mixtral（Jiang et al., 2024）和 Arctic（Snowflake, 2024）中。Llama 3 在这些模型中表现更优，表明密集架构并非限制因素，但在训练和推理效率以及大规模模型稳定性方面仍存在许多权衡。

Open source. Open weights foundation models have rapidly improved over the last year, with Llama3-405B now competitive with the current closed weight state-of-the-art. Numerous model families have recently been developed, including Mistral (Jiang et al., 2023), Falcon (Almazrouei et al., 2023), MPT (Databricks, 2024), Pythia (Biderman et al., 2023), Arctic (Snowflake, 2024), OpenELM (Mehta et al., 2024), OLMo (Groeneveld et al., 2024), StableLM (Bellagente et al., 2024), OpenLLaMA (Geng and Liu, 2023), Qwen (Bai et al., 2023), Gemma (Team et al., 2024), Grok (XAI, 2024), and Phi (Abdin et al., 2024).

开源。开放权重基础模型在过去一年中迅速改进，现在 Llama3-405B 已经与当前的封闭权重最先进技术相竞争。近期开发了众多模型系列，包括 Mistral（Jiang et al., 2023）、Falcon（Almazrouei et al., 2023）、MPT（Databricks, 2024）、Pythia（Biderman et al., 2023）、Arctic（Snowflake, 2024）、OpenELM（Mehta et al., 2024）、OLMo（Groeneveld et al., 2024）、StableLM（Bellagente et al., 2024）、OpenLLaMA（Geng and Liu, 2023）、Qwen（Bai et al., 2023）、Gemma（Team et al., 2024）、Grok（XAI, 2024）和 Phi（Abdin et al., 2024）。

Post-training. Post-training Llama 3 follows the established strategy of instruction tuning (Chung et al., 2022; Ouyang et al., 2022) followed by alignment with human feedback (Kaufmann et al., 2023). While some studies have shown the surprising effectiveness of lightweight alignment procedures (Zhou et al., 2024), Llama 3 uses millions of human instructions and preference judgments to improve the pre-trained model, including techniques such as rejection sampling (Bai et al., 2022), supervised finetuning (Sanh et al., 2022), and Direct Preference Optimization (Rafailov et al., 2023). In order to curate these instruction and preference examples, we deploy earlier versions of Llama 3 to filter (Liu et al., 2024c), re-write (Pan et al., 2024), or generate prompts and responses (Liu et al., 2024b) and apply these techniques through multiple rounds of post-training.

训练后。Llama 3 遵循既定的策略，即先进行指令调优（Chung et al., 2022; Ouyang et al., 2022），然后通过人类反馈进行对齐（Kaufmann et al., 2023）。尽管一些研究表明轻量级对齐程序具有惊人的有效性（Zhou et al., 2024），但 Llama 3 使用数百万条人类指令和偏好判断来改进预训练模型，包括拒绝采样（Bai et al., 2022）、监督微调（Sanh et al., 2022）和直接偏好优化（Rafailov et al., 2023）等技术。为了策划这些指令和偏好示例，我们部署了早期版本的 Llama 3 进行过滤（Liu et al., 2024c）、重写（Pan et al., 2024）或生成提示和响应（Liu et al., 2024b），并通过多轮训练后应用这些技术。

9.2 Multimodality 多模态

Our experiments with multimodal capabilities for Llama 3 are part of a long line of work on foundation models that jointly model multiple modalities.

我们对 Llama 3 的多模态能力的实验是关于基础模型联合建模多种模态的长期工作的一部分。

Images. A substantial body of work has trained image-recognition models on large amounts of image-text pairs, for example, Mahajan et al. (2018); Xiao et al. (2024a); Team (2024); OpenAI (2023b). Radford et al. (2021) presented one of the first models to jointly embed images and text via contrastive learning. More recently, a series of models has studied approaches similar to the one used in Llama 3, for example, Alayrac et al. (2022); Dai et al. (2023); Liu et al. (2023c,b); Yang et al. (2023b); Ye et al. (2023); Zhu et al. (2023). Our approach in Llama 3 combines ideas from many of these papers to achieve results that are comparable with Gemini 1.0 Ultra (Google, 2023) and GPT-4 Vision (OpenAI, 2023b); see Section 7.6.

图像。大量工作已经在大规模图像-文本对上训练了图像识别模型，例如，Mahajan 等人（2018）；Xiao 等人（2024a）；Team（2024）；OpenAI（2023b）。Radford 等人（2021）提出了首批通过对比学习联合嵌入图像和文本的模型之一。最近，一系列模型研究了类似于 Llama 3 中使用的方法，例如，Alayrac 等人（2022）；Dai 等人（2023）；Liu 等人（2023c,b）；Yang 等人（2023b）；Ye 等人（2023）；Zhu 等人（2023）。我们在 Llama 3 中的方法结合了这些论文中的许多思想，实现了与 Gemini 1.0 Ultra（Google，2023）和 GPT-4 Vision（OpenAI，2023b）相当的结果；参见第 7.6 节。

Video. Although video inputs are supported by an increasing number of foundation models (Google, 2023; OpenAI, 2023b), the body of work on joint modeling of videos and language is not that large. Akin to Llama 3, most current studies adopt an adapter approach to align video and language representations and unlock question-answering and reasoning about videos (Lin et al., 2023; Li et al., 2023a; Maaz et al., 2024; Zhang et al., 2023; Zhao et al., 2022). We find that such approaches produce results that are competitive with the state-of-the-art; see Section 7.7.

视频。尽管越来越多的基础模型支持视频输入（Google，2023；OpenAI，2023b），但关于视频和语言联合建模的工作量并不大。与 Llama 3 类似，当前大多数研究采用适配器方法来对齐视频和语言表示，并解锁关于视频的问答和推理（Lin 等人，2023；Li 等人，2023a；Maaz 等人，2024；Zhang 等人，2023；Zhao 等人，2022）。我们发现这些方法产生的结果与最先进水平相当；参见第 7.7 节。

Speech. Our work also fits in a larger body of work combining language and speech modeling. Earlier joint models of text and speech include AudioPaLM (Rubenstein et al., 2023), VioLA (Wang et al., 2023b), VoxtLM Maiti et al. (2023), SUTLM (Chou et al., 2023), and Spirit-LM (Nguyen et al., 2024). Our work builds on prior compositional approaches to combining speech and language like Fathullah et al. (2024). Unlike most prior work, we opt to not finetune the language model itself for speech tasks as doing so may lead to contention on non-speech tasks. We find that at larger model scales, strong performances are attainable even without such finetuning; see Section 8.4.

语音。我们的工作也符合将语言和语音建模结合起来的更大范围的工作。早期的文本和语音联合模型包括 AudioPaLM（Rubenstein 等人，2023）、VioLA（Wang 等人，2023b）、VoxtLM Maiti 等人（2023）、SUTLM（Chou 等人，2023）和 Spirit-LM（Nguyen 等人，2024）。我们的工作建立在先前结合语音和语言的组合方法上，如 Fathullah 等人（2024）。与大多数先前的工作不同，我们选择不对语言模型本身进行语音任务的微调，因为这样做可能会导致非语音任务上的冲突。我们发现，在更大的模型规模下，即使没有这种微调，也能达到强大的性能；参见第 8.4 节。

10 Conclusion 结论

In many ways, the development of high-quality foundation models is still in its infancy. Our experience in developing Llama 3 suggests that substantial further improvements of these models are on the horizon. Throughout the development of the Llama 3 model family, we found that a strong focus on high-quality data, scale, and simplicity consistently yielded the best results. In preliminary experiments, we explored more complex model architectures and training recipes but did not find the benefits of such approaches to outweigh the additional complexity they introduce in model development.

在许多方面，高质量基础模型的发展仍处于起步阶段。我们在开发 Llama 3 的经验表明，这些模型的进一步重大改进即将到来。在整个 Llama 3 模型家族的开发过程中，我们发现对高质量数据、规模和简单性的强烈关注始终能产生最佳结果。在初步实验中，我们探索了更复杂的模型架构和训练方案，但没有发现这些方法的好处能超过它们在模型开发中引入的额外复杂性。

Developing a flagship foundation model such as Llama 3 involves overcoming a plethora of deep technical problems but also requires clever organizational decisions. For example, to ensure Llama 3 is not accidentally overfitted on commonly used benchmarks, our pre-training data was procured and processed by a separate team that was strongly incentivized to prevent contamination of that pre-training data with external benchmarks. As another example, we ensure that our human evaluations remain trustworthy by allowing only a small set of researchers who do not contribute to model development to perform and access these evaluations. While such organizational decisions are rarely discussed in technical papers, we found them to be pivotal to the successful development of the Llama 3 family of models.

开发像 Llama 3 这样的旗舰基础模型不仅涉及克服众多深层次技术问题，还需要巧妙的组织决策。例如，为了确保 Llama 3 不会意外地过度适应常用基准测试，我们的预训练数据由一个单独的团队采购和处理，该团队受到强烈激励以防止预训练数据受到外部基准的污染。再比如，我们通过只允许一小部分不参与模型开发的研究人员进行并访问这些评估，来确保我们的人工评估保持可信度。尽管这类组织决策在技术论文中很少被讨论，但我们发现它们对 Llama 3 系列模型的成功开发至关重要。

We shared the details of our development process because we believe this will: (1) help the larger research community understand the key factors of foundation model development and (2) contribute to a more informed debate about the future of foundation models in the general public. We also shared preliminary experiments with integrating multimodal capabilities into Llama 3. While these models are still under active development and not yet ready for release, we hope sharing our results early will accelerate research in this direction.

我们分享了开发过程的细节，因为我们相信这将：（1）帮助更大的研究社区理解基础模型开发的关键因素；（2）有助于公众对基础模型未来进行更明智的辩论。我们还分享了将多模态能力集成到 Llama 3 中的初步实验。虽然这些模型仍在积极开发中，尚未准备好发布，但我们希望早期分享我们的结果将加速这一方向的研究。

Following the positive outcomes of the detailed safety analyses presented in this paper, we publicly release our Llama 3 language models in order to accelerate the development of AI systems for a plethora of societally relevant use cases and enable the research community to scrutinize our models and identify ways to make these models better and safer. We believe that the public release of foundation models plays a key role in the responsible development of such models, and we hope that the release of Llama 3 encourages the industry to embrace the open, responsible development of AGI.

基于本文详细安全分析的积极结果，我们公开发布了 Llama 3 语言模型，以加速开发对社会相关用例有广泛影响的 AI 系统，并使研究社区能够仔细审查我们的模型并找出使这些模型更好、更安全的方法。我们相信，基础模型的公开发布在负责任的模型开发中起着关键作用，我们希望 Llama 3 的发布能鼓励业界接受开放、负责任的 AGI 开发。

Contributors and Acknowledgements 贡献者和致谢

Llama 3 is the result of the work of a large number of people at Meta. Below, we list all core contributors (people who worked on Llama 3 for at least \(2/3\mathrm{{rd}}\) of the runtime of the project) and contributors (people who worked on Llama 3 for at least \({}^{1}/\) 5th of the runtime of the project). We list all contributors in alphabetical order of first name.

Llama 3 是 Meta 众多人员工作的成果。以下，我们列出了所有核心贡献者（在项目运行时间内至少工作了 \(2/3\mathrm{{rd}}\) 的人员）和贡献者（在项目运行时间内至少工作了 \({}^{1}/\) 的五分之一的人员）。我们按名字首字母顺序列出了所有贡献者。

Core Contributors 核心贡献者

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livsh*ts, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Ksh*tiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier duch*enne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, and Zoe Papakipos.

Contributors 贡献者

Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hanco*ck, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi (Jack) Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu (Sid) Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao.

Acknowledgements 致谢

We thank Mark Zuckerberg, Chris Cox, Ahmad Al-Dahle, Santosh Jardanhan, Joelle Pineau, Yann LeCun, Aparna Ramani, Yee Jiun Song, and Ash Jhaveri for their invaluable support for Llama 3.

We also thank Aasish Pappu, Adebissy Tharinger, Adnan Aziz, Aisha Iqbal, Ajit Mathews, Albert Lin, Amar Budhiraja, Amit Nagpal, Amos Teo, Andrew Prasetyo Jo, Ankit Jain, Antonio Prado, Aran Mun, Armand Kok, Ashmitha Jeevaraj Shetty, Aya Ibrahim, Bardiya Sadeghi, Beibei Zhu, Bell Praditchai, Benjamin Muller, Botao Chen, Carolina Tsai, Cen Peng, Cen Zhao, Chana Greene, Chenguang Zhu, Christian Fuegen, Christophe Ropers, Christopher Luc, Cynthia Gao, Dalton Flanagan, Damien Sereni, Dan Johnson, Daniel Haziza, Daniel Kim, David Kessel, Divya Shah, Dong Li, Elisabeth Michaels, Elissa Jones, Emad El-Haraty, Eric Alamillo, Eric Hambro, Erika Lal, Eugen Hotaj, Fabian Gloeckle, Fadli Basyari, Faith Eischen, Fei Kou, Ferdi Adeputra, Feryandi Nurdiantoro, Flaurencya Ciputra, Forest Zheng, Francisco Massa, Furn Techaletumpai, Gobinda Saha, Gokul Nadathur, Greg Steinbrecher, Gregory Chanan, Guille Cobo, Guillem Brasó, Hakan Inan, Hany Morsy, Haonan Sun, Hardik Shah, Henry Erksine Crum, Hongbo Zhang, Hongjiang Lv, Hongye Yang, Hyunbin Park, Ian Graves, Jack Wu, Jack Zhang, Jalpa Patel, James Beldock, James Zeng, Janice Lam, Jeff Camp, Jesse He, Jilong Wu, Jim Jetsada Machom, Jinho Hwang, Jonas Gehring, Jonas Kohler, Jose Leitao, Josh Fromm, Juan Pino, Julia Rezende, Julian Garces, Kae Hansanti, Kartik Khandelwal, Keito Uchiyama, Kevin McAlister, Kody Bartelt, Kristina Pereyra, Kunhao Zheng, Lien Thai, Marco Campana, Mariana Velasquez, Marta R. Costa-jussa, Mayank Khamesra, Mengjiao MJ Wang, Mengqi Mu, Miao Liu, Michael Suo, Mikel Jimenez Fernandez, Mustafa Ozdal, Na Li, Nahiyan Malik, Naoya Miyanohara, Narges Torabi, Nathan Davis, Nico Lopero, Nikhil Mehta, Ning Li, Octary Azis, PK Khambanonda, Padchara Bubphasan, Pian Pawakapan, Prabhav Agrawal, Praveen Gollakota, Purin Waranimman, Qian Sun, Quentin Carbonneaux, Rajasi Saha, Rhea Nayak, Ricardo Lopez-Barquilla, Richard Huang, Richard Qiu, Richard Tosi, Rishi Godugu, Rochit Sapra, Rolando Rodriguez Antunez, Ruihan Shan, Sakshi Boolchandani, Sam Corbett-Davies, Samuel Djunaedi, Sarunya Pumma, Saskia Adams, Shankar Kalyanaraman, Shashi Gandham, Shengjie Bi, Shengxing Cindy, Shervin Shahidi, Shishir Patil, Sho Yaida, Shoubhik Debnath, Sirirut Sonjai, Srikanth Sundaresan, Stephanie Worland, Susana Contrera, Tejas Shah, Tony Cao, Tony Lee, Tristan Rice, Vishy Poosala, Vítor Albiero, Wenyu Chen, Wesley Lee, William Held, Xiaozhu Meng, Xinhua Wang, Xintian Wu, Yaroslava Kuzmina, Yifan Wang, Yu Zhao, Yue Zhao, Yun Wang, Zaibo Wang, and Zixi Qi for helpful contributions to Llama 3.

References 参考文献

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023.

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867, 2023.

Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M. Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. CoRR, abs/2402.01781, 2024. doi: 10.48550/ARXIV.2402.01781. https://doi.org/10.48550/arXiv.2402.01781.

Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019.

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088, 2023a.

Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes llm better reasoner. arXiv preprint arXiv:2310.20689, 2023b.

Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. Anthropic, April, 2024.

Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 929-947, 2024.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhang, Zhang Zhang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom

Brown, and Jared Kaplan. Constitutional AI: harmlessness from AI feedback. CoRR, abs/2212.08073, 2022. doi: 10.48550/ARXIV.2212.08073. https://doi.org/10.48550/arXiv.2212.08073.

Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia Gonzalez, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and Mary Williamson. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187, 2023.

Robin Battey and Sumit Gupta. Training llama: A storage perspective, 2024. https://atscaleconference.com/videos/ training-llama-a-storage-perspective/.

Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 21.6 b technical report. arXiv preprint arXiv:2402.17834, 2024.

Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. Worldsense: A synthetic benchmark for grounded reasoning in large language models. CoRR, abs/2311.15930, 2023. doi: 10.48550/ARXIV.2311.15930. https://doi.org/10.48550/arXiv.2311.15930.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. https://aclanthology.org/D13-1160.

Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724, 2023.

Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, et al. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161, 2024.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.

Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. How human is machine translationese? comparing human and machine translations of text and speech. In Marcello Federico, Alex Waibel, Kevin Knight, Satoshi Nakamura, Hermann Ney, Jan Niehues, Sebastian Stüker, Dekai Wu, Joseph Mariani, and Francois Yvon, editors, Proceedings of the 17th International Conference on Spoken Language Translation, pages 280-290, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.iwslt-1.34. https://aclanthology.org/2020.iwslt-1.34.

Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training, 2024. https://arxiv.org/abs/2406.03476.

Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie, Pietro Astolfi, Reyhane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri, Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, and Vikas Chandra. An introduction to vision-language modeling. 2024.

A.Z. Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), pages 21–29, 1997. doi: 10.1109/SEQUEN.1997.666900.

Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv:2202.07646, 2022. https://arxiv.org/abs/2202.07646.

Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation. IEEE Trans. Software Eng., 49(7):3675-3691, 2023.

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.

Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations, 2023. https://arxiv.org/abs/2310.20246.

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024.

Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pages 3915-3924. PMLR, 2022.

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1241. https://aclanthology.org/D18-1241.

Ju-Chieh Chou, Chung-Ming Chien, Wei-Ning Hsu, Karen Livescu, Arun Babu, Alexis Conneau, Alexei Baevski, and Michael Auli. Toward joint language modeling for speech units and text. 2023.

Arnab Choudhury, Yang Wang, Tuomas Pelkonen, Kutta Srinivasan, Abha Jain, Shenghao Lin, Delia David, Siavash Soleimanifard, Michael Chen, Abhishek Yadav, Ritesh Tijoriwala, Denis Samoylov, and Chunqiang Tang. MAST: Global scheduling of ml training across geo-distributed datacenters at hyperscale. In Proceedings from 18th USENIX Symposium on Operating Systems Design and Implementation, 2024.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. CoRR, abs/2210.11416, 2022. doi: 10.48550/ARXIV.2210.11416. https://doi.org/10.48550/arXiv.2210.11416.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.

Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798-805, 2023. doi: 10.1109/SLT54892.2023.10023141.

Marta R. Costa-jussà, Mariano Coria Meglioli, Pierre Andrews, David Dale, Prangthip Hansanti, Elahe Kalbassi, Alex Mourachko, Christophe Ropers, and Carleigh Wood. Mutox: Universal multilingual audio-based toxicity dataset and zero-shot detector. 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. 2023.

Databricks. Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs blog. https: //www.databricks.com/blog/mpt-7b, 2024.

DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024. https://arxiv.org/abs/2406.11931.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of llms: An exploration in mathematical problem solving. arXiv preprint arXiv:2405.12205, 2024.

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929, 2020.

Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368- 2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1246. https://aclanthology.org/N19-1246.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.

Hany Farid. An overview of perceptual hashing. Journal of Online Trust and Safety, 1(1), 2021.

Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Audiochatllama: Towards general-purpose speech abilities for llms. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5522-5532, 2024.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.

Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, Shuqiang Zhang, Mikel Jimenez Fernandez, Shashidhar Gandham, and Hongyi Zeng. RDMA over Ethernet for Distributed AI Training at Meta Scale. In ACM Special Interest Group on Data Communication (SIGCOMM), 2024. https://doi.org/10.1145/3651890.3672233.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.

Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations?, 2024.

Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, 2023. https://github.com/openlm-research/ open_llama.

Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023.

Gemini Team Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452, 2023.

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Aksh*ta Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024. https://arxiv.org/abs/2402.00838.

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.

Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to-speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. Changing answer order can decrease mmlu accuracy. arXiv preprint:2406.19470, 2024. https://arxiv.org/abs/2406.19470.

Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8342-8360. Association for Computational Linguistics, 2020. doi: 10.18653/V1/2020.ACL-MAIN.740. https://doi.org/10.18653/v1/2020.acl-main.740.

Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427-5444, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.438. https://aclanthology.org/2020.emnlp-main.438.

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509, 2022.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. https://openreview.net/forum?id=d7KBjm13GmQ.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021b. https://datasets-benchmarks-proceedings. neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican,

George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2019.

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuginne, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations. 2023.

Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In C. Maria Keet, Hung-Yi Lee, and Sina Zarriek, editors, Proceedings of the 16th International Natural Language Generation Conference, pages 28-53, Prague, Czechia, September 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.inlg-main.3. https://aclanthology.org/2023.inlg-main.3.

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization, 2019. https://arxiv.org/abs/1803.05407.

Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv preprint arXiv:2103.03206, 2021.

Meng Ji, Meng Ji, Pierrette Bouillon, and Mark Seligman. Cultural and Linguistic Bias of Neural Machine Translation Technology, page 100-128. Studies in Natural Language Processing. Cambridge University Press, 2023.

Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1215. https://aclanthology.org/D17-1215.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-547, 2019.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601- 1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. https://aclanthology.org/P17-1147.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431. Association for Computational Linguistics, April 2017.

Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pages 2410-2419. PMLR, 2018.

Gregory Kamradt. Llmtest_needleinahaystack. https://github.com/gkamradt/LLMTest_NeedleInAHaystack/blob/ main/README.md, 2023.

Wonjune Kang, Yun Wang, Shun Zhang, Arthur Hinsvark, and Qing He. Multi-task learning for front-end text processing in tts. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10796-10800, 2024. doi: 10.1109/ICASSP48485.2024.10446241.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.

Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, and Santu Rana. Alpaca against vicuna: Using llms to uncover memorization of llms, 2024. https://arxiv.org/abs/ 2403.04801.

Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 2023.

Aniruddha Kembhavi, Michael Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. ArXiv, abs/1603.07396, 2016. https://api.semanticscholar.org/CorpusID:2682274.

Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.03264, 2021.

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.324. https://aclanthology.org/2021.naacl-main.324.

Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code, 2022. https://arxiv.org/abs/2211.15533.

Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152-1157, 2016.

Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. https://proceedings.neurips.cc/paper_files/paper/ 2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1082. https://aclanthology.org/D17-1082.

Joel Lamy-Poirier. Breadth-first pipeline parallelism. Proceedings of Machine Learning and Systems, 5:48-67, 2023.

Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pages 18893-18912. PMLR, 2023.

Kevin Lee and Shubho Sengupta. Introducing the AI Research SuperCluster - Meta's cutting-edge AI supercomputer for AI research, 2022. https://ai.meta.com/blog/ai-rsc/.

Kevin Lee, Adi Gangidi, and Mathew Oldham. Building meta's genai infrastructure. 2024.

Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. In EMNLP, 2018.

Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pages 6265-6274. PMLR, 2021.

Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 2024a.

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash fa*ghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2024b. https://arxiv.org/abs/2406.11794.

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023a.

Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models, 2022. https://arxiv.org/abs/2208. 03306.

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023b.

Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. arXiv preprint arXiv:2402.19255, 2024c.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. CoRR, abs/2211.09110, 2022. doi: 10.48550/ARXIV.2211.09110. https://doi.org/10.48550/arXiv.2211.09110.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.

Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.

Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. \({arXiv}\) preprint arXiv:2310.01889, 2023a.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023b.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023c.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.

Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou,and Andrew M. Dai. Best practices and lessons learned on synthetic data for language models. CoRR, abs/2404.07503, 2024b. doi: 10.48550/ARXIV.2404.07503. https://doi.org/10.48550/arXiv.2404.07503.

Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning, 2024c. https://arxiv.org/abs/2312.15685.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019a.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019b. http://arxiv.org/abs/1907.11692.

Llama-Team. Meta llama guard 2. https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/ MODEL_CARD.md, 2024.

Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Instag: Instruction tagging for analyzing supervised fine-tuning of large language models, 2023.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.556. https://aclanthology.org/2022.acl-long.556.

Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583, 2023.

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In \({ACL},{2024}\) .

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024a.

Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. arXiv preprint arXiv:2406.10229, 2024b.

Neelu Madan, Andreas Moegelmose, Rajat Modi, Yogesh S. Rawat, and Thomas B. Moeslund. Foundation models for video understanding: A survey. 2024.

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.

Soumi Maiti, Yifan Peng, Shukjae Choi, Jee weon Jung, Xuankai Chang, and Shinji Watanabe. Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks. 2023.

Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.177. https://aclanthology.org/2022.findings-acl.177.

Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2199-2208, 2020. https://api.semanticscholar.org/CorpusID:220280200.

Jeremy Baumgartner Matt Bowman. Meta open compute project, grand teton ai platform, 2022. https://engineering. fb.com/2022/10/18/open-source/ocp-summit-2022-grand-teton/.

Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open-source training and inference framework. arXiv preprint arXiv:2404.14619, 2024.

Dheeraj Mekala, Jason Weston, Jack Lanchantin, Roberta Raileanu, Maria Lomeli, Jingbo Shang, and Jane Dwivedi-Yu. Toolverifier: Generalization to new tools via self-verification. arXiv preprint arXiv:2402.14158, 2024.

Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023a.

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023b.

Sabrina J. Mielke, Arthur Szlam, Y-Lan Boureau, and Emily Dinan. Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness. CoRR, abs/2012.14983, 2020. https://arxiv.org/abs/ 2012.14983.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1260. https://aclanthology.org/D18-1260.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. Reframing instructional prompts to GPTk's language. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 589-612, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.50. https://aclanthology.org/2022.findings-acl.50.

Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024.

Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites, 2015. https://arxiv.org/abs/1504. 04909.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991-16111, 2023.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zahariat. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15, 2021.

Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. \({ArXiv}\) , abs/2311.17035, 2023. https://api.semanticscholar.org/CorpusID:265466445.

Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. Spirit-Im: Interleaved spoken and written language model. 2024.

Marta R. Costa-jussà NLLB Team, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation. 2022.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023a.

OpenAI. GPT-4 blog. https://openai.com/index/gpt-4-research/, 2023b.

OpenAI. simple-evals. https://github.com/openai/simple-evals, 2024.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.

Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the Landscape of Diverse Automated Correction Strategies. Trans. Assoc. Comput. Linguistics, 12:484-506, 2024. doi: 10.1162/TACL_A\_00660. https://doi.org/10.1162/tacl_a_00660.

Satadru Pan Pan, Theano Stavrinos, Yunqiao Zhang, Atul Sikaria, Pavel Zakharov, Abhinav Sharma, Shiva Shankar, Mike Shuey, Richard Wareing, Monika Gangapuram, Guanglei Cao, Christian Preseau, Pratap Singh, Kestutis Patiejunas, JR Tipton, Ethan Katz-Bassett, and Wyatt Lloyd. Facebook's tectonic filesystem: Efficiency from exascale. In Proceedings of the 19th USENIX Conference on File and Storage Technologies, pages 217-231, 2021.

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.

Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5336-5358, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.391. https://aclanthology.org/2022.naacl-main.391.

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024.

Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022.

Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.

Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self-supervised descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14532-14542, 2022.

B.T. Polyak. New stochastic approximation type procedures. Automation and Remote Control, 7(7), 1991.

Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020.

Prokopis Prokopidis, Vassilis Papavassiliou, and Stelios Piperidis. Parallel global voices: a collection of multilingual corpora with citizen media stories. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Äsuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may 2016. European Language Resources Association (ELRA). ISBN 978-2-9517408-9-1.

Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In NeurIPS, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on

Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492-28518. PMLR, 23-29 Jul 2023. https://proceedings.mlr.press/v202/radford23a.html.

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. ArXiv, abs/2112.11446, 2021. https://api.semanticscholar.org/CorpusID:245353475.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. https://arxiv.org/abs/1910.02054.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. https://aclanthology.org/D16-1264.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. https://aclanthology.org/P18-2124.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. https://arxiv.org/abs/2311. 12022.

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training, 2021. https://arxiv.org/abs/2101.06840.

Joshua Robinson and David Wingate. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. https://openreview.net/pdf?id=yKbprarjc5B.

Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/2308.12950, 2023. doi: 10.48550/ARXIV.2308.12950. https://doi.org/10.48550/arXiv.2308.12950.

Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield,

James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Velimirović, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, and Christian Frank. Audiopalm: A large language model that can speak and listen. 2023.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.

Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts, 2024. https://arxiv.org/abs/2402.16822.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harsh*t Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Ändrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. https://openreview.net/forum?id=9Vrb9D0WI4.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454. https://aclanthology.org/D19-1454.

Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845–874, 08 2021. ISSN 2307-387X. doi: 10.1162/ tacl_a_00401. https://doi.org/10.1162/tacl_a_00401.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Seamless Communication, Loic Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Celebi Onur Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. Seamlessm4t massively multilingual & multimodal machine translation. ArXiv, 2023.

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196, 2023.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners, 2022. https://arxiv.org/abs/2210.03057.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2019. http://arxiv.org/abs/1909.08053.

Aaditya Singh, Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. Evaluation data contamination in llms: how do we measure it and (when) does it matter? 2024.

Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019.

Snowflake. Snowflake Arctic: The Best LLM for Enterprise AI - Efficiently Intelligent, Truly Open blog. https: //www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/, 2024.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048-6058, 2023.

Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Damon Mosk-Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang. Nexusraven: a commercially-permissive language model for function calling. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.824. https://aclanthology.org/2023.findings-acl. 824.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. https://aclanthology.org/N19-1421.

Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl. Holistic Configuration Management at Facebook. In Proceedings of the 25th Symposium on Operating Systems Principles, pages 328-343, 2015.

Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. 2024.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.

David Thiel. Identifying and eliminating csam in generative ml training data and models. Technical report, Stanford Internet Observatory, 2023.

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022. https://arxiv.org/abs/2201.08239.

Jörg Tiedemann. Parallel data, tools and interfaces in opus. In International Conference on Language Resources and Evaluation, 2012. https://api.semanticscholar.org/CorpusID:15453873.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.

Bertie Vidgen, Adarsh Agrawal, Ahmed M Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Borhane Blili-Hamelin, et al. Introducing v0.5 of the ai safety benchmark from mlcommons. arXiv preprint arXiv:2404.12241, 2024.

Saranyan Vigraham and Benjamin Leonhardi. Maintaining large-scale ai capacity at meta. 2024.

Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions, 2024. https://arxiv.org/abs/2404.13208.

Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390, 2021a.

Changhan Wang, Anne Wu, and Juan Pino. Covost 2 and massively multilingual speech-to-text translation. \({arXiv}\) preprint arXiv:2007.10310, 2021b.

Haochun Wang, Sendong Zhao, Zewen Qiang, Bing Qin, and Ting Liu. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models. CoRR, abs/2402.01349, 2024a. doi: 10.48550/ARXIV.2402.01349. https://doi.org/10.48550/arXiv.2402.01349.

Jun Wang, Benjamin Rubinstein, and Trevor Cohn. Measuring and mitigating name biases in neural machine translation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2576-2590, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.184. https://aclanthology.org/2022.acl-long. 184.

Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935, 2023a.

Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Unified codec language models for speech recognition, synthesis, and translation. 2023b.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on \({1600} +\) nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, 2022b.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024b.

Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814, 2017.

Lucas Weber, Elia Bruni, and Dieuwke Hupkes. Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning. In Jing Jiang, David Reitter, and Shumin Deng, editors, Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 294-313, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.conll-1.20. https://aclanthology.org/2023. conll-1.20.

Lucas Weber, Elia Bruni, and Dieuwke Hupkes. The icl consistency test. arXiv preprint arXiv:2312.04945, 2023b.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022b. https://openreview.net/forum?id=yzkSU5zdwD.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022c.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct, 2024. https://arxiv.org/abs/2312.02120.

Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053, 2022.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data, 2019. https: //arxiv.org/abs/1911.00359.

Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. https://arxiv.org/ abs/2203.05482.

Chunyang Wu, Zhiping Xiu, Yangyang Shi, Ozlem Kalinli, Christian Fuegen, Thilo Koehler, and Qing He. Transformer-based acoustic modeling for streaming speech synthesis. In Interspeech, pages 146-150, 2021.

Haoyi Wu, Wenyang Hui, Yezeng Chen, Weiqi Wu, Kewei Tu, and Yi Zhou. Conic10k: A challenging math problem understanding and reasoning dataset, 2023. https://arxiv.org/abs/2311.05113.

Zhibiao Wu and Martha Palmer. Verb semantics and lexical selection. In ACL, 1994.

XAI. Open Release of Grok-1 blog. https://x.ai/blog/grok-os, 2024.

Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. 2024a.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024b.

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, 2021.

Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451, 2024.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Ksh*tiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023.

Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023.

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_ leaderboard.html, 2024.

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023a.

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. 2023b.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality. 2023.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.

Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024a.

Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548, 2024b.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.

Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang,Zhiyuan Liu,et al. \(\infty\) bench: Extending long context evaluation beyond \({100}\mathrm{k}\) tokens. arXiv preprint arXiv:2402.13718, 2024.

Xinyu Zhang, Ian Colbert, Ken Kreutz-Delgado, and Srinjoy Das. Training deep neural networks with joint quantization and pruning of weights and activations, 2021.

Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1131. https://aclanthology.org/N19-1131.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. \(\operatorname{arXiv}\) preprint arXiv:2303.18223, 2023a. http://arxiv.org/abs/2303.18223.

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023b.

Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In arXiv preprint arXiv:2212.04501, 2022.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International

Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR, 2021. http://proceedings.mlr.press/v139/zhao21c.html.

Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. CoRR, abs/2309.03882, 2023. doi: 10.48550/ARXIV.2309.03882. https://doi.org/10.48550/arXiv. 2309.03882.

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.

Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. 2023.