Qwen/Qwen2-72B-Instruct cover image
featured

Qwen/Qwen2-72B-Instruct

The 72 billion parameter Qwen2 excels in language understanding, multilingual capabilities, coding, mathematics, and reasoning.

The 72 billion parameter Qwen2 excels in language understanding, multilingual capabilities, coding, mathematics, and reasoning.

Public
$0.56/$0.77 in/out Mtoken
bfloat16
32k
Function
ProjectLicense
Qwen/Qwen2-72B-Instruct cover image

Qwen2 72b

Ask me anything

0.00s

Qwen2-72B

Introduction

Qwen2 is the new series of Qwen large language models. For Qwen2, we release a number of base language models and instruction-tuned language models ranging from 0.5 to 72 billion parameters, including a Mixture-of-Experts model. This repo contains the 72B Qwen2 base language model.

Compared with the state-of-the-art opensource language models, including the previous released Qwen1.5, Qwen2 has generally surpassed most opensource models and demonstrated competitiveness against proprietary models across a series of benchmarks targeting for language understanding, language generation, multilingual capability, coding, mathematics, reasoning, etc.

For more details, please refer to our blog, GitHub, and Documentation.

Model Details

Qwen2 is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. It is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes.

Performance

The evaluation of base models mainly focuses on the model performance of natural language understanding, general question answering, coding, mathematics, scientific knowledge, reasoning, multilingual capability, etc.

The datasets for evaluation include:

English Tasks: MMLU (5-shot), MMLU-Pro (5-shot), GPQA (5shot), Theorem QA (5-shot), BBH (3-shot), HellaSwag (10-shot), Winogrande (5-shot), TruthfulQA (0-shot), ARC-C (25-shot)

Coding Tasks: EvalPlus (0-shot) (HumanEval, MBPP, HumanEval+, MBPP+), MultiPL-E (0-shot) (Python, C++, JAVA, PHP, TypeScript, C#, Bash, JavaScript)

Math Tasks: GSM8K (4-shot), MATH (4-shot)

Chinese Tasks: C-Eval (5-shot), CMMLU (5-shot)

Multilingual Tasks: Multi-Exam (M3Exam 5-shot, IndoMMLU 3-shot, ruMMLU 5-shot, mMMLU 5-shot), Multi-Understanding (BELEBELE 5-shot, XCOPA 5-shot, XWinograd 5-shot, XStoryCloze 0-shot, PAWS-X 5-shot), Multi-Mathematics (MGSM 8-shot), Multi-Translation (Flores-101 5-shot)

Qwen2-72B performance

DatasetsDeepSeek-V2Mixtral-8x22BLlama-3-70BQwen1.5-72BQwen1.5-110BQwen2-72B
ArchitectureMoEMoEDenseDenseDenseDense
#Activated Params21B39B70B72B110B72B
#Params236B140B70B72B110B72B
English
MMLU78.577.879.577.580.484.2
MMLU-Pro-49.552.845.849.455.6
GPQA-34.336.336.335.937.9
Theorem QA-35.932.329.334.943.1
BBH78.978.981.065.574.882.4
HellaSwag87.888.788.086.087.587.6
WindoGrande84.885.085.383.083.585.1
ARC-C70.070.768.865.969.668.9
TruthfulQA42.251.045.659.649.654.8
Coding
HumanEval45.746.348.246.354.364.6
MBPP73.971.770.466.970.976.9
EvalPlus55.054.154.852.957.765.4
MultiPL-E44.446.746.341.852.759.6
Mathematics
GSM8K79.283.783.079.585.489.5
MATH43.641.742.534.149.651.1
Chinese
C-Eval81.754.665.284.189.191.0
CMMLU84.053.467.283.588.390.1
Multilingual
Mulit-Exam67.563.570.066.475.676.6
Multi-Understanding77.077.779.978.278.280.7
Multi-Mathematics58.862.967.161.764.476.0
Multi-Translation36.023.338.035.636.237.8