WanJuan-CC Dataset

WanJuan 2.0
Dataset

WanJuan2.0 (WanJuan-CC) is a high-quality English web text dataset of 1T Tokens obtained from CommonCrawl. Released by the Shanghai Artificial Intelligence Laboratory, it has undergone a rigorous quality filtering pipeline, including URL filtering, text extraction, deduplication, quality assessment, and more.

Get the Dataset Now Online Preview

1T Tokens English Text CommonCrawl Apache-2.0

📊

Total Tokens

🌍

English Text

Language Type

🕸️

CommonCrawl

Data Source

📜

Apache-2.0

Open Source License

Dataset Highlights

Wanjun 2.0 is a carefully designed quality filtering pipeline that provides high-quality English corpus for pre-training large language models.

🚀

Trillion-scale Corpus

Contains high-quality English text with 1 trillion tokens, providing ample data support for pre-training large-scale language models to meet the training needs of GPT-level models.

🔧

Strict Quality Pipeline

Undergoes multi-stage refined processing including URL filtering, text extraction, content deduplication, and quality scoring to ensure the high quality and usability of each data point.

🕸️

CommonCrawl Source

Extracted from CommonCrawl, the world's largest open web crawling archive, covering various types of web pages including news, encyclopedias, academic articles, and blogs.

📖

Open Source Apache-2.0

Uses a permissive Apache-2.0 open source license, allowing commercial use, modification, and redistribution, lowering the barrier to use.

🤖

LLM Pre-training Optimization

The dataset is specifically designed and optimized for pre-training scenarios of large language models, with targeted data cleaning and quality screening, directly compatible with mainstream training frameworks.

📄

Comprehensive Documentation

Accompanied by detailed methodology papers and data processing flow descriptions, clearly documenting the strategies and parameters of each filtering stage for easy reproduction and secondary research.

Applicable Scenarios

From large model pre-training to data quality research, covering various cutting-edge NLP research directions

🧠

LLM Pre-training

Providing trillion-level English pre-training corpus for large language models like GPT and LLaMA, supporting large-scale language model training from scratch

🔬

Text Quality Research

Studying the effectiveness of web text quality filtering strategies, analyzing the impact of different filtering methods on downstream model performance

📊

NLP Benchmark Testing

Using large-scale English corpus to build and evaluate NLP task benchmarks, measuring model performance on various natural language processing tasks

🔍

Data Curation Research

Studying the impact of data quality on model performance, exploring optimal data selection and ratio strategies, guiding pre-training dataset construction

English Corpus CommonCrawl Pre-training Large Language Models Text Quality Data Filtering

Data Preview

The following is a JSONL format example of the WanJuan-CC dataset, each record contains text content, source URL, and quality score

JSONL

{"text": "The development of large language models has transformed natural language processing...", "url": "https://example.com/article1", "quality_score": 0.92}
{"text": "Recent advances in deep learning architectures have enabled...", "url": "https://example.com/article2", "quality_score": 0.88}
{"text": "Machine learning applications in healthcare continue to expand...", "url": "https://example.com/article3", "quality_score": 0.95}

3 Steps to Get Started Quickly

From browsing to training, you can start your large model pre-training project in just a few minutes

Browse Dataset

View the details of the WanJuan-CC dataset on the Ace Data Cloud platform, including metadata such as data scale, format description, and licensing agreements.

Download Data Shards

Obtain data files through the download methods provided by the platform, supporting streaming or bulk downloads to accommodate different storage and bandwidth conditions.

Tokenization and Training

Perform Tokenize processing on the downloaded text data, and you can directly input it into your large language model training process to start pre-training experiments.

Get Dataset

Start Exploring WanJuan-CC Data

Trillions of high-quality English corpus, Apache-2.0 open source license, available immediately. Whether you are a large language model researcher or a pre-training engineer, WanJuan-CC is an ideal choice for training data.

Get Dataset Now

Dataset

WanJuan 2.0 Dataset

WanJuan2.0 (WanJuan-CC) is a high-quality English web text dataset of 1T tokens obtained from CommonCrawl. The results show that, compared to various open-source English CC corpora, WanJuan2.0 exhibits higher safety in different dimensions of the Perspective API evaluation. Additionally, its practicality is demonstrated through perplexity (PPL) on 4 validation sets and accuracy on 6 downstream tasks. WanJuan2.0 shows competitive PPL performance across various validation sets, especially on datasets like tiny-storys that require higher language fluency. By comparing with similar datasets in 1B model training, using perplexity of the validation dataset and accuracy of downstream tasks as evaluation metrics, experiments prove that WanJuan2.0 significantly enhances the performance of English text completion and general English capability tasks.

—Records

—File size

—Delivery formats

—Data updated

WanJuan2.0(WanJuan-CC)

WanJuan 2.0Dataset