SlimPajama: A 627B token, cleaned and deduplicated version of RedPajama

	Tokens	Open source	Curated data sources	Deduplication level
SlimPajama	627B	Yes	Yes	Extensive
RedPajama	1.21T	Yes	Yes	Partial
RefinedWeb-600B	600B	Yes	No	Extensive
RefinedWeb-5T	5T	No	No	Extensive
LLaMA	1.4T	No	Yes	Partial
MPT	1T	No	Yes	Partial
MassiveText	1.4T	No	Yes	Extensive

SlimPajama

Since LLaMA (Touvron et al. 2023) demonstrated the power of training smaller models on more than 1T tokens, there has been a push in the open-source community to replicate and extend these findings. The LLaMA data collection methodology was quickly replicated and released as the RedPajama 1.2T token open-source dataset.

Our original intention was to use the RedPajama data as-is, but upon analysis we discovered some corpora contained missing files while others had a large percentage of duplicates. RedPajama followed the deduplication guidelines in the LLaMA paper, which was less strict and only operated within each data source, not between them. To improve compute efficiency and data quality, we decided to produce a cleaned and extensively deduplicated version of the data, which led to the development of SlimPajama.

To produce SlimPajama, we first removed short, low quality documents from RedPajama. After removing punctuation, space symbols, newlines and tabs, we filtered out documents with less than 200 characters. These documents typically contain only meta data and no useful information. Low-length filter was applied to every corpora other than Books and GitHub where we found useful short documents. In total this removed 1.86% of documents from RedPajama.

Data source	Document low-length filter rate
Commoncrawl	0.02%
C4	4.7%
GitHub	0.00%
Books	0.00%
ArXiv	0.62%
Wikipedia	0.00%
StackExchange	0.32%
Total	1.86%

Training on deduplicated data makes language models better by improving training compute efficiency and reducing the chance of models producing text memorized from the training data (Penedo et al., 2023; Abbas et al., 2023; Lee et al., 2021; Holtzman, 2019; Face, 2023). Every corpus contained duplicates with the most significant duplication found in CommonCrawl and GitHub. Penedo et al. (2023) found similar rates of duplication in CommonCrawl data. In total, we were able to prune 49.6% of bytes from RedPajama, leaving us with the 627B token SlimPajama dataset.

To perform deduplication we used MinHashLSH Leskovec et al. (2014) with a Jaccard similarity threshold of 0.8. We construct document signatures on top of pre-processed lower-cased 13-grams. Pre-processing includes removing punctuation, consecutive spaces, newlines and tabs. We also strip the documents to remove any leading or trailing escape characters. Deduplication was performed both within and between data sources.

Data source	Byte duplication rate
Commoncrawl	63.76%
C4	6.85%
GitHub	46.16%
books	2.01%
ArXiv	0.06%
Wikipedia	2.24%
StackExchange	0.20%
Total	49.60%

Below we compare the data compositions of several related datasets RedPajama (2023); Touvron et al. (2023); Team (2023); Penedo et al. (2023). SlimPajama has comparatively lower proportions of web data than RedPajama and RefinedWeb, and comparatively higher proportions of high quality data from Books, arXiv, and Wikipedia. While Penedo et al. (2023) rightly point out that purely filtering web data alone is more scalable, SlimPajama provides higher quality data through curation of sources.

Data source	SlimPajama	RedPajama	LLaMA	MPT	RefinedWeb	MassiveText
Commoncrawl	52.2%	72.6%	67.0%	10.0%	100%	0.0%
C4	26.7%	14.4%	15.0%	0.0%	0.0%	10.0%
GitHub	5.2%	4.9%	4.5%	0.0%	0.0%	4.0%
Books	4.2%	2.1%	4.5%	3.0%	0.0%	30.0%
ArXiv	4.6%	2.3%	2.5%	1.9%	0.0%	0.0%
Wikipedia	3.8%	2.0%	4.5%	4.0%	0.0%	1.0%
StackExchange	3.3%	1.7%	2.0%	1.4%	0.0%	0.0%
mC4 3.1.0 – English (200+ words)	0.0%	0.0%	0.0%	33.0%	0.0%	0.0%
C4 – English – SemDedup 80%	0.0%	0.0%	0.0%	29.9%	0.0%	0.0%
The Stack – Selected Languages	0.0%	0.0%	0.0%	10.0%	0.0%	0.0%
The Stack – Markdown	0.0%	0.0%	0.0%	3.5%	0.0%	0.0%
Semantic Scholar ORC	0.0%	0.0%	0.0%	3.3%	0.0%	0.0%
MassiveWeb	0.0%	0.0%	0.0%	0.0%	0.0%	45.0%
News	0.0%	0.0%	0.0%	0.0%	0.0%	10.0%

Similar to RedPajama, there can be distributional bias in the SlimPajama dataset that can manifest in various forms in the downstream model deployment. There are other risks associated with large language models such as amplifying social stereotypes, memorizing training data, or revealing private or secure information.

SlimPajama Data Processing Pipeline

The existing open-source infrastructure for pre-processing text datasets did not scale to datasets with 1 Trillion tokens. The most significant bottlenecks occurred within the interleaving, shuffling and deduplication steps. Our optimizations are inspired by producer-consumer patterns that we implemented using a standard multiprocessing library available in Python. We also had to rewrite the datasketch (Zhu 2023) implementation to reduce both required memory and make the code more efficient in a distributed setting. Details of our implementation are available at https://github.com/Cerebras/modelzoo/tree/main/modelzoo/transformers/data_processing/slimpajama. The end-to-end pre-processing took 2.5 days with a 64 core CPU. The largest memory consumption observed was 1.4 TB.

Next Steps

We built SlimPajama with the intention of training a LLaMA style model for our partner Opentensor. While we expect training to begin in the coming weeks, both Opentensor and Cerebras believe it would be beneficial to the ML community to release the dataset first. The team at Cerebras also believes opensourcing our Trillion-token scale, data processing library will help others to further improve open source LLM datasets.

Authors

Cerebras: Daria Soboleva*, Faisal Al-Khateeb*, Joel Hestness, Nolan Dey
Opentensor: Robert Myers, Jacob Robert Steeves
*Equal contribution

Resources

• • SlimPajama dataset
  • Libraries for data pre-processing
  • Discord
  • Opentensor / Bittensor
    

Acknowledgements

We’d like to thank Together, Ontocord.ai,  ETH DS3Lab , AAI CERC Lab for creating the original RedPajama dataset and releasing it open source.
This release was made possible with the support and collaboration of Opentensor.
Easy cloud access to Cerebras systems is provided by our partner Cirrascale.

5 Citation

To cite our work please use:

@misc{cerebras2023slimpajama,
author = {Soboleva, Daria and Al-Khateeb, Faisal and Myers, Robert and Steeves, Jacob R and Hestness, Joel and Dey, Nolan},
title = {{SlimPajama: A 627B token cleaned and deduplicated version of RedPajama}},
month = June,
year = 2023,
howpublished = {\url{https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama}},
url = {https://huggingface.co/datasets/cerebras/SlimPajama-627B},
}

References

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. SemDeDup: Data-efficient learning at web-scale through semantic deduplication. 2023. URL https://arxiv.org/abs/2303.09540.

Together Computer. RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset, 2023. URL https://github.com/togethercomputer/RedPajama-Data.

Hugging Face. Large-scale Near-deduplication Behind BigCode, 2023. URL https://huggingface.co/ blog/dedup.

Ari Holtzman. The Curious Case of Neural Text Degeneration. 2019. URL https://arxiv.org/abs/1904. 09751.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better. 2021. URL https://arxiv.org/abs/2107.06499.

Jure Leskovec, Anand Rajaraman, and Jeffrey D. Ullman. Mining of Massive Datasets. Cambridge University Press, 2014.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. 2023. URL https://arxiv.org/abs/2306.01116.

The MosaicML NLP Team. Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs, 05 2023. URL https://www.mosaicml.com/blog/mpt-7b.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. 2023. URL https://arxiv.org/abs/2302.13971.

Eric Zhu. datasketch: Big Data Looks Small — datasketch 1.5.9 documentation. http://ekzhu.com/datasketch/, 2023.

Our pre-processing pipeline consists of multiple stages such as NFC normalization, cleaning, deduplication, document interleaving, document shuffling, split into train and holdout sets, deduplication of train set against holdout. All these steps are presented at Figure 1. Additional steps such as tokenization, sequence packing, sequence-level shuffling as well as upsampling can be performed using our scripts located at https://github.com/Cerebras/modelzoo/tree/main/modelzoo/transformers/data_processing/slimpajama. All steps here assume that the whole dataset cannot fit in the available RAM and distributed across multiple processes. We are welcoming any additional datasets preparation steps or suggestions on how to make this even more efficient on the large scale datasets!