The Pile is a large-scale corpus for language model training, consisting of 885 GB of text from various sources, including books, articles, and websites.
Hugging Face Datasets provides a wide range of large-scale text corpora for language model training, including the popular Wikipedia and BookCorpus datasets.
Common Crawl is a non-profit organization that provides large-scale web corpora for language model training, with over 24 terabytes of text data available.
The Stanford Natural Language Processing Group provides a range of resources for language model training, including large-scale text corpora and pre-trained models.
Google has released a massive text corpus for language model training, consisting of over 45 terabytes of text data from various sources, including books and articles.
The OpenWebText Corpus is a large-scale corpus for language model training, consisting of over 38 GB of text from the web, with a focus on diversity and representativeness.
The Wikipedia Corpus is a large-scale text corpus that can be used for language model training, with over 50 million articles in hundreds of languages.
This survey provides an overview of large-scale text corpora for language model training, including the Pile, Common Crawl, and Wikipedia corpora, and discusses their strengths and limitations.