LLM & VLM courseBlog
34Track 3: Pretraining and scale · session 4 of 22

Pretraining data: from web crawl to training mixture

A base model can only learn what is in its training data. This session follows text from a raw web crawl to the final training mixture: how pages become clean text, how rules and small classifiers throw away bad pages, how MinHash finds near-copies among billions of documents, and how we decide how much of each source to show the model. We end with synthetic data and with test-set contamination, the quiet bug that makes benchmark scores lie.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: BeginnerBefore this: Sessions 6, 8 and 31; session 35 builds on this one

What you will learn

  • Where pretraining text comes from (web, books, code, papers, math, many languages) and why the web dominates
  • The cleaning pipeline: text extraction, URL filters, language ID, rule filters (Gopher, C4, FineWeb) and model-based quality classifiers
  • Exact deduplication with hashes, and near-duplicate detection with MinHash: why the collision probability equals the Jaccard similarity
  • LSH banding: how b bands of r rows give an S-shaped curve with a threshold near (1/b)^(1/r), and how FineWeb and Lee et al. chose b and r
  • Data mixtures: temperature sampling, epochs per source, the GPT-3 and Llama 3 mixes, annealing on high-quality data
  • Synthetic data and contamination: what each one helps or breaks, and how to check a benchmark with n-gram overlap

Why this matters

A language model is a compressed copy of its training data. The architecture decides how well it can learn; the data decides what it learns. Two models with the same size and the same compute can differ a lot only because one was trained on cleaner text.

In session 31 we fixed the task: predict the next token. In session 33 we saw that RoBERTa beat BERT partly by using more data. This session answers the question every pretraining team spends most of its time on: which tokens do we feed the model, and how many times each?

The scale is large. FineWeb, an open English web dataset, has 15 trillion tokens taken from 96 Common Crawl snapshots [Certain]. Llama 3 was pretrained on about 15 trillion tokens [Certain]. No human can read even a tiny part of this. Every decision must be made by programs: rules, hash functions and small classifiers. If they are wrong, nobody notices until the model is trained.

Bad data costs in three ways. Duplicates make the model memorize: Lee et al. found a single 61-word sentence repeated more than 60,000 times in C4, and showed that deduplicated training makes models emit memorized text about ten times less often [Certain]. Junk (menus, logs, spam) wastes compute on tokens that teach nothing. Contamination (benchmark questions inside the training data) makes evaluation scores look better than the model really is. The next session, session 35, tells you how many tokens you need; this one tells you how to get good ones.

What goes in: the sources

Pretraining data is a mix of sources. Each one teaches something different:

SourceWhat it teachesProblems
Web pages (Common Crawl)Almost every topic and style; the largest source by farMostly boilerplate, spam and copies; needs heavy filtering
BooksLong, coherent text; long-range structureCopyright and licensing; small compared to the web
Code (public repositories)Exact syntax, logic, step-by-step structureLicenses, generated files, many forks of the same project
Scientific papers (arXiv and similar)Technical language, formulas in LaTeXPDF extraction errors; tables and math break easily
Math (web pages and problem sets)Multi-step reasoning, notationRare on the web; easy to lose in HTML extraction
Many languagesUse in other languages; transfer between languagesVery uneven sizes: English is much larger than most languages
Encyclopedias, forums, Q&AFacts; question-and-answer patternsSmall; forums contain toxic text

Two real recipes show how different the shares can be. GPT-3 (2020) used five sources [Certain]:

GPT-3 sourceTokensWeight in the mixEpochs in 300B tokens
Common Crawl (filtered)410B60%0.44
WebText219B22%2.9
Books112B8%1.9
Books255B8%0.43
Wikipedia3B3%3.4

Look at the weights. Common Crawl is 82% of all available tokens (410 of 499 billion), but only 60% of the training mix. Wikipedia is 0.6% of the tokens but 3% of the mix, so it is seen more than three times. The weights are a choice, not a fact about the data. (The table is copied from the paper; its rounded numbers do not reproduce every epoch exactly, for example \(0.22 \times 300 / 19 = 3.5\) for WebText2, not 2.9.)

Llama 3 (2024) describes its final mix by content instead of by source: about 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual tokens [Certain]. Compared with GPT-3, much more code and math. Teams found that code and math help reasoning in general, not only on code and math tasks [Likely].

From crawl to clean text

Common Crawl is a non-profit that crawls the web and publishes each crawl as a snapshot, a dump of billions of pages [Likely]. Each snapshot comes in two main formats: WARC files (the raw HTTP responses, with full HTML) and WET files (text that Common Crawl already extracted). FineWeb extracted text from WARC files with the trafilatura library, because the WET text “retained too much boilerplate and menu text” [Certain]. DCLM did the same with another extractor (resiliparse) and found it better than WET by at least 2.5 points on its benchmark average [Certain]. The extractor is the first filter, and one of the most important.

A typical pipeline, here in the order FineWeb used, then the extra steps many teams add. Click a stage in Figure 1.

Figure 1. A web data pipeline. Stages 1 to 8 follow FineWeb; stages 9 to 11 are common additions (FineWeb-Edu, DCLM and Llama 3 use model-based filters; every serious team checks for contamination). Token counts are the ones FineWeb reports, in GPT-2 tokens.
How to read Figure 1

Click any box or use Next. The readout says what the stage does, why, and (where the paper gives it) how much data is left. Notice how much is removed: after the cheap first filters FineWeb had about 36 trillion tokens, after deduplication about 20 trillion, and the final dataset is 15 trillion. The cheap stages run first, because they shrink the data before the expensive ones (MinHash, classifiers) run.

URL filtering and language identification

Two cheap filters run early. A URL blocklist removes known adult and malware domains (FineWeb used the UT1 list) [Certain]. Language identification (language ID) is a small text classifier that predicts the language of a page and a confidence score. FineWeb used a fastText classifier and kept pages with an English score of at least 0.65 [Certain]. For multilingual datasets the same classifier sorts pages into one bucket per language; mC4, used to train mT5, covers 101 languages [Certain].

Rule filters: cheap tests for junk

Most web pages that survive extraction are still not good text: navigation bars, product lists, server logs, hashtag spam, templates. Simple rules catch most of them. A rule filter computes a few numbers per document and removes it if one number is out of range. The best-known set comes from Gopher’s MassiveWeb [Certain]:

  • between 50 and 100,000 words;
  • mean word length between 3 and 10 characters;
  • at most 0.1 “#” symbols or ellipses per word;
  • at most 90% of lines starting with a bullet, at most 30% ending with an ellipsis;
  • at least 80% of words contain a letter;
  • at least two of the stop words the, be, to, of, and, that, have, with (a cheap “is this real English prose?” test);
  • repetition limits, for example at most 30% duplicate lines.

C4 added its own rules (remove pages containing “lorem ipsum”, curly brackets or “javascript”, keep only lines ending in punctuation). FineWeb kept the C4 rules except the punctuation rule, which alone removed about 30% of tokens, and added three new ones [Certain]: remove a document if at most 12% of its lines end with punctuation, if at least 10% of its characters are in duplicated lines, or if at least 67% of its lines are shorter than 30 characters. Together these three removed about 22% of tokens [Certain].

Try the rules on ten small documents in Figure 2. They are toy examples written for this page, but the features are computed exactly as the rules define them.

keptremoved (the reasons are listed)
Figure 2. Gopher, C4 and FineWeb rules on ten toy documents. Turn rules on and off, move the word limit, and click a document to see its text and the measured features.
How to read Figure 2

With all rules on and the Gopher limit of 50 words, 3 of 10 documents survive (235 of 579 words). Each rule removes the junk it was designed for: the menu bar fails four rules, the server log fails the letter test, the hashtag spam fails the symbol rule, the boilerplate fails the duplicate-line rule. But look at two cases. The “Short but good” document is a fine piece of text that dies only because it has 33 words: rules have false positives. And the “Science article (copy)” survives every rule: rules cannot see that two documents are the same. That is the next section.

Algorithm 1: rule filtering of one document
  1. input: text \(t\), thresholds (word range, symbol ratio, letter fraction, stop words, repetition limits)
  2. \(w \leftarrow\) split \(t\) on white space; \(L \leftarrow\) non-empty lines of \(t\)
  3. if \(|w| \notin [50,\ 100\,000]\) or mean word length \(\notin [3, 10]\): return remove
  4. if (count of “#” and “…”) \(/\ |w| \gt 0.1\): return remove
  5. if fraction of words with a letter \(\lt 0.8\) or stop words \(\lt 2\): return remove
  6. if fraction of duplicate lines \(\gt 0.3\) or fraction of lines ending in punctuation \(\le 0.12\): return remove
  7. if \(t\) contains “lorem ipsum”: return remove
  8. return keep // each test is \(O(|t|)\): this runs on billions of pages

Model-based quality filtering

Rules remove obvious junk. They cannot tell a clear explanation from a dull, correct-looking page. For that, teams train a small quality classifier: a model that gives each document a score, trained on examples of text we want (positives) and random web text (negatives). In its simplest form:

\[ q(x) = \sigma\big(\mathbf{w}^\top \phi(x) + c\big), \qquad \text{keep } x \text{ if } q(x) \gt \tau \tag{34.1} \]
Read the symbols
\(\phi(x)\)
a feature vector for document \(x\): word n-gram counts (fastText) or an embedding from a small encoder
\(\mathbf{w}, c\)
weights and bias learned by logistic regression
\(\sigma\)
the sigmoid, so \(q(x)\) is between 0 and 1
\(\tau\)
the threshold; often chosen so that a fixed top fraction of documents is kept

Real examples, all [Certain]:

  • GPT-3: logistic regression; positives were WebText, Wikipedia and a web books corpus; negatives were unfiltered Common Crawl. Instead of a hard threshold, a document was kept when a random Pareto number (\(\alpha = 9\)) was larger than \(1 - q(x)\), so some low-score pages survive and the data stays diverse.
  • FineWeb-Edu: Llama-3-70B-Instruct scored 460,000 pages for educational value on a 0 to 5 scale. A linear regression on top of a small embedding model learned to copy these scores. Keeping pages with score at least 3 gave a 1.3 trillion token dataset. Classifying all of FineWeb took about 6,000 H100 GPU hours.
  • DCLM: a fastText classifier with positives from instruction data (OpenHermes 2.5) and high-scoring r/ExplainLikeImFive posts, negatives from a RefinedWeb-style web sample, about 400,000 documents in total. Keeping only the top 10% worked better than 15% or 20%. Their 7B model trained on 2.6T tokens of this data reached 64% on MMLU (5-shot). The paper concludes that model-based filtering is key.
  • Llama 3: a fastText classifier trained to recognize text that Wikipedia would reference, plus RoBERTa-based classifiers trained on Llama 2 predictions, with DistilRoberta used to score everything efficiently.

Deduplication, part 1: exact copies

The web is full of copies: the same news story on a hundred sites, the same license text in thousands of repositories, the same page crawled in many snapshots. Why remove them?

  • Memorization. Text seen thousands of times is memorized and repeated word for word.
  • Wasted compute. Ten copies of a page cost ten times the FLOPs for almost no new information.
  • A skewed mixture. Whatever is copied most gets the largest weight, without anybody choosing it.
  • Honest evaluation. Lee et al. found that 4.6% of C4’s validation examples had a near-duplicate in the training set [Certain].

Exact deduplication is easy. Normalize the text (lower case, collapse white space), compute a cryptographic hash such as SHA-256, and keep one document per hash. A hash table of billions of 32-byte keys fits on a few machines. The same idea works at other levels:

  • URL level: Llama 3 kept only the most recent version of each URL [Certain].
  • Line level: Llama 3, following ccNet, removed lines that appeared more than 6 times in each bucket of 30 million documents. This kills cookie banners and footers inside otherwise good pages [Certain].
  • Substring level: Lee et al.’s ExactSubstr uses a suffix array (a sorted list of all suffixes of the whole dataset) to find any span of at least 50 BPE tokens that occurs more than once. In C4 it flagged 7.18% of training tokens [Certain].

But exact hashing misses near-duplicates: the same article with a different date, a different footer, or a few changed words. One changed character gives a completely different hash. In Figure 2, the “Science article (copy)” differs only by one sentence at the end. We need a way to measure “almost the same” and to find such pairs without comparing every pair.

Deduplication, part 2: MinHash for near-duplicates

Documents as sets of shingles

Turn each document into a set of shingles: all runs of \(n\) consecutive words. FineWeb and Lee et al. both used 5-grams of words [Certain]. Then compare two documents by the Jaccard similarity of their sets:

\[ J(A, B) = \frac{|A \cap B|}{|A \cup B|} \tag{34.2} \]

\(J = 1\) means the same set of shingles; \(J = 0\) means no shingle in common. Example: \(A = \{a, b, c, d\}\) and \(B = \{c, d, e\}\) share 2 elements out of 5 distinct ones, so \(J = 0.4\).

Computing \(J\) for every pair is impossible. A billion documents give about \(5 \times 10^{17}\) pairs. We need a short signature per document such that similar documents have similar signatures.

The MinHash trick

Take a random hash function \(h\) that maps each shingle to a number. Think of it as a random ranking of all possible shingles. For a document, keep only the smallest hash value among its shingles: \(\min h(A)\). The key fact:

\[ \Pr\big[\min h(A) = \min h(B)\big] = J(A, B) \tag{34.3} \]

Why? Look at the union \(A \cup B\). Under a random ranking, each of its elements is equally likely to be the smallest. The two minimums are equal exactly when that smallest element lies in both sets, that is, in \(A \cap B\). The chance of that is \(|A \cap B| / |A \cup B|\). That is all.

One hash gives a yes/no answer. With \(k\) independent hash functions \(h_1, \dots, h_k\), the signature of a document is the list of its \(k\) minimums, and the fraction of positions where two signatures agree estimates \(J\):

\[ \hat{J} = \frac{1}{k} \sum_{i=1}^{k} \mathbf{1}\big[\min h_i(A) = \min h_i(B)\big], \qquad \operatorname{SE}(\hat{J}) = \sqrt{\frac{J(1 - J)}{k}} \tag{34.4} \]
Read the symbols
\(k\)
the number of hash functions, which is the signature length
\(\mathbf{1}[\cdot]\)
1 if the condition is true, 0 otherwise
\(\operatorname{SE}\)
the standard error: the typical size of the estimation error (each position is a coin flip with probability \(J\))

The signature has a fixed size, whatever the document’s length. FineWeb used \(k = 112\); at \(J = 0.75\) that gives a standard error of 0.041. Lee et al. used \(k = 9{,}000\), which gives 0.0046 [Certain for \(k\); the errors are computed with formula 34.4].

signature position where A and B agreepositions that differ
Figure 3. MinHash on two short documents. Document A is fixed (the “Science article” from Figure 2). Each small square is one of the \(k\) hash functions. The fraction of green squares is the MinHash estimate of the Jaccard similarity.
How to read Figure 3

Start with “Three small edits” and \(n = 5\): three changed words make the exact Jaccard similarity 0.689 (62 shared 5-grams of 90), and MinHash with 112 hashes estimates 0.679. Each changed word breaks up to \(n\) shingles, so larger \(n\) makes the same edit look bigger: with \(n = 3\) the similarity is 0.814, with \(n = 1\) it is 0.944. Move \(k\): the estimate jumps around the exact value, and the jumps shrink like \(1/\sqrt{k}\). Type in box B: the numbers update.

LSH banding: finding candidate pairs without comparing all pairs

Signatures are short, but comparing all pairs of signatures is still \(O(N^2)\). Locality-sensitive hashing (LSH) fixes this. Cut each signature of \(k = b \cdot r\) numbers into \(b\) bands of \(r\) rows each. Hash each band to a bucket. Two documents become a candidate pair if they land in the same bucket in at least one band, that is, if all \(r\) numbers of some band agree. Each band agrees with probability \(s^r\), where \(s = J(A, B)\), so

\[ \Pr[\text{candidate}] = 1 - \left(1 - s^{r}\right)^{b} \tag{34.5} \]

This curve is an S shape. It is close to 0 for low similarity and close to 1 for high similarity, with a steep rise in between. The rise is near the point where \(s^r = 1/b\):

\[ s^{*} \approx \left(\frac{1}{b}\right)^{1/r} \tag{34.6} \]

More rows \(r\) push the threshold up and make each band stricter; more bands \(b\) give more chances and push it down. Only documents in the same bucket are compared, so the work is close to linear in \(N\). Candidates are then grouped (FineWeb clusters them transitively and keeps one random document per cluster [Certain]; Lee et al. also check the edit similarity of each candidate pair [Certain]).

Figure 4. The LSH S-curve, formula (34.5). The dashed line is the approximate threshold (34.6). The dot shows the pair with similarity \(s\).
How to read Figure 4

FineWeb’s 14 bands of 8 rows (112 hashes) put the threshold at about 0.72; a pair with \(s = 0.75\) becomes a candidate 77% of the time, a pair with \(s = 0.5\) only 5% of the time. Lee et al.’s 450 bands of 20 rows (9,000 hashes) give a much steeper curve: 99.5% at \(s = 0.8\), 1.6% at \(s = 0.6\). The cost of a steeper curve is a longer signature. Try \(r = 1\): any single shared minimum is enough, and almost everything becomes a candidate.

Worked example: what FineWeb’s settings catch

FineWeb: \(b = 14\), \(r = 8\), target similarity 0.75 [Certain]. The threshold is \(s^* \approx (1/14)^{1/8} = 0.719\).

A pair with \(s = 0.8\): one band matches with probability \(0.8^8 = 0.168\). No band matches with probability \((1 - 0.168)^{14} = 0.077\). So the pair is caught with probability \(1 - 0.077 = 0.924\).

A pair with \(s = 0.5\): \(0.5^8 = 0.0039\), and \(1 - (1 - 0.0039)^{14} = 0.053\). Only 5.3% of such pairs are even compared.

The edited document of Figure 3 (\(s = 0.689\) with 5-grams) would be caught with probability 0.52, about one half. Three edits in an 80-word text are already close to the border of “duplicate”.

Algorithm 2: MinHash + LSH near-duplicate removal
  1. input: documents \(d_1, \dots, d_N\); shingle size \(n\); bands \(b\), rows \(r\); \(k = b r\) hash functions
  2. for each document \(d\): \(S \leftarrow\) set of word \(n\)-grams of \(d\); \(\text{sig}[d] \leftarrow (\min_{x \in S} h_1(x), \dots, \min_{x \in S} h_k(x))\) // formula (34.3)
  3. for band \(j = 1 \dots b\): put \(d\) in bucket \(\text{hash}(\text{sig}[d][\,(j-1)r : jr\,])\) of table \(j\)
  4. candidate pairs \(\leftarrow\) all pairs that share a bucket in some table // formula (34.5)
  5. optional: keep only pairs with estimated (34.4) or exact Jaccard above the target
  6. build a graph with an edge per pair; find connected components (union-find)
  7. keep one document per component; remove the rest

A surprise: more deduplication is not always better

FineWeb tried two ways to deduplicate 96 snapshots [Certain]. Global deduplication across all snapshots left 4 trillion tokens, and models trained on it were barely better than on non-deduplicated data. Deduplicating each snapshot separately left 20 trillion tokens and matched the strong RefinedWeb dataset. A likely explanation from their analysis: in old snapshots, the pages that survive global deduplication are disproportionately low quality, because good pages tend to be copied and recrawled [Likely]. The authors suggest that the gains of deduplication come mainly from removing the very large clusters of copies. Llama 3, in contrast, reports global MinHash deduplication [Certain]. The right amount of deduplication depends on the rest of the pipeline. Always measure with small training runs.

The data mixture

After cleaning, we have several pools: web, code, math, books, each language. The mixture says what fraction of training tokens comes from each pool. Two simple rules are common.

Temperature sampling

If pool \(i\) has \(n_i\) tokens, sample it with probability

\[ p_i = \frac{n_i^{\alpha}}{\sum_j n_j^{\alpha}}, \qquad 0 \le \alpha \le 1 \tag{34.7} \]
Read the symbols
\(n_i\)
size of pool \(i\) (tokens or documents)
\(\alpha\)
the sampling exponent; \(\alpha = 1\) samples in proportion to size, \(\alpha = 0\) gives every pool the same share. Some papers write \(\alpha = 1/T\) with a “temperature” \(T\).
\(p_i\)
the probability that the next training sequence comes from pool \(i\)

Small \(\alpha\) boosts small pools. mT5 used \(\alpha = 0.3\) over 101 languages, after comparing 0.2, 0.3 and 0.7, as a compromise between high- and low-resource languages [Certain]. With two languages of 1,000B and 10B tokens, \(\alpha = 1\) gives the small one 0.99% of the mix, \(\alpha = 0.7\) gives 3.8%, and \(\alpha = 0.3\) gives 20.1%.

Epochs: the price of boosting

Boosting a small pool means seeing it many times. If we train on \(D\) tokens in total, pool \(i\) is repeated

\[ e_i = \frac{p_i \, D}{n_i} \ \text{ epochs} \tag{34.8} \]

Repetition is not free, but a little is fine. Muennighoff et al. found that up to about 4 epochs of repeated data give almost the same loss as fresh data; with much more repetition, extra compute gives almost nothing [Certain]. So a rule of thumb: a small high-quality pool can be repeated a few times, not tens of times [Likely]. Figure 5 lets you see the trade-off with GPT-3’s real pool sizes.

share of available tokensshare of the training mixepochs above 4
Figure 5. A mixture over GPT-3’s five sources (sizes from the GPT-3 paper). For each source: its share of the available tokens, its share of the mix, and how many epochs it gets for a training run of \(D\) tokens, formula (34.8).
How to read Figure 5

At \(\alpha = 1\) every source gets the same number of epochs (0.60 at 300B), and Common Crawl takes 82% of the mix. At \(\alpha = 0.3\) Common Crawl falls to 39.7% and Wikipedia rises to 9.1%, but Wikipedia is then repeated 9.1 times, far beyond the 4 epochs that are almost free. GPT-3’s hand-chosen weights sit in between: more weight on the sources the team trusted, at most about 3.5 epochs. Increase \(D\) and watch the small sources run out first.

Curriculum and annealing

The mixture does not have to be constant. A curriculum changes it over training. The most common form today is annealing: at the end of pretraining, while the learning rate goes to zero, upsample the best data. Llama 3 annealed the learning rate linearly to 0 over the final 40M tokens and, in that phase, upsampled very high-quality sources [Certain]. Its annealing data used upsampled high-quality code and math and, the paper notes, no training sets of common benchmarks [Certain]. This fits well with the warmup-stable-decay schedule from session 4: the decay phase is a natural place for the best data. Choosing the mixture itself is often done with small proxy models: train many small models on different mixtures, fit how the loss changes, and pick the best mixture for the large run [Likely].

Synthetic data

Synthetic data is text written by another model. It is used when good human text is rare: step-by-step math solutions, textbook-style explanations, exercises, translations into low-resource languages, rewritten (cleaner) versions of web pages. A well-known example is phi-1: a 1.3B code model trained on about 6B tokens of filtered “textbook quality” web data plus about 1B tokens of synthetic textbooks and exercises generated with GPT-3.5. It reached 50.6% pass@1 on HumanEval [Certain].

The risks:

  • Less diversity. A generator repeats its own favorite phrasings and topics. Prompts must vary topic, level and style on purpose. [Likely]
  • Errors copied at scale. A wrong fact generated a million times is learned well. Verifiable domains (math with checked answers, code with tests) are safer.
  • Model collapse. Training generation after generation mostly on model output loses the rare “tails” of the real distribution. Mixing in real data reduces this. [Likely]
  • Hidden contamination. A generator that has seen a benchmark can reproduce its questions.

Contamination: when the test is in the training data

Contamination means that benchmark questions (and often their answers) appear in the pretraining data. The model can then answer from memory, and the score measures memory, not skill. On the web this happens easily: benchmarks are posted on GitHub, in blogs, in forum questions.

The standard check is n-gram overlap. For a test item \(x\), let \(G_n(x)\) be its set of word \(n\)-grams and \(G_n(\mathcal{D})\) the set of all \(n\)-grams in the training data. The overlap is

\[ \text{overlap}(x) = \frac{|G_n(x) \cap G_n(\mathcal{D})|}{|G_n(x)|} \tag{34.9} \]

An item is flagged as contaminated if the overlap (or the length of the longest shared span) is above a threshold. GPT-3 searched for 13-gram overlaps between benchmarks and training data, and later analysed each benchmark with \(n\) between 8 and 13 [Certain]. It also shows how hard this is in practice: a bug in the filtering left some overlaps in the data, and retraining was too expensive, so the authors measured the effect afterwards instead [Certain].

What to do about it:

  • Before training: remove training documents that overlap with the benchmarks you will report (decontamination). Keep the list of benchmarks fixed and public.
  • After training: report scores on the clean and the overlapping subsets separately. A large gap is a warning.
  • Use fresh tests: benchmarks written after the data cutoff, or held-out private sets (session 40).

The whole pipeline in code

A small, deterministic version of every step: rule filter, exact dedup with SHA-256, MinHash with \(k = 14 \times 8\) hash functions of the form \((a x + b) \bmod p\), LSH banding, a temperature mixture over GPT-3’s pool sizes, and an 8-gram contamination check. It runs in about a second on a CPU.

import hashlib, re, torch
torch.manual_seed(0)

docs = [
    "The tides are caused mainly by the pull of the Moon. The side of the Earth that faces the Moon is pulled a little more than the center, and the far side a little less.",
    "The tides are caused mainly by the pull of the Moon. The side of the Earth that faces the Moon is pulled a little more than the center, and the far side a little less. Share this page!",
    "the tides are caused mainly by the pull of the moon.  The side of the Earth that faces the Moon is pulled a little more than the center, and the far side a little less.",
    "To make tomato soup, cook two onions in olive oil, add the tomatoes and water, and let it cook slowly for thirty minutes before you blend it.",
    "Home | About | Contact | Login | Cart",
]

# 1) rule filters (a few Gopher rules, Algorithm 1)
STOP = {"the", "be", "to", "of", "and", "that", "have", "with"}
def passes_rules(t, min_words=20):
    w = t.split()
    if len(w) < min_words: return False
    if not 3 <= sum(map(len, w)) / len(w) <= 10: return False
    if sum(bool(re.search("[A-Za-z]", x)) for x in w) / len(w) < 0.8: return False
    return sum(re.sub("[^a-z]", "", x.lower()) in STOP for x in w) >= 2
docs = [d for d in docs if passes_rules(d)]
print("after rules:", len(docs))                       # 4: the menu bar is gone

# 2) exact dedup: hash of normalized text
def norm(t): return " ".join(re.sub(r"[^a-z0-9 ]", " ", t.lower()).split())
seen, exact = set(), []
for d in docs:
    h = hashlib.sha256(norm(d).encode()).hexdigest()
    if h not in seen: seen.add(h); exact.append(d)
print("after exact dedup:", len(exact))                # 3: case/space copy removed

# 3) MinHash signatures, formula (34.3), with k = b * r hash functions
def shingles(t, n=5):
    w = norm(t).split()
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
P = (1 << 31) - 1                                      # a prime; a*x stays below 2^62
b, r = 14, 8; k = b * r                                # FineWeb: 14 bands x 8 rows
g = torch.Generator().manual_seed(1)
A = torch.randint(1, P, (k, 1), generator=g); B = torch.randint(0, P, (k, 1), generator=g)
def signature(S):
    x = torch.tensor([int.from_bytes(hashlib.blake2b(s.encode(), digest_size=8).digest(), "little") % P for s in S])
    return ((A * x[None, :] + B) % P).min(dim=1).values  # (k,) min over shingles
sig = torch.stack([signature(shingles(d)) for d in exact])
for i in range(len(exact)):
    for j in range(i + 1, len(exact)):
        Si, Sj = shingles(exact[i]), shingles(exact[j])
        J = len(Si & Sj) / len(Si | Sj)
        est = (sig[i] == sig[j]).float().mean().item()     # formula (34.4)
        cand = bool((sig[i].view(b, r) == sig[j].view(b, r)).all(dim=1).any())  # LSH, (34.5)
        print(f"docs {i},{j}: Jaccard {J:.3f}  MinHash {est:.3f}  candidate {cand}")

# 4) mixture by temperature, formula (34.7), then sample sources for 10,000 sequences
sizes = torch.tensor([410., 19., 12., 55., 3.])        # GPT-3 sources, billions of tokens
alpha = 0.3
p = sizes ** alpha / (sizes ** alpha).sum()
draws = torch.multinomial(p, 10_000, replacement=True, generator=torch.Generator().manual_seed(2))
print("weights:", [round(v, 3) for v in p.tolist()])
print("sampled:", [round(v, 3) for v in (torch.bincount(draws, minlength=5) / 10_000).tolist()])
print("epochs at 300B:", [round(v, 2) for v in (p * 300 / sizes).tolist()])   # formula (34.8)

# 5) contamination check: share of a test item's 8-grams found in the training data
def ngrams(t, n=8):
    w = norm(t).split(); return {tuple(w[i:i + n]) for i in range(len(w) - n + 1)}
train_grams = set().union(*(ngrams(d) for d in exact))
test_item = "Question: the side of the Earth that faces the Moon is pulled a little more than what?"
tg = ngrams(test_item)
print(f"overlap: {len(tg & train_grams)}/{len(tg)} 8-grams")   # formula (34.9)

Output (PyTorch 2.8, CPU):

after rules: 4
after exact dedup: 3
docs 0,1: Jaccard 0.912  MinHash 0.920  candidate True
docs 0,2: Jaccard 0.000  MinHash 0.000  candidate False
docs 1,2: Jaccard 0.000  MinHash 0.000  candidate False
weights: [0.397, 0.158, 0.138, 0.217, 0.091]
sampled: [0.4, 0.157, 0.139, 0.219, 0.086]
epochs at 300B: [0.29, 2.49, 3.44, 1.18, 9.07]
overlap: 8/10 8-grams

Read it from top to bottom. The menu bar fails the rules. The lower-case copy has the same normalized text, so exact hashing removes it. The copy with “Share this page!” has a different hash but Jaccard 0.912; MinHash estimates 0.920 and LSH flags the pair. The test question shares 8 of its 10 word 8-grams with a training document: a clear case of contamination. In a real pipeline each step runs distributed over thousands of CPU cores (for example with the datatrove library used for FineWeb [Likely]), but the logic is the same.

Where this is used

SettingWhich ideasNotes
LLM pretraining datasets (FineWeb, DCLM, Llama 3)Extraction, language ID, rules, MinHash, quality classifiers, mixtures, annealingThe full pipeline of this session
Code datasetsExact and near dedup of files (forks), license filters, rules for generated or minified filesNear-duplicate files are extremely common in code [Likely]
Fine-tuning and SFT dataDedup, quality scoring with an LLM judge, decontamination against evalsSmall sets, so each bad example matters more (session 61)
Image-text data for VLMsURL and NSFW filters, image dedup with perceptual hashes or embeddings, CLIP-score filtering of captionsThe same pipeline shape with other filters (session 187)
Video fingerprinting and copy detectionPer-frame or per-shot signatures, LSH buckets, then exact verificationThe same “signature, bucket, verify” plan as Algorithm 2
Video-text training sets and video retrieval indexesDedup of re-uploaded clips; balancing categories with a temperature rule; caption quality filtersRe-uploads inflate some content (session 201)
Video-RAG and search indexesNear-duplicate removal before indexing so results are not ten copies of one clipMinHash on transcripts, embeddings for frames
Benchmark hygienen-gram overlap (34.9), clean vs contaminated subsetsNeeded for any honest model report

Common mistakes

  • Trusting the WET text. Pre-extracted text keeps menus and boilerplate. The extractor choice changes downstream scores by points, not decimals.
  • Only exact deduplication. Hashes miss near-copies with a changed date or footer. Use MinHash (or suffix arrays) as well.
  • Judging a filter by how much it removes. Removal rate says nothing about quality. Judge filters by training small models on the filtered data and evaluating them, as FineWeb and DCLM did.
  • Forgetting epochs when boosting a source. A strong upweight of a small pool can mean ten or more epochs (formula 34.8). Compute \(e_i\) for every pool before you launch.
  • Using one language-ID threshold for every language. Scores are less reliable for small languages and short pages; a fixed threshold can drop whole languages. [Likely]
  • Decontaminating after training, or against a different list of benchmarks than the one you report.
  • Shuffling badly. If documents from one source or one snapshot arrive in long runs, the model sees a drifting distribution; shuffle across sources and shards. [Likely]
  • Mixing evaluation data into the annealing set. The final high-quality phase has a strong effect, so contamination there is the most harmful. Llama 3 explicitly excluded benchmark training sets from it.

Interview questions

How would you turn Common Crawl into a pretraining dataset?basic
Extract text from the raw WARC HTML with a good extractor (trafilatura, resiliparse), not the WET files. Apply cheap filters first: URL blocklist, language ID (FineWeb: fastText, English score \(\ge 0.65\)), then rule filters (Gopher, C4 and FineWeb rules for length, symbols, stop words, repetition). Deduplicate: exact hashing at URL, document and line level, then MinHash + LSH for near-duplicates. Optionally score quality with a trained classifier (FineWeb-Edu, DCLM). Remove PII such as emails, decontaminate against the benchmarks you will report, tokenize, and build the mixture with other sources. Validate every step by training small models on the result.
Why does deduplication help?basic
Repeated text is memorized and repeated verbatim; Lee et al. showed deduplicated training makes models emit memorized text about ten times less often and reach the same or better accuracy in fewer steps. It also saves compute, stops copied content from silently taking a big share of the mixture, and removes train-test overlap (4.6% of C4 validation examples had a near-duplicate in training).
Explain MinHash. Why does the collision probability equal the Jaccard similarity?core
Represent each document as a set of shingles (word 5-grams). A hash function acts as a random ranking; the MinHash value of a set is its smallest hash. Under a random ranking each element of \(A \cup B\) is equally likely to be the overall minimum, and \(\min h(A) = \min h(B)\) exactly when that element is in \(A \cap B\). So the probability is \(|A \cap B| / |A \cup B| = J\). With \(k\) hashes, the fraction of agreeing positions estimates \(J\) with standard error \(\sqrt{J(1-J)/k}\), for example 0.041 at \(J = 0.75\), \(k = 112\).
How does LSH banding work, and how do you choose \(b\) and \(r\)?core
Split the \(k = br\) signature into \(b\) bands of \(r\) values; hash each band to a bucket; pairs sharing a bucket in any band are candidates. A pair with similarity \(s\) becomes a candidate with probability \(1 - (1 - s^r)^b\), an S-curve with threshold about \((1/b)^{1/r}\). Pick \(r\) and \(b\) so the threshold sits at your target similarity, and make \(k\) larger if you need a steeper curve (fewer false positives and false negatives). FineWeb: \(14 \times 8\), threshold \(\approx 0.72\), target 0.75. Lee et al.: \(450 \times 20\) with 9,000 hashes, threshold \(\approx 0.74\) and very steep.
Exact-substring dedup vs MinHash: what does each catch?core
MinHash works on whole documents: it finds documents whose shingle sets are mostly the same. It misses a long copied paragraph inside two otherwise different documents. ExactSubstr builds a suffix array over the corpus and finds every repeated span of at least \(L\) tokens (Lee et al.: 50 BPE tokens), so it removes repeated boilerplate and quoted passages inside unique documents. It is heavier in memory and compute. Line-level dedup (ccNet, Llama 3) is a cheap middle ground for footers and banners.
Rules or a quality classifier? What are the risks of each?core
Rules are cheap, transparent and remove obvious junk, but they have false positives (short good pages, poetry, lists) and miss fluent low-value text. Classifiers learn “looks like the positives”, which raises benchmark scores (DCLM keeps the top 10%; FineWeb-Edu keeps score \(\ge 3\)), but they inherit the bias of the positive set and can remove dialects, informal language and whole topics, and they can overfit to what the benchmarks measure. Mitigations: soft thresholds (GPT-3’s Pareto sampling), keeping some unfiltered data, auditing removals per language and domain.
FineWeb found that global deduplication over all snapshots was worse than per-snapshot deduplication. Why might that be?advanced
Global dedup removed far more (4T tokens left vs 20T) and gave little improvement over no dedup. The authors’ analysis suggests that in older snapshots, the documents that are unique across all crawls are disproportionately low quality, while good pages are recrawled and copied, so global dedup throws away many good pages and keeps the odd ones. Most of the benefit comes from removing very large duplicate clusters, which per-snapshot dedup already does. Lesson: dedup is a data-selection decision, measure it with ablation runs.
How do you choose a data mixture across languages or sources?core
Start from temperature sampling \(p_i \propto n_i^{\alpha}\): \(\alpha = 1\) is proportional, smaller \(\alpha\) boosts small pools (mT5: 0.3). Then compute epochs \(e_i = p_i D / n_i\) and cap them (a few epochs, since up to about 4 is nearly free per Muennighoff et al.). Upweight sources that help target abilities (code and math: Llama 3 used 17% code, 25% math and reasoning). Validate with small proxy runs, and consider an annealing phase that upsamples the best data at the end.
Your data is limited. Is it OK to train for several epochs?advanced
Some repetition is fine: Muennighoff et al. found that up to about 4 epochs give almost the same loss as unique data, and that the value of more repetition falls off until extra compute gives nothing. So repeat high-quality pools a few times, but beyond that prefer more (filtered) unique data, synthetic data, or a smaller model trained compute-optimally (session 35). Repetition also raises memorization risk for the repeated text.
What is test-set contamination, and how do you detect and handle it?core
Benchmark items present in pretraining data, so the model can answer from memory. Detect with n-gram overlap between each test item and the corpus (GPT-3: 13-grams, later 8 to 13 per benchmark), or longest-common-substring checks; flag items above a threshold. Handle by removing overlapping training documents before training, reporting clean vs contaminated subsets, and using benchmarks created after the data cutoff. n-gram checks miss paraphrases, so they give a lower bound on contamination.
When is synthetic pretraining data a good idea, and what can go wrong?advanced
Good when real data is scarce or noisy in a domain you care about: textbook-style explanations and exercises (phi-1: about 1B synthetic tokens plus 6B filtered web tokens reached 50.6% on HumanEval at 1.3B parameters), verified math and code, cleaner rewrites of web pages. Risks: low diversity, the generator’s errors repeated at scale, loss of distribution tails when models are trained mostly on model output (model collapse), and leakage of benchmarks the generator saw. Use diverse prompts, verification where possible, and mix with real data.

Exercises

  1. \(A = \{\text{the cat}, \text{cat sat}, \text{sat on}, \text{on the}, \text{the mat}\}\), \(B = \{\text{the cat}, \text{cat sat}, \text{sat on}, \text{on a}, \text{a mat}\}\). Compute \(J(A, B)\).
  2. With \(b = 20\) bands of \(r = 5\) rows, find the approximate threshold, and the probability that a pair with \(s = 0.5\) becomes a candidate. Check it in Figure 4.
  3. You want MinHash to estimate \(J\) near 0.5 with a standard error of at most 0.01. How many hash functions do you need?
  4. Two languages have 1,000B and 10B tokens. What are their sampling shares with \(\alpha = 0.5\)?
  5. Training on \(D = 300\)B tokens, Wikipedia (3B tokens) has 3% of the mix. How many epochs is that? What weight gives exactly 4 epochs?
  6. The test question in the code above has 10 word 8-grams, and 8 of them appear in the training data. What is its overlap (34.9)? Would a 13-gram check flag it too? When would it not?
Answerscheck
1. Shared: the cat, cat sat, sat on (3). Union: 7 distinct bigrams. \(J = 3/7 = 0.429\). 2. \(s^* \approx (1/20)^{1/5} = 0.549\). \(0.5^5 = 0.03125\), \((1 - 0.03125)^{20} = 0.530\), so the probability is 0.470. 3. \(\sqrt{0.25/k} \le 0.01\) gives \(k \ge 2{,}500\). 4. \(\sqrt{1000} = 31.62\) and \(\sqrt{10} = 3.162\), a ratio of 10 to 1: shares 90.9% and 9.1%. 5. \(0.03 \times 300 / 3 = 3.0\) epochs. Four epochs needs \(p = 4 \times 3 / 300 = 4\%\). 6. Overlap \(8/10 = 0.8\), so it is flagged. The question has 17 normalized words, and the shared run (“the side … more than”) is 15 words long. With 13-grams the question has 5 of them and 3 are shared (overlap 0.6), so it is flagged too. But if the shared run were shorter than 13 words, a 13-gram check would find nothing. Smaller \(n\) catches more contamination, with more false alarms on common phrases.

Key takeaways

  • Data decides what a model learns. Real pipelines (FineWeb: 96 snapshots to 15T tokens) are a series of filters, cheap ones first.
  • Extract from raw HTML, filter by URL and language, then apply rule filters (Gopher, C4, FineWeb); then model-based quality classifiers (34.1) such as FineWeb-Edu and DCLM.
  • Exact dedup uses hashes at URL, document, line and substring level. Near-dedup uses MinHash: \(\Pr[\min h(A) = \min h(B)] = J\) (34.3), estimated with \(k\) hashes (34.4).
  • LSH banding gives the S-curve \(1 - (1 - s^r)^b\) (34.5) with threshold \((1/b)^{1/r}\) (34.6); FineWeb used \(14 \times 8\).
  • Mixtures: temperature sampling (34.7) boosts small pools, but watch the epochs (34.8); up to about 4 epochs is nearly free. Annealing on the best data at the end helps.
  • Synthetic data fills gaps but can reduce diversity; contamination (34.9) must be checked before training and reported after.

Key formulas

\[ J = \frac{|A \cap B|}{|A \cup B|}, \quad \Pr[\min h(A) = \min h(B)] = J, \quad \operatorname{SE} = \sqrt{\frac{J(1-J)}{k}} \]
\[ \Pr[\text{cand}] = 1 - (1 - s^r)^b, \quad s^* \approx b^{-1/r}, \quad p_i = \frac{n_i^{\alpha}}{\sum_j n_j^{\alpha}}, \quad e_i = \frac{p_i D}{n_i} \]

References for this session

  1. Penedo et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
  2. Lee et al. (2021). Deduplicating Training Data Makes Language Models Better
  3. Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 7: Pre-training
  4. Brown et al. (2020). Language Models are Few-Shot Learners
  5. Llama Team, Meta (2024). The Llama 3 Herd of Models
  6. Xue et al. (2020). mT5: A massively multilingual pre-trained text-to-text transformer
  7. Rae et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher
  8. Li et al. (2024). DataComp-LM: In Search of the Next Generation of Training Sets for Language Models
  9. Muennighoff et al. (2023). Scaling Data-Constrained Language Models
  10. Gunasekar et al. (2023). Textbooks Are All You Need