Pretraining data: from web crawl to training mixture
A base model can only learn what is in its training data. This session follows text from a raw web crawl to the final training mixture: how pages become clean text, how rules and small classifiers throw away bad pages, how MinHash finds near-copies among billions of documents, and how we decide how much of each source to show the model. We end with synthetic data and with test-set contamination, the quiet bug that makes benchmark scores lie.
What you will learn
- Where pretraining text comes from (web, books, code, papers, math, many languages) and why the web dominates
- The cleaning pipeline: text extraction, URL filters, language ID, rule filters (Gopher, C4, FineWeb) and model-based quality classifiers
- Exact deduplication with hashes, and near-duplicate detection with MinHash: why the collision probability equals the Jaccard similarity
- LSH banding: how b bands of r rows give an S-shaped curve with a threshold near (1/b)^(1/r), and how FineWeb and Lee et al. chose b and r
- Data mixtures: temperature sampling, epochs per source, the GPT-3 and Llama 3 mixes, annealing on high-quality data
- Synthetic data and contamination: what each one helps or breaks, and how to check a benchmark with n-gram overlap
Why this matters
A language model is a compressed copy of its training data. The architecture decides how well it can learn; the data decides what it learns. Two models with the same size and the same compute can differ a lot only because one was trained on cleaner text.
In session 31 we fixed the task: predict the next token. In session 33 we saw that RoBERTa beat BERT partly by using more data. This session answers the question every pretraining team spends most of its time on: which tokens do we feed the model, and how many times each?
The scale is large. FineWeb, an open English web dataset, has 15 trillion tokens taken from 96 Common Crawl snapshots [Certain]. Llama 3 was pretrained on about 15 trillion tokens [Certain]. No human can read even a tiny part of this. Every decision must be made by programs: rules, hash functions and small classifiers. If they are wrong, nobody notices until the model is trained.
Bad data costs in three ways. Duplicates make the model memorize: Lee et al. found a single 61-word sentence repeated more than 60,000 times in C4, and showed that deduplicated training makes models emit memorized text about ten times less often [Certain]. Junk (menus, logs, spam) wastes compute on tokens that teach nothing. Contamination (benchmark questions inside the training data) makes evaluation scores look better than the model really is. The next session, session 35, tells you how many tokens you need; this one tells you how to get good ones.
What goes in: the sources
Pretraining data is a mix of sources. Each one teaches something different:
| Source | What it teaches | Problems |
|---|---|---|
| Web pages (Common Crawl) | Almost every topic and style; the largest source by far | Mostly boilerplate, spam and copies; needs heavy filtering |
| Books | Long, coherent text; long-range structure | Copyright and licensing; small compared to the web |
| Code (public repositories) | Exact syntax, logic, step-by-step structure | Licenses, generated files, many forks of the same project |
| Scientific papers (arXiv and similar) | Technical language, formulas in LaTeX | PDF extraction errors; tables and math break easily |
| Math (web pages and problem sets) | Multi-step reasoning, notation | Rare on the web; easy to lose in HTML extraction |
| Many languages | Use in other languages; transfer between languages | Very uneven sizes: English is much larger than most languages |
| Encyclopedias, forums, Q&A | Facts; question-and-answer patterns | Small; forums contain toxic text |
Two real recipes show how different the shares can be. GPT-3 (2020) used five sources [Certain]:
| GPT-3 source | Tokens | Weight in the mix | Epochs in 300B tokens |
|---|---|---|---|
| Common Crawl (filtered) | 410B | 60% | 0.44 |
| WebText2 | 19B | 22% | 2.9 |
| Books1 | 12B | 8% | 1.9 |
| Books2 | 55B | 8% | 0.43 |
| Wikipedia | 3B | 3% | 3.4 |
Look at the weights. Common Crawl is 82% of all available tokens (410 of 499 billion), but only 60% of the training mix. Wikipedia is 0.6% of the tokens but 3% of the mix, so it is seen more than three times. The weights are a choice, not a fact about the data. (The table is copied from the paper; its rounded numbers do not reproduce every epoch exactly, for example \(0.22 \times 300 / 19 = 3.5\) for WebText2, not 2.9.)
Llama 3 (2024) describes its final mix by content instead of by source: about 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual tokens [Certain]. Compared with GPT-3, much more code and math. Teams found that code and math help reasoning in general, not only on code and math tasks [Likely].
From crawl to clean text
Common Crawl is a non-profit that crawls the web and publishes each crawl as a snapshot, a dump of billions of pages [Likely]. Each snapshot comes in two main formats: WARC files (the raw HTTP responses, with full HTML) and WET files (text that Common Crawl already extracted). FineWeb extracted text from WARC files with the trafilatura library, because the WET text “retained too much boilerplate and menu text” [Certain]. DCLM did the same with another extractor (resiliparse) and found it better than WET by at least 2.5 points on its benchmark average [Certain]. The extractor is the first filter, and one of the most important.
A typical pipeline, here in the order FineWeb used, then the extra steps many teams add. Click a stage in Figure 1.
How to read Figure 1
Click any box or use Next. The readout says what the stage does, why, and (where the paper gives it) how much data is left. Notice how much is removed: after the cheap first filters FineWeb had about 36 trillion tokens, after deduplication about 20 trillion, and the final dataset is 15 trillion. The cheap stages run first, because they shrink the data before the expensive ones (MinHash, classifiers) run.
URL filtering and language identification
Two cheap filters run early. A URL blocklist removes known adult and malware domains (FineWeb used the UT1 list) [Certain]. Language identification (language ID) is a small text classifier that predicts the language of a page and a confidence score. FineWeb used a fastText classifier and kept pages with an English score of at least 0.65 [Certain]. For multilingual datasets the same classifier sorts pages into one bucket per language; mC4, used to train mT5, covers 101 languages [Certain].
Rule filters: cheap tests for junk
Most web pages that survive extraction are still not good text: navigation bars, product lists, server logs, hashtag spam, templates. Simple rules catch most of them. A rule filter computes a few numbers per document and removes it if one number is out of range. The best-known set comes from Gopher’s MassiveWeb [Certain]:
- between 50 and 100,000 words;
- mean word length between 3 and 10 characters;
- at most 0.1 “#” symbols or ellipses per word;
- at most 90% of lines starting with a bullet, at most 30% ending with an ellipsis;
- at least 80% of words contain a letter;
- at least two of the stop words the, be, to, of, and, that, have, with (a cheap “is this real English prose?” test);
- repetition limits, for example at most 30% duplicate lines.
C4 added its own rules (remove pages containing “lorem ipsum”, curly brackets or “javascript”, keep only lines ending in punctuation). FineWeb kept the C4 rules except the punctuation rule, which alone removed about 30% of tokens, and added three new ones [Certain]: remove a document if at most 12% of its lines end with punctuation, if at least 10% of its characters are in duplicated lines, or if at least 67% of its lines are shorter than 30 characters. Together these three removed about 22% of tokens [Certain].
Try the rules on ten small documents in Figure 2. They are toy examples written for this page, but the features are computed exactly as the rules define them.
How to read Figure 2
With all rules on and the Gopher limit of 50 words, 3 of 10 documents survive (235 of 579 words). Each rule removes the junk it was designed for: the menu bar fails four rules, the server log fails the letter test, the hashtag spam fails the symbol rule, the boilerplate fails the duplicate-line rule. But look at two cases. The “Short but good” document is a fine piece of text that dies only because it has 33 words: rules have false positives. And the “Science article (copy)” survives every rule: rules cannot see that two documents are the same. That is the next section.
- input: text \(t\), thresholds (word range, symbol ratio, letter fraction, stop words, repetition limits)
- \(w \leftarrow\) split \(t\) on white space; \(L \leftarrow\) non-empty lines of \(t\)
- if \(|w| \notin [50,\ 100\,000]\) or mean word length \(\notin [3, 10]\): return remove
- if (count of “#” and “…”) \(/\ |w| \gt 0.1\): return remove
- if fraction of words with a letter \(\lt 0.8\) or stop words \(\lt 2\): return remove
- if fraction of duplicate lines \(\gt 0.3\) or fraction of lines ending in punctuation \(\le 0.12\): return remove
- if \(t\) contains “lorem ipsum”: return remove
- return keep // each test is \(O(|t|)\): this runs on billions of pages
Model-based quality filtering
Rules remove obvious junk. They cannot tell a clear explanation from a dull, correct-looking page. For that, teams train a small quality classifier: a model that gives each document a score, trained on examples of text we want (positives) and random web text (negatives). In its simplest form:
- Read the symbols
- \(\phi(x)\)
- a feature vector for document \(x\): word n-gram counts (fastText) or an embedding from a small encoder
- \(\mathbf{w}, c\)
- weights and bias learned by logistic regression
- \(\sigma\)
- the sigmoid, so \(q(x)\) is between 0 and 1
- \(\tau\)
- the threshold; often chosen so that a fixed top fraction of documents is kept
Real examples, all [Certain]:
- GPT-3: logistic regression; positives were WebText, Wikipedia and a web books corpus; negatives were unfiltered Common Crawl. Instead of a hard threshold, a document was kept when a random Pareto number (\(\alpha = 9\)) was larger than \(1 - q(x)\), so some low-score pages survive and the data stays diverse.
- FineWeb-Edu: Llama-3-70B-Instruct scored 460,000 pages for educational value on a 0 to 5 scale. A linear regression on top of a small embedding model learned to copy these scores. Keeping pages with score at least 3 gave a 1.3 trillion token dataset. Classifying all of FineWeb took about 6,000 H100 GPU hours.
- DCLM: a fastText classifier with positives from instruction data (OpenHermes 2.5) and high-scoring r/ExplainLikeImFive posts, negatives from a RefinedWeb-style web sample, about 400,000 documents in total. Keeping only the top 10% worked better than 15% or 20%. Their 7B model trained on 2.6T tokens of this data reached 64% on MMLU (5-shot). The paper concludes that model-based filtering is key.
- Llama 3: a fastText classifier trained to recognize text that Wikipedia would reference, plus RoBERTa-based classifiers trained on Llama 2 predictions, with DistilRoberta used to score everything efficiently.
Deduplication, part 1: exact copies
The web is full of copies: the same news story on a hundred sites, the same license text in thousands of repositories, the same page crawled in many snapshots. Why remove them?
- Memorization. Text seen thousands of times is memorized and repeated word for word.
- Wasted compute. Ten copies of a page cost ten times the FLOPs for almost no new information.
- A skewed mixture. Whatever is copied most gets the largest weight, without anybody choosing it.
- Honest evaluation. Lee et al. found that 4.6% of C4’s validation examples had a near-duplicate in the training set [Certain].
Exact deduplication is easy. Normalize the text (lower case, collapse white space), compute a cryptographic hash such as SHA-256, and keep one document per hash. A hash table of billions of 32-byte keys fits on a few machines. The same idea works at other levels:
- URL level: Llama 3 kept only the most recent version of each URL [Certain].
- Line level: Llama 3, following ccNet, removed lines that appeared more than 6 times in each bucket of 30 million documents. This kills cookie banners and footers inside otherwise good pages [Certain].
- Substring level: Lee et al.’s ExactSubstr uses a suffix array (a sorted list of all suffixes of the whole dataset) to find any span of at least 50 BPE tokens that occurs more than once. In C4 it flagged 7.18% of training tokens [Certain].
But exact hashing misses near-duplicates: the same article with a different date, a different footer, or a few changed words. One changed character gives a completely different hash. In Figure 2, the “Science article (copy)” differs only by one sentence at the end. We need a way to measure “almost the same” and to find such pairs without comparing every pair.
Deduplication, part 2: MinHash for near-duplicates
Documents as sets of shingles
Turn each document into a set of shingles: all runs of \(n\) consecutive words. FineWeb and Lee et al. both used 5-grams of words [Certain]. Then compare two documents by the Jaccard similarity of their sets:
\(J = 1\) means the same set of shingles; \(J = 0\) means no shingle in common. Example: \(A = \{a, b, c, d\}\) and \(B = \{c, d, e\}\) share 2 elements out of 5 distinct ones, so \(J = 0.4\).
Computing \(J\) for every pair is impossible. A billion documents give about \(5 \times 10^{17}\) pairs. We need a short signature per document such that similar documents have similar signatures.
The MinHash trick
Take a random hash function \(h\) that maps each shingle to a number. Think of it as a random ranking of all possible shingles. For a document, keep only the smallest hash value among its shingles: \(\min h(A)\). The key fact:
Why? Look at the union \(A \cup B\). Under a random ranking, each of its elements is equally likely to be the smallest. The two minimums are equal exactly when that smallest element lies in both sets, that is, in \(A \cap B\). The chance of that is \(|A \cap B| / |A \cup B|\). That is all.
One hash gives a yes/no answer. With \(k\) independent hash functions \(h_1, \dots, h_k\), the signature of a document is the list of its \(k\) minimums, and the fraction of positions where two signatures agree estimates \(J\):
- Read the symbols
- \(k\)
- the number of hash functions, which is the signature length
- \(\mathbf{1}[\cdot]\)
- 1 if the condition is true, 0 otherwise
- \(\operatorname{SE}\)
- the standard error: the typical size of the estimation error (each position is a coin flip with probability \(J\))
The signature has a fixed size, whatever the document’s length. FineWeb used \(k = 112\); at \(J = 0.75\) that gives a standard error of 0.041. Lee et al. used \(k = 9{,}000\), which gives 0.0046 [Certain for \(k\); the errors are computed with formula 34.4].
How to read Figure 3
Start with “Three small edits” and \(n = 5\): three changed words make the exact Jaccard similarity 0.689 (62 shared 5-grams of 90), and MinHash with 112 hashes estimates 0.679. Each changed word breaks up to \(n\) shingles, so larger \(n\) makes the same edit look bigger: with \(n = 3\) the similarity is 0.814, with \(n = 1\) it is 0.944. Move \(k\): the estimate jumps around the exact value, and the jumps shrink like \(1/\sqrt{k}\). Type in box B: the numbers update.
LSH banding: finding candidate pairs without comparing all pairs
Signatures are short, but comparing all pairs of signatures is still \(O(N^2)\). Locality-sensitive hashing (LSH) fixes this. Cut each signature of \(k = b \cdot r\) numbers into \(b\) bands of \(r\) rows each. Hash each band to a bucket. Two documents become a candidate pair if they land in the same bucket in at least one band, that is, if all \(r\) numbers of some band agree. Each band agrees with probability \(s^r\), where \(s = J(A, B)\), so
This curve is an S shape. It is close to 0 for low similarity and close to 1 for high similarity, with a steep rise in between. The rise is near the point where \(s^r = 1/b\):
More rows \(r\) push the threshold up and make each band stricter; more bands \(b\) give more chances and push it down. Only documents in the same bucket are compared, so the work is close to linear in \(N\). Candidates are then grouped (FineWeb clusters them transitively and keeps one random document per cluster [Certain]; Lee et al. also check the edit similarity of each candidate pair [Certain]).
How to read Figure 4
FineWeb’s 14 bands of 8 rows (112 hashes) put the threshold at about 0.72; a pair with \(s = 0.75\) becomes a candidate 77% of the time, a pair with \(s = 0.5\) only 5% of the time. Lee et al.’s 450 bands of 20 rows (9,000 hashes) give a much steeper curve: 99.5% at \(s = 0.8\), 1.6% at \(s = 0.6\). The cost of a steeper curve is a longer signature. Try \(r = 1\): any single shared minimum is enough, and almost everything becomes a candidate.
FineWeb: \(b = 14\), \(r = 8\), target similarity 0.75 [Certain]. The threshold is \(s^* \approx (1/14)^{1/8} = 0.719\).
A pair with \(s = 0.8\): one band matches with probability \(0.8^8 = 0.168\). No band matches with probability \((1 - 0.168)^{14} = 0.077\). So the pair is caught with probability \(1 - 0.077 = 0.924\).
A pair with \(s = 0.5\): \(0.5^8 = 0.0039\), and \(1 - (1 - 0.0039)^{14} = 0.053\). Only 5.3% of such pairs are even compared.
The edited document of Figure 3 (\(s = 0.689\) with 5-grams) would be caught with probability 0.52, about one half. Three edits in an 80-word text are already close to the border of “duplicate”.
- input: documents \(d_1, \dots, d_N\); shingle size \(n\); bands \(b\), rows \(r\); \(k = b r\) hash functions
- for each document \(d\): \(S \leftarrow\) set of word \(n\)-grams of \(d\); \(\text{sig}[d] \leftarrow (\min_{x \in S} h_1(x), \dots, \min_{x \in S} h_k(x))\) // formula (34.3)
- for band \(j = 1 \dots b\): put \(d\) in bucket \(\text{hash}(\text{sig}[d][\,(j-1)r : jr\,])\) of table \(j\)
- candidate pairs \(\leftarrow\) all pairs that share a bucket in some table // formula (34.5)
- optional: keep only pairs with estimated (34.4) or exact Jaccard above the target
- build a graph with an edge per pair; find connected components (union-find)
- keep one document per component; remove the rest
A surprise: more deduplication is not always better
FineWeb tried two ways to deduplicate 96 snapshots [Certain]. Global deduplication across all snapshots left 4 trillion tokens, and models trained on it were barely better than on non-deduplicated data. Deduplicating each snapshot separately left 20 trillion tokens and matched the strong RefinedWeb dataset. A likely explanation from their analysis: in old snapshots, the pages that survive global deduplication are disproportionately low quality, because good pages tend to be copied and recrawled [Likely]. The authors suggest that the gains of deduplication come mainly from removing the very large clusters of copies. Llama 3, in contrast, reports global MinHash deduplication [Certain]. The right amount of deduplication depends on the rest of the pipeline. Always measure with small training runs.
The data mixture
After cleaning, we have several pools: web, code, math, books, each language. The mixture says what fraction of training tokens comes from each pool. Two simple rules are common.
Temperature sampling
If pool \(i\) has \(n_i\) tokens, sample it with probability
- Read the symbols
- \(n_i\)
- size of pool \(i\) (tokens or documents)
- \(\alpha\)
- the sampling exponent; \(\alpha = 1\) samples in proportion to size, \(\alpha = 0\) gives every pool the same share. Some papers write \(\alpha = 1/T\) with a “temperature” \(T\).
- \(p_i\)
- the probability that the next training sequence comes from pool \(i\)
Small \(\alpha\) boosts small pools. mT5 used \(\alpha = 0.3\) over 101 languages, after comparing 0.2, 0.3 and 0.7, as a compromise between high- and low-resource languages [Certain]. With two languages of 1,000B and 10B tokens, \(\alpha = 1\) gives the small one 0.99% of the mix, \(\alpha = 0.7\) gives 3.8%, and \(\alpha = 0.3\) gives 20.1%.
Epochs: the price of boosting
Boosting a small pool means seeing it many times. If we train on \(D\) tokens in total, pool \(i\) is repeated
Repetition is not free, but a little is fine. Muennighoff et al. found that up to about 4 epochs of repeated data give almost the same loss as fresh data; with much more repetition, extra compute gives almost nothing [Certain]. So a rule of thumb: a small high-quality pool can be repeated a few times, not tens of times [Likely]. Figure 5 lets you see the trade-off with GPT-3’s real pool sizes.
How to read Figure 5
At \(\alpha = 1\) every source gets the same number of epochs (0.60 at 300B), and Common Crawl takes 82% of the mix. At \(\alpha = 0.3\) Common Crawl falls to 39.7% and Wikipedia rises to 9.1%, but Wikipedia is then repeated 9.1 times, far beyond the 4 epochs that are almost free. GPT-3’s hand-chosen weights sit in between: more weight on the sources the team trusted, at most about 3.5 epochs. Increase \(D\) and watch the small sources run out first.
Curriculum and annealing
The mixture does not have to be constant. A curriculum changes it over training. The most common form today is annealing: at the end of pretraining, while the learning rate goes to zero, upsample the best data. Llama 3 annealed the learning rate linearly to 0 over the final 40M tokens and, in that phase, upsampled very high-quality sources [Certain]. Its annealing data used upsampled high-quality code and math and, the paper notes, no training sets of common benchmarks [Certain]. This fits well with the warmup-stable-decay schedule from session 4: the decay phase is a natural place for the best data. Choosing the mixture itself is often done with small proxy models: train many small models on different mixtures, fit how the loss changes, and pick the best mixture for the large run [Likely].
Synthetic data
Synthetic data is text written by another model. It is used when good human text is rare: step-by-step math solutions, textbook-style explanations, exercises, translations into low-resource languages, rewritten (cleaner) versions of web pages. A well-known example is phi-1: a 1.3B code model trained on about 6B tokens of filtered “textbook quality” web data plus about 1B tokens of synthetic textbooks and exercises generated with GPT-3.5. It reached 50.6% pass@1 on HumanEval [Certain].
The risks:
- Less diversity. A generator repeats its own favorite phrasings and topics. Prompts must vary topic, level and style on purpose. [Likely]
- Errors copied at scale. A wrong fact generated a million times is learned well. Verifiable domains (math with checked answers, code with tests) are safer.
- Model collapse. Training generation after generation mostly on model output loses the rare “tails” of the real distribution. Mixing in real data reduces this. [Likely]
- Hidden contamination. A generator that has seen a benchmark can reproduce its questions.
Contamination: when the test is in the training data
Contamination means that benchmark questions (and often their answers) appear in the pretraining data. The model can then answer from memory, and the score measures memory, not skill. On the web this happens easily: benchmarks are posted on GitHub, in blogs, in forum questions.
The standard check is n-gram overlap. For a test item \(x\), let \(G_n(x)\) be its set of word \(n\)-grams and \(G_n(\mathcal{D})\) the set of all \(n\)-grams in the training data. The overlap is
An item is flagged as contaminated if the overlap (or the length of the longest shared span) is above a threshold. GPT-3 searched for 13-gram overlaps between benchmarks and training data, and later analysed each benchmark with \(n\) between 8 and 13 [Certain]. It also shows how hard this is in practice: a bug in the filtering left some overlaps in the data, and retraining was too expensive, so the authors measured the effect afterwards instead [Certain].
What to do about it:
- Before training: remove training documents that overlap with the benchmarks you will report (decontamination). Keep the list of benchmarks fixed and public.
- After training: report scores on the clean and the overlapping subsets separately. A large gap is a warning.
- Use fresh tests: benchmarks written after the data cutoff, or held-out private sets (session 40).
The whole pipeline in code
A small, deterministic version of every step: rule filter, exact dedup with SHA-256, MinHash with \(k = 14 \times 8\) hash functions of the form \((a x + b) \bmod p\), LSH banding, a temperature mixture over GPT-3’s pool sizes, and an 8-gram contamination check. It runs in about a second on a CPU.
import hashlib, re, torch torch.manual_seed(0) docs = [ "The tides are caused mainly by the pull of the Moon. The side of the Earth that faces the Moon is pulled a little more than the center, and the far side a little less.", "The tides are caused mainly by the pull of the Moon. The side of the Earth that faces the Moon is pulled a little more than the center, and the far side a little less. Share this page!", "the tides are caused mainly by the pull of the moon. The side of the Earth that faces the Moon is pulled a little more than the center, and the far side a little less.", "To make tomato soup, cook two onions in olive oil, add the tomatoes and water, and let it cook slowly for thirty minutes before you blend it.", "Home | About | Contact | Login | Cart", ] # 1) rule filters (a few Gopher rules, Algorithm 1) STOP = {"the", "be", "to", "of", "and", "that", "have", "with"} def passes_rules(t, min_words=20): w = t.split() if len(w) < min_words: return False if not 3 <= sum(map(len, w)) / len(w) <= 10: return False if sum(bool(re.search("[A-Za-z]", x)) for x in w) / len(w) < 0.8: return False return sum(re.sub("[^a-z]", "", x.lower()) in STOP for x in w) >= 2 docs = [d for d in docs if passes_rules(d)] print("after rules:", len(docs)) # 4: the menu bar is gone # 2) exact dedup: hash of normalized text def norm(t): return " ".join(re.sub(r"[^a-z0-9 ]", " ", t.lower()).split()) seen, exact = set(), [] for d in docs: h = hashlib.sha256(norm(d).encode()).hexdigest() if h not in seen: seen.add(h); exact.append(d) print("after exact dedup:", len(exact)) # 3: case/space copy removed # 3) MinHash signatures, formula (34.3), with k = b * r hash functions def shingles(t, n=5): w = norm(t).split() return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)} P = (1 << 31) - 1 # a prime; a*x stays below 2^62 b, r = 14, 8; k = b * r # FineWeb: 14 bands x 8 rows g = torch.Generator().manual_seed(1) A = torch.randint(1, P, (k, 1), generator=g); B = torch.randint(0, P, (k, 1), generator=g) def signature(S): x = torch.tensor([int.from_bytes(hashlib.blake2b(s.encode(), digest_size=8).digest(), "little") % P for s in S]) return ((A * x[None, :] + B) % P).min(dim=1).values # (k,) min over shingles sig = torch.stack([signature(shingles(d)) for d in exact]) for i in range(len(exact)): for j in range(i + 1, len(exact)): Si, Sj = shingles(exact[i]), shingles(exact[j]) J = len(Si & Sj) / len(Si | Sj) est = (sig[i] == sig[j]).float().mean().item() # formula (34.4) cand = bool((sig[i].view(b, r) == sig[j].view(b, r)).all(dim=1).any()) # LSH, (34.5) print(f"docs {i},{j}: Jaccard {J:.3f} MinHash {est:.3f} candidate {cand}") # 4) mixture by temperature, formula (34.7), then sample sources for 10,000 sequences sizes = torch.tensor([410., 19., 12., 55., 3.]) # GPT-3 sources, billions of tokens alpha = 0.3 p = sizes ** alpha / (sizes ** alpha).sum() draws = torch.multinomial(p, 10_000, replacement=True, generator=torch.Generator().manual_seed(2)) print("weights:", [round(v, 3) for v in p.tolist()]) print("sampled:", [round(v, 3) for v in (torch.bincount(draws, minlength=5) / 10_000).tolist()]) print("epochs at 300B:", [round(v, 2) for v in (p * 300 / sizes).tolist()]) # formula (34.8) # 5) contamination check: share of a test item's 8-grams found in the training data def ngrams(t, n=8): w = norm(t).split(); return {tuple(w[i:i + n]) for i in range(len(w) - n + 1)} train_grams = set().union(*(ngrams(d) for d in exact)) test_item = "Question: the side of the Earth that faces the Moon is pulled a little more than what?" tg = ngrams(test_item) print(f"overlap: {len(tg & train_grams)}/{len(tg)} 8-grams") # formula (34.9)
Output (PyTorch 2.8, CPU):
after rules: 4 after exact dedup: 3 docs 0,1: Jaccard 0.912 MinHash 0.920 candidate True docs 0,2: Jaccard 0.000 MinHash 0.000 candidate False docs 1,2: Jaccard 0.000 MinHash 0.000 candidate False weights: [0.397, 0.158, 0.138, 0.217, 0.091] sampled: [0.4, 0.157, 0.139, 0.219, 0.086] epochs at 300B: [0.29, 2.49, 3.44, 1.18, 9.07] overlap: 8/10 8-grams
Read it from top to bottom. The menu bar fails the rules. The lower-case copy has the same normalized text, so exact hashing removes it. The copy with “Share this page!” has a different hash but Jaccard 0.912; MinHash estimates 0.920 and LSH flags the pair. The test question shares 8 of its 10 word 8-grams with a training document: a clear case of contamination. In a real pipeline each step runs distributed over thousands of CPU cores (for example with the datatrove library used for FineWeb [Likely]), but the logic is the same.
Where this is used
| Setting | Which ideas | Notes |
|---|---|---|
| LLM pretraining datasets (FineWeb, DCLM, Llama 3) | Extraction, language ID, rules, MinHash, quality classifiers, mixtures, annealing | The full pipeline of this session |
| Code datasets | Exact and near dedup of files (forks), license filters, rules for generated or minified files | Near-duplicate files are extremely common in code [Likely] |
| Fine-tuning and SFT data | Dedup, quality scoring with an LLM judge, decontamination against evals | Small sets, so each bad example matters more (session 61) |
| Image-text data for VLMs | URL and NSFW filters, image dedup with perceptual hashes or embeddings, CLIP-score filtering of captions | The same pipeline shape with other filters (session 187) |
| Video fingerprinting and copy detection | Per-frame or per-shot signatures, LSH buckets, then exact verification | The same “signature, bucket, verify” plan as Algorithm 2 |
| Video-text training sets and video retrieval indexes | Dedup of re-uploaded clips; balancing categories with a temperature rule; caption quality filters | Re-uploads inflate some content (session 201) |
| Video-RAG and search indexes | Near-duplicate removal before indexing so results are not ten copies of one clip | MinHash on transcripts, embeddings for frames |
| Benchmark hygiene | n-gram overlap (34.9), clean vs contaminated subsets | Needed for any honest model report |
Common mistakes
- Trusting the WET text. Pre-extracted text keeps menus and boilerplate. The extractor choice changes downstream scores by points, not decimals.
- Only exact deduplication. Hashes miss near-copies with a changed date or footer. Use MinHash (or suffix arrays) as well.
- Judging a filter by how much it removes. Removal rate says nothing about quality. Judge filters by training small models on the filtered data and evaluating them, as FineWeb and DCLM did.
- Forgetting epochs when boosting a source. A strong upweight of a small pool can mean ten or more epochs (formula 34.8). Compute \(e_i\) for every pool before you launch.
- Using one language-ID threshold for every language. Scores are less reliable for small languages and short pages; a fixed threshold can drop whole languages. [Likely]
- Decontaminating after training, or against a different list of benchmarks than the one you report.
- Shuffling badly. If documents from one source or one snapshot arrive in long runs, the model sees a drifting distribution; shuffle across sources and shards. [Likely]
- Mixing evaluation data into the annealing set. The final high-quality phase has a strong effect, so contamination there is the most harmful. Llama 3 explicitly excluded benchmark training sets from it.
Interview questions
How would you turn Common Crawl into a pretraining dataset?basic
Why does deduplication help?basic
Explain MinHash. Why does the collision probability equal the Jaccard similarity?core
How does LSH banding work, and how do you choose \(b\) and \(r\)?core
Exact-substring dedup vs MinHash: what does each catch?core
Rules or a quality classifier? What are the risks of each?core
FineWeb found that global deduplication over all snapshots was worse than per-snapshot deduplication. Why might that be?advanced
How do you choose a data mixture across languages or sources?core
Your data is limited. Is it OK to train for several epochs?advanced
What is test-set contamination, and how do you detect and handle it?core
When is synthetic pretraining data a good idea, and what can go wrong?advanced
Exercises
- \(A = \{\text{the cat}, \text{cat sat}, \text{sat on}, \text{on the}, \text{the mat}\}\), \(B = \{\text{the cat}, \text{cat sat}, \text{sat on}, \text{on a}, \text{a mat}\}\). Compute \(J(A, B)\).
- With \(b = 20\) bands of \(r = 5\) rows, find the approximate threshold, and the probability that a pair with \(s = 0.5\) becomes a candidate. Check it in Figure 4.
- You want MinHash to estimate \(J\) near 0.5 with a standard error of at most 0.01. How many hash functions do you need?
- Two languages have 1,000B and 10B tokens. What are their sampling shares with \(\alpha = 0.5\)?
- Training on \(D = 300\)B tokens, Wikipedia (3B tokens) has 3% of the mix. How many epochs is that? What weight gives exactly 4 epochs?
- The test question in the code above has 10 word 8-grams, and 8 of them appear in the training data. What is its overlap (34.9)? Would a 13-gram check flag it too? When would it not?
Answerscheck
Key takeaways
- Data decides what a model learns. Real pipelines (FineWeb: 96 snapshots to 15T tokens) are a series of filters, cheap ones first.
- Extract from raw HTML, filter by URL and language, then apply rule filters (Gopher, C4, FineWeb); then model-based quality classifiers (34.1) such as FineWeb-Edu and DCLM.
- Exact dedup uses hashes at URL, document, line and substring level. Near-dedup uses MinHash: \(\Pr[\min h(A) = \min h(B)] = J\) (34.3), estimated with \(k\) hashes (34.4).
- LSH banding gives the S-curve \(1 - (1 - s^r)^b\) (34.5) with threshold \((1/b)^{1/r}\) (34.6); FineWeb used \(14 \times 8\).
- Mixtures: temperature sampling (34.7) boosts small pools, but watch the epochs (34.8); up to about 4 epochs is nearly free. Annealing on the best data at the end helps.
- Synthetic data fills gaps but can reduce diversity; contamination (34.9) must be checked before training and reported after.
Key formulas
References for this session
- Penedo et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Lee et al. (2021). Deduplicating Training Data Makes Language Models Better
- Xiao & Zhu, NLP: Neural Networks and Large Language Models, chapter 7: Pre-training
- Brown et al. (2020). Language Models are Few-Shot Learners
- Llama Team, Meta (2024). The Llama 3 Herd of Models
- Xue et al. (2020). mT5: A massively multilingual pre-trained text-to-text transformer
- Rae et al. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- Li et al. (2024). DataComp-LM: In Search of the Next Generation of Training Sets for Language Models
- Muennighoff et al. (2023). Scaling Data-Constrained Language Models
- Gunasekar et al. (2023). Textbooks Are All You Need