LLM & VLM courseBlog
187Track 7: Vision and VLMs · session 25 of 38

Multimodal data at scale

A vision-language model learns what it sees in its data, and with the model and compute fixed, data choices alone can move results by several points. This session follows multimodal data from the raw web crawl to the training batch: image–text pairs (LAION, DataComp), interleaved web documents (OBELICS), and video–text data; filtering with CLIP score, deduplication and safety filters; synthetic dense captions; and the data mixture, a hidden hyperparameter that you tune with careful ablations.

Time: 3 to 4 hours (reading 2 hours, exercises 1 to 2 hours)Level: IntermediateBefore this: Sessions on pretraining data (34), CLIP (166) and VLM training stages (186); the anatomy of a VLM (177) helps

What you will learn

  • The four kinds of multimodal data (image–text pairs, interleaved documents, video–text, synthetic captions) and what each one teaches, with the real sizes of LAION-5B, DataComp, OBELICS, MINT-1T, HowTo100M, InternVid and Panda-70M
  • How a dataset is built as a funnel from Common Crawl, with the real counts of the LAION-5B, DataComp and OBELICS pipelines
  • CLIP-score filtering by threshold or quantile, why the top 30% works best at fixed compute in DataComp, and its failure modes such as text spotting
  • Deduplication at three levels (exact hash, perceptual hash, semantic embeddings) and how SemDeDup uses k-means to cut the cost by five orders of magnitude
  • Safety filtering (NSFW, toxic text, illegal content, faces, opt-outs, eval decontamination) and why the base rate decides what a classifier really does
  • Re-captioning with dense synthetic captions (ShareGPT4V, RECAP), its measured gains, and its risks: hallucination and loss of world knowledge
  • The data mixture as a hidden hyperparameter: sampling weights, temperature sampling, epochs per source, MM1's 45/45/10 mix, and how to design ablations that you can trust

Why this matters

A vision-language model learns what it sees in its data. The architecture of most open VLMs is now quite similar: a vision encoder, a connector, and an LLM (session 177). What separates a good model from a weak one is, more and more, the data: where it comes from, what is thrown away, how captions are written, and how the sources are mixed.

Here is the clearest proof. In the DataComp benchmark, every team trains the same CLIP model with the same number of steps; only the training set changes. At the largest scale, simple filtering raised ImageNet zero-shot accuracy from 72.3% (no filtering of the 12.8B-pair pool) to 79.2% (the filtered DataComp-1B subset, 1.4B pairs). That is 6.9 points from choosing data, with zero change to the model or the compute. [Certain] The same filtered set beat a CLIP ViT-L/14 trained on LAION-2B with the same compute by 6.1 points, and beat OpenAI's own ViT-L/14 by 3.7 points. [Certain]

The last session, session 186, showed when each kind of data is used: caption pairs to align the connector, large mixed data for multimodal pretraining, instructions for fine-tuning. This session is about what goes into those stages and how it is made at the scale of billions of examples. The next session, session 188, shows how these examples are packed into training batches. The text-only side of the story (crawling, MinHash, quality classifiers) is in session 34; here we focus on what is new when images and video join.

Four kinds of multimodal data

Almost all VLM training data falls into four groups. Each group teaches the model something different.

  1. Image–text pairs. One image and one short text. On the web, the text is usually the alt-text: an HTML attribute that describes an image for screen readers and search engines. It is free and it exists for billions of images. It is also noisy: “IMG_2041.jpg”, “click here”, a product code, or a description of something else on the page.
  2. Interleaved documents. A whole web page kept in its original order: paragraph, image, paragraph, image. The text is long and only loosely tied to each image, but it teaches the model to read several images in context, the way a person reads an article.
  3. Video–text data. A clip and some text: the speech in the video turned into text by speech recognition (ASR), a title, a stock-footage description, or a caption written by a model.
  4. Synthetic and curated data. Captions or question–answer pairs written by a strong model or by people: dense captions, OCR text, charts, instructions. Small, expensive, and very effective.
Image–text pair<image> “Golden retriever puppy playing in snow”
Interleaved document“How to repot an orchid.” <image> “First, remove the old bark…” <image> “Cut the dead roots…”
Video–text<frames 0–8 s> ASR: “so now we're going to add the flour slowly”
Dense synthetic caption<image> “A golden retriever puppy, about three months old, jumps through fresh snow in a backyard. A red ball lies on the left…”

The table lists the public datasets you will hear about most. The numbers come from each dataset's paper.

DatasetKindSizeWhere the text comes from
LAION-5B (2022)pairs5.85B pairs, 2.32B in Englishalt-text, kept if CLIP agrees with it
DataComp CommonPool / DataComp-1B (2023)pairs12.8B candidate pool; 1.4B filteredalt-text; filtering is the research question
mmc4 (2023)interleaved103M documents, 585M imagesC4 web text with images placed by CLIP matching
OBELICS (2023)interleaved141M documents, 353M images, 115B text tokensthe web page itself, in order
MINT-1T (2024)interleaved1 trillion text tokens, 3.4B imagesHTML, PDFs and arXiv papers
HowTo100M (2019)video136M clips from 1.22M narrated videosASR of the narration
WebVid-2M (2021)videoover 2M videosstock-footage descriptions
InternVid (2023)videoover 7M videos, 234M clips, 4.1B wordscaptions generated with models, at several time scales
Panda-70M (2024)video70M captioned clips from 3.8M videosseveral captioning “teacher” models; a retrieval model picks the best caption
ShareGPT4V (2023)dense captions100K GPT-4V captions + 1.2M from a trained captionera strong VLM describes the image in detail

From crawl to dataset: the funnel

Every large dataset is made by the same basic recipe: take a huge raw pool from Common Crawl (a free monthly snapshot of billions of web pages), then remove most of it in stages. Cheap tests run first on everything; expensive tests (running a neural network on each image) run later on what is left. Figure 1 shows three real funnels.

Figure 1. Three real data funnels, with the counts reported in the LAION-5B, DataComp and OBELICS papers. Bar width is on a log scale: equal changes in width mean equal factors in count.
How to read Figure 1

Click a bar to see its stage, the share kept from the stage before, and the share kept from the start. Notice three things. First, downloading fails a lot: DataComp tried about 40B URLs and got 16.8B images (42%), because web links die. Second, the biggest cut is a quality step: LAION's CLIP filter removed about 90% of 50B candidates, and OBELICS's English and quality filters kept 2.67% of 41.2B documents. Third, the final DataComp-1B is only 1.59% of the 88B raw pairs.

Worked example: the DataComp funnel in numbers

DataComp starts from about 88B URL–text pairs. It attempts about 40B downloads and gets 16.8B images. Unsafe text and NSFW images are about 19% of those: \(16.8 \to 13.6\)B, and \(1 - 13.6/16.8 = 0.19\). Near-duplicates of the evaluation images are about 3% of the 16.8B (\(\approx 0.5\)B), leaving 13.1B. A random 12.8B is the CommonPool. The best filter keeps 1.4B: \(1.4 / 12.8 = 10.9\%\) of the pool, and \(1.4 / 88 = 1.59\%\) of the raw pairs. [Certain]

A funnel must also be stored well. Datasets like LAION and DataComp are released as lists of URLs plus metadata, not as images, for legal and size reasons. Each user downloads the images again (with the open tool img2dataset), and gets fewer of them every year as links break. DataComp also publishes a SHA256 hash of every image, so that a downloader can detect when the content behind a URL has changed (for example, when someone buys an expired domain to inject bad images). [Certain]

Algorithm 1: building an image–text pair dataset from a web crawl
  1. input: Common Crawl snapshots; a CLIP model; safety classifiers; the evaluation sets you will report on
  2. parse every HTML <img> tag; keep (URL, alt-text) pairs with non-empty alt-text; remove exact duplicate (URL, text) pairs
  3. download images in parallel; drop failures, tiny files (LAION: under 5 KB) and short texts (LAION: under 5 characters)
  4. detect the language of the text (LAION used CLD3, DataComp used fastText)
  5. compute CLIP image and text embeddings once; store them, because every later filter reuses them
  6. safety: drop or tag NSFW images (classifier on CLIP embeddings), toxic text, known illegal content (hash lists); blur faces if required
  7. remove near-duplicates of the evaluation images // or your benchmark numbers mean nothing
  8. quality: keep pairs with high CLIP score (formula 187.2), and optionally other filters (image clusters, text rules)
  9. deduplicate the result (exact hashes, then semantic dedup, Algorithm 2)
  10. output: URL, text, embeddings, scores and tags for every kept pair

Filtering with CLIP score

How do you decide, for 10 billion pairs, whether the text describes the image? You cannot ask people. The trick used by LAION and many others is to ask a model that was already trained on image–text pairs: CLIP (session 166). CLIP maps an image \(x\) and a text \(y\) to unit vectors in the same space. If the text describes the image, the two vectors point in a similar direction. The CLIP score is their cosine similarity:

\[ s(x, y) = \cos\big(f(x), g(y)\big) = \frac{f(x)^\top g(y)}{\lVert f(x) \rVert\, \lVert g(y) \rVert} \tag{187.1} \]
Read the symbols
\(x, y\)
one image and its text (alt-text or caption)
\(f, g\)
the image encoder and text encoder of a trained CLIP model
\(s\)
the CLIP score, between \(-1\) and \(1\); for CLIP ViT-B/32, matching web pairs usually score around 0.3, unrelated ones much lower

There are two ways to turn the score into a decision:

\[ \text{threshold: keep } i \text{ if } s_i \ge \tau, \qquad \text{quantile: keep the } \lceil qN \rceil \text{ pairs with the highest } s_i \tag{187.2} \]
Read the symbols
\(\tau\)
a fixed threshold. LAION-5B used CLIP ViT-B/32 with \(\tau = 0.28\) for English and \(\tau = 0.26\) for other languages [Certain]
\(q\)
the fraction to keep. DataComp's strongest CLIP-score baseline keeps the top 30% by CLIP ViT-L/14 score [Certain]
\(N\)
the number of pairs in the pool

A quantile is easier to compare across scoring models, because different CLIP models give scores on different scales. A threshold of 0.28 for ViT-B/32 means nothing for ViT-L/14.

Why a mismatched pair hurts

CLIP-style models train with a contrastive loss (session 166). For a batch of \(B\) pairs, the image-to-text part is

\[ \mathcal{L}_{i \to t} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp\big(s(x_i, y_i)/T\big)}{\sum_{j=1}^{B} \exp\big(s(x_i, y_j)/T\big)} \tag{187.3} \]

with a temperature \(T\). The loss pulls each image toward its own text and pushes it away from the others. If \(y_i\) is “IMG_2041.jpg”, the model is pushed to make that image similar to a meaningless string, and the gradient spent on that pair teaches nothing useful. A VLM trained with next-token loss on such a pair learns to write file names. Filtering raises the share of pairs whose gradient points in a useful direction.

caption describes the imagevague but relatedunrelated textcaption repeats text written in the image
Figure 2. CLIP-score filtering on a toy pool of 4,000 pairs (seeded random scores, not real data). Each group has its own score distribution. Bars to the right of the threshold are kept (full color); bars to the left are dropped (faded).
How to read Figure 2

Move \(\tau\). A low threshold keeps almost everything, including the gray unrelated pairs. A high threshold keeps only the best green pairs but throws away most of the useful blue ones, so the dataset becomes small and less diverse. Now look at the orange group: images that contain written text (a poster, a slide, a meme) whose caption just repeats those words. CLIP is very good at reading text, so these pairs get the highest scores, and no threshold can remove them. The T-MARS paper reports that nearly 40% of LAION images contain text that overlaps a lot with the caption. [Certain] Tick the box: if the text in the image is masked before scoring, these pairs drop to the score of unrelated pairs and the threshold now removes them.

What can go wrong with CLIP-score filtering

  • The filter copies the filter model's blind spots. Whatever CLIP does not understand (rare languages, medical images, handwriting, non-Western objects) gets low scores and is removed. The new model then cannot learn it either.
  • Text spotting. As in Figure 2, pairs where the caption repeats the visible text score high but teach reading, not seeing.
  • Too strict means too small. In DataComp, a random subset is always worse when it is smaller, but for CLIP-score filtering the best subset has an intermediate size: about the top 30% of the pool. [Certain] Below that, the set loses diversity faster than it gains quality.
  • A better CLIP is not a better filter. The Data Filtering Networks paper found that a model's quality as a filter is not the same as its accuracy on benchmarks: a model with high ImageNet accuracy can produce a worse training set than a weaker model trained on a small, high-quality dataset. Their filtered DFN-5B set trained a ViT-H to 84.4% ImageNet zero-shot. [Certain]

DataComp: an experiment about data, not models

Before DataComp, papers compared datasets trained with different models, different numbers of steps and different evaluations, so nobody could say which dataset was better. DataComp fixes everything except the data. It gives one common pool, four scales, and a fixed training recipe for each scale:

ScaleModelPool size = samples seenTraining compute (MACs)
smallViT-B/3212.8M\(9.5 \times 10^{16}\)
mediumViT-B/32128M\(9.5 \times 10^{17}\)
largeViT-B/161.28B\(2.6 \times 10^{19}\)
xlargeViT-L/1412.8B\(1.1 \times 10^{21}\)

The key rule: the number of samples seen is fixed. If your filter keeps 30% of a 128M pool, you still train on 128M samples, so you see each kept sample about 3.3 times. A smaller set is not cheaper; it is repeated more. This makes the comparison fair: every dataset gets the same compute. Every model is then tested on the same 38 tasks (ImageNet, distribution shifts, the VTAB suite, retrieval, and more). Figure 3 shows the paper's main results.

Figure 3. DataComp filtering baselines (Table 3 of the paper). Each bar is one CLIP model trained on the filtered subset with the fixed compute of that scale; the label after the value is the subset size. The best strategy at the chosen scale is orange.
How to read Figure 3

At every scale, all filters beat no filtering. Switch between scales and watch the ranking. At large and xlarge, the intersection of image-based filtering (keep images close to ImageNet clusters) and CLIP score is best. At the small scale it is worse than CLIP score alone, because a 1.4M subset repeated about 9 times is too little diversity for 12.8M samples seen. This is the most important lesson for ablations: a ranking measured at a small scale may not hold at a large scale.

Deduplication

The web repeats itself. The same product photo appears on a thousand shops; memes are re-uploaded with a new crop; icons and ads appear on every page. Repeats waste compute, push the model to memorize, and, if they include test images, make benchmark scores lie. There are three levels of “the same”:

  1. Exact duplicates: the same bytes. A cryptographic hash (SHA256, MD5) of the file finds them in one pass.
  2. Near-duplicates: the same picture after resizing, re-compression or a small crop. A perceptual hash (pHash, dHash) turns the image into, say, 64 bits that change little under such edits. Two images are near-duplicates when the Hamming distance (the number of different bits) is small:
\[ d_H(\mathbf{h}_a, \mathbf{h}_b) = \sum_{j=1}^{64} \big[\, h_{a,j} \ne h_{b,j} \,\big] \le k \tag{187.4} \]
  1. Semantic duplicates: different photos of the same content, for example 20 shots of the same white sneaker on a white background. Only an embedding model can see this.

Semantic dedup (SemDeDup)

SemDeDup embeds every image with CLIP and calls two items duplicates when their cosine similarity is at least \(1 - \varepsilon\). For unit vectors, cosine and distance are tied together:

\[ \lVert \mathbf{a} - \mathbf{b} \rVert^2 = 2\,(1 - \cos(\mathbf{a}, \mathbf{b})), \qquad \text{so} \qquad \cos \ge 1 - \varepsilon \iff \lVert \mathbf{a} - \mathbf{b} \rVert \le \sqrt{2\varepsilon} \tag{187.5} \]

Comparing all pairs is impossible at scale: for \(n\) items there are about \(n^2/2\) pairs. So SemDeDup first clusters the embeddings with k-means and compares only inside each cluster. With \(k\) clusters of sizes \(n_1, \dots, n_k\):

\[ \text{checks} = \sum_{c=1}^{k} \frac{n_c (n_c - 1)}{2} \;\ge\; \frac{n}{2}\left(\frac{n}{k} - 1\right) \approx \frac{n^2}{2k} \tag{187.6} \]
Read the symbols
\(\varepsilon\)
how different two items may be and still count as duplicates; larger \(\varepsilon\) removes more
\(k\)
the number of k-means clusters; more clusters means fewer checks, but duplicates that land in two different clusters are never compared
\(n_c\)
the number of items in cluster \(c\); the bound is reached when all clusters have the same size \(n/k\)
Worked example: the cost of dedup on LAION-440M

With \(n = 440\)M, all pairs need about \(n^2 = 1.9 \times 10^{17}\) similarity computations (the paper's count). With \(k = 50{,}000\) clusters of equal size, \(n^2/k = 3.9 \times 10^{12}\) (counting ordered pairs, as the paper does). The paper reports about \(4.6 \times 10^{12}\): a bit more, because real clusters have unequal sizes and formula (187.6) is smallest for equal sizes. That is still five orders of magnitude fewer. [Certain] The result: SemDeDup removed 50% of LAION-440M with a performance drop under 0.5%, and up to 37% with no drop at all, halving training time. [Certain] The paper also found that 30% of LAION-440M images have a semantic duplicate even at the very strict \(\varepsilon = 0.00095\). [Certain]

Algorithm 2: SemDeDup (following the paper)
  1. input: unit embeddings \(\mathbf{e}_1, \dots, \mathbf{e}_n\) (e.g. CLIP image embeddings), clusters \(k\), threshold \(\varepsilon\)
  2. run k-means with \(k\) clusters; get centroids \(\mathbf{c}_1, \dots, \mathbf{c}_k\)
  3. for each cluster \(c\):
  4.   sort its items by similarity to \(\mathbf{c}_c\), lowest first // the item farthest from the centroid is kept
  5.   compute the pairwise cosine matrix \(S\) of the cluster; keep only the upper triangle
  6.   remove item \(j\) if \(\max_{i \lt j} S_{ij} \ge 1 - \varepsilon\) // it is a near-copy of an earlier item
  7. output: the kept items

Why keep the item farthest from the centroid? Items near the centroid are the most typical, so they carry the least new information; the paper keeps the least typical one in each group. Figure 4 runs the algorithm on a small 2-D picture of an embedding space.

Figure 4. SemDeDup on a seeded toy pool: six topics, each with 30 spread-out items and four “bursts” of near-copies (re-uploads of one photo). Colors show k-means clusters, black plus signs are centroids, faded dark crosses are removed items. Two points are duplicates when their distance is at most \(\sqrt{2\varepsilon}\) (formula 187.5); the circle at the bottom right shows that radius.
How to read Figure 4

With small \(\varepsilon\), only the tight bursts are removed: each burst shrinks to one point. Raise \(\varepsilon\): soon it also removes spread-out items that are merely similar. That is no longer “dedup” but pruning, and it can remove real diversity. Now change \(k\). With \(k = 1\) every pair is checked. With more clusters the checks drop sharply (with \(k = 12\) to about 10% of all pairs). The price: at larger \(\varepsilon\), some duplicate pairs fall on a cluster border and are never compared (the last number in the readout). Real systems use tens of thousands of clusters and accept a few misses.

For interleaved documents, dedup has more levels. OBELICS removed images that appear more than 10 times in the whole dataset (mostly icons and ads), but on purpose kept images repeated only a few times, because their surrounding text was different. It then removed documents with the same URL or the same set of images, and exact-duplicate paragraphs within one domain, which removed about 15% of the text (mostly repeated boilerplate). [Certain] Text-level near-dedup with MinHash works as in session 34. For video, dedup uses fingerprints of frames or clips (session 209).

Safety filtering

A web crawl contains everything the web contains. Before training, large datasets apply several safety steps:

StepHow it is doneReal example
NSFW imagesa small classifier on CLIP embeddingsLAION tags NSFW (about 3% of images) instead of deleting; DataComp removes it [Certain]
Toxic texta text classifierDataComp uses Detoxify; with NSFW images, about 19% of downloads removed [Certain]
Illegal contentmatching against hash lists from child-safety organizationsLAION-5B was taken down in December 2023 after a Stanford Internet Observatory report found links to child sexual abuse material; Re-LAION-5B (August 2024, about 5.5B pairs) removed 2,236 matching links [Certain]
Privacyface detection and blurring; PII removal from textDataComp blurs faces and finds it has little effect on accuracy [Certain]
Consentremove content whose owners opted out of AI trainingOBELICS checks every image with the Spawning API [Certain]
Evaluation leakagenear-duplicate search against all test setsDataComp flags about 3% of its 16.8B images [Certain]

Safety classifiers make mistakes in both directions, and at web scale the base rate (how rare the bad class is) matters as much as the classifier's accuracy.

Worked example: what a 96% classifier really does

LAION tested one of its NSFW image classifiers on a hand-checked test set: it caught 96.45% of NSFW images and wrongly flagged 7.96% of safe images. [Certain] Suppose NSFW images are 3% of the pool, as LAION's tags suggest. Per million images: \(30{,}000 \times 0.9645 = 28{,}935\) NSFW images are caught, but \(970{,}000 \times 0.0796 = 77{,}212\) safe images are flagged too. So only \(28{,}935 / 106{,}147 = 27\%\) of flagged images are really NSFW, and about \(1{,}065\) NSFW images per million still pass. Removing everything flagged costs 10.6% of the data. This is why pipelines use several filters, look at samples by hand, and choose thresholds by the cost of each kind of error.

Interleaved web documents

Image–text pairs teach the model to name what it sees. But a chat with a VLM looks more like a document: several images, long text, references back to earlier images. Flamingo was trained on a private dataset of interleaved web pages (M3W), and OBELICS was built as an open version of the same idea. [Certain]

Building it is harder than building pairs. You must keep the page's order, remove menus, ads and footers, and decide which images belong to the article. OBELICS simplifies the HTML tree with hand-made rules (the files become more than 10 times smaller), extracts text and images in reading order, and then filters at three levels: single images (too small, odd shapes, URLs containing “logo”), paragraphs (too short, repetitive, too many special characters, high perplexity under a Wikipedia n-gram model), and whole documents (no images, or too many). [Certain] The result has on average \(353/141 \approx 2.5\) images and \(115\text{B}/141\text{M} \approx 816\) text tokens per document.

How is an interleaved document trained? The images are turned into visual tokens and placed where they appeared in the page. The model predicts only the text tokens; image tokens are inputs, never targets:

\[ \mathcal{L} = -\frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \log p_\theta\big(x_t \mid x_{\lt t}\big), \qquad \mathcal{T} = \{\, t : x_t \text{ is a text token} \,\} \tag{187.7} \]

Here \(x_{\lt t}\) includes every earlier text token and every earlier image. So text after an image can use it, and text that refers back to the first image can still see it. The masks that make this work are the topic of session 188.

What each kind of data teaches

Two ablations answer this clearly.

  • OBELICS (9B models): training on interleaved documents reached the same average performance as training on image–text pairs with about 10 times fewer images, and was better on visual question answering. Pairs kept a small advantage on captioning, classification and OCR, because a caption is close to a label. Mixing both was best for a given compute. [Certain]
  • MM1 (Apple): as the share of caption data grows, zero-shot performance rises from 25.8% to 39.3%. But few-shot performance needs at least 50% interleaved data: with it, 8-shot stays above 61%; without it, 8-shot falls to 45% and 4-shot to 43.7%. Text-only data protects the LLM's language skills. The final MM1 mix is 45% interleaved, 45% caption pairs, 10% text-only. [Certain]

Video–text data

Video adds time, and time makes text alignment much harder. The main sources of text, and their problems:

  • Speech (ASR). HowTo100M uses the narration of 1.22M instructional videos, cut into 136M clips. [Certain] It is free and huge, but people often talk about what they will do or did, not what is on screen now, and many words (“hi everyone, please subscribe”) describe nothing visual. [Likely]
  • Alt-text of stock footage. WebVid has over 2M videos with short descriptions scraped from a stock site. [Certain] They match the clip well but are short and cover one scene.
  • Model-written captions. InternVid describes clips at several time scales with captioning models and an LLM, giving 234M clips and 4.1B words. [Certain] Panda-70M splits 3.8M long videos into clips that each show one consistent scene, captions each clip with several “teacher” models, and trains a retrieval model (on a small human-labeled set) to pick the best caption. [Certain]

Video pipelines add their own steps before any of this: cut long videos at shot boundaries, drop static or near-black clips, deduplicate re-uploads with fingerprints, and decide how many frames to sample, because every frame costs visual tokens (session 183). Training details for video models are in session 212.

Synthetic and re-captioned data

Alt-text describes an image in about 50 characters, if it describes it at all. A strong VLM can write a 900-character description that names objects, colors, positions, text and style. Re-captioning means replacing (or adding to) the web text with such model-written captions.

Figure 5. Average caption length in characters (Table 1 of the ShareGPT4V paper). Human COCO captions and BLIP model captions are about 50 characters; GPT-4V's detailed captions average 942, and the trained Share-Captioner's 826.

The ShareGPT4V recipe shows the standard way to make dense captions at scale without paying a strong API for every image:

Algorithm 3: re-captioning at scale (the ShareGPT4V recipe)
  1. input: a diverse image pool; a strong but expensive captioner (GPT-4V); a trainable open VLM
  2. pick a diverse seed set (100K images from many sources); write data-specific prompts asking for objects, attributes, spatial relations, world knowledge and aesthetics
  3. caption the seed set with the strong model // average 942 characters
  4. fine-tune the open VLM on these 100K captions // this becomes the "Share-Captioner"
  5. check quality with people: in 100 side-by-side tests, 38.2% preferred GPT-4V, 35.3% preferred Share-Captioner, 26.5% found them equal
  6. caption the large pool with the cheap captioner // 1.2M images in about 44 A100 GPU-days
  7. output: dense captions for pretraining (ShareGPT4V-PT) and the best 100K for fine-tuning

44 A100 GPU-days for 1.246M images is about \(44 \times 86{,}400 / 1.246\text{M} \approx 3\) seconds of one A100 per image. The best ablation in the paper keeps the images fixed and only changes the captions, which is exactly the controlled test you want:

Pretraining data (same 558K images)MME perceptionMMBenchSEED (image)
Baseline (LLaVA-1.5-7B recipe)1516.965.366.8
+ the images with BLIP captions1521.6 (+4.7)66.2 (+0.9)66.9 (+0.1)
+ the images with ShareGPT4V-PT dense captions1539.8 (+22.9)68.3 (+3.0)68.9 (+2.1)

Same images, same compute, only better text: about five times the MME gain of short captions. [Certain] Re-captioning also helps image generation: the RECAP paper re-captioned the training set of a text-to-image model and improved FID from 17.87 to 14.84. [Certain] At the largest scale, Recap-DataComp-1B re-captioned 1.3B DataComp images with a LLaVA model built on Llama-3-8B. [Certain]

MM1 confirms the mixing view: synthetic VeCap captions were only 7% of its caption data but gave 2.4 and 4 points of few-shot gain. [Certain]

The data mixture: a hidden hyperparameter

A pretraining run draws each training sequence from one of several sources. The mixture weights decide how often. They are as important as the learning rate, but they are rarely tuned with the same care, and many papers report them in one line. The VLM book calls the mixture a hidden hyperparameter. With weights \(w_i\) for sources \(i = 1, \dots, m\):

\[ p_i = \frac{w_i}{\sum_{j=1}^{m} w_j}, \qquad \text{epochs}_i = \frac{p_i \cdot S \cdot u_i}{N_i} \tag{187.8} \]
Read the symbols
\(p_i\)
the probability that the next training sequence comes from source \(i\)
\(S\)
the total number of training sequences (\(\text{steps} \times \text{batch size}\))
\(u_i\)
how many items of source \(i\) fit in one sequence (for example 16 packed caption pairs, 1 document, or 4,096 text tokens)
\(N_i\)
the size of source \(i\), in the same items
\(\text{epochs}_i\)
how many times the run sees source \(i\) on average; above 1 means repetition

A common way to set the weights from the sizes is temperature sampling:

\[ p_i = \frac{N_i^{\alpha}}{\sum_{j} N_j^{\alpha}}, \qquad 0 \le \alpha \le 1 \tag{187.9} \]

With \(\alpha = 1\), sources are sampled in proportion to their size (big sources dominate). With \(\alpha = 0\), all sources get equal shares (small sources repeat many times). Values in between, such as \(\alpha = 0.5\), boost small high-quality sources without letting them repeat too much. For three sources of 2B, 600M and 300M items: \(\alpha = 1\) gives 0.69 / 0.21 / 0.10, \(\alpha = 0.5\) gives 0.52 / 0.28 / 0.20, and \(\alpha = 0\) gives 1/3 each.

Worked example: MM1's mixture and its budget

MM1 found a caption : interleaved : text ratio of 5 : 5 : 1 to balance image and text skills: \(5/11 = 0.455\) and \(1/11 = 0.091\), which it rounds to 45% / 45% / 10%. [Certain] It trains for 200k steps with 512 sequences of 4,096 tokens: \(S = 102.4\)M sequences, \(419\)B token slots (the paper says “approx. 400B tokens”). [Certain]

Assume (our assumption) 16 packed caption pairs per sequence (MM1's maximum of 16 images per sequence) and one interleaved document per sequence. Then caption pairs: \(0.45 \times 102.4\text{M} \times 16 = 737\)M pairs, or 0.37 epochs of the 2B pairs. Interleaved: \(46.1\)M documents, 0.077 epochs of 600M. Text: \(10.24\text{M} \times 4096 = 41.9\)B tokens, 0.021 epochs of 2T. Nothing repeats. At this scale, the weights decide the balance of skills, not repetition. In smaller open recipes, a small high-quality source with a big weight is repeated many times, and that is when the weights start to cause overfitting.

Figure 6. A mixture calculator using formulas (187.8) and (187.9) with MM1's source sizes (2B caption pairs, 600M interleaved documents, 2T text tokens). Top bar: the share of sequences from each source. Bottom bars: epochs per source; the red dashed line is one epoch. Packing assumptions: 16 pairs, 1 document or 4,096 text tokens per sequence. Temperature sampling uses the source sizes counted in sequences.
How to read Figure 6

Start at MM1's 45 / 45 / 10 and 200k steps: no source reaches one epoch. Push the steps to 1M: caption pairs pass 1.8 epochs first, because 16 pairs fit in one sequence. Switch to temperature sampling: \(\alpha\) near 1 lets the huge text source take most of the batch, and the model would hardly see images; \(\alpha = 0\) gives each source one third. Note what this figure cannot tell you: which mixture gives the best model. Only experiments can, which is the next section.

Designing data ablations

A data ablation is a controlled experiment: change one thing about the data, keep everything else fixed, and measure the effect. It sounds simple; it is easy to do badly. DataComp, MM1 and ShareGPT4V's caption test give the rules.

Algorithm 4: a data ablation you can trust
  1. input: a baseline data recipe; one change to test (a filter, a source, a weight, a caption type)
  2. fix the model, the optimizer, the steps and the samples seen // a smaller dataset is repeated, not trained for less time
  3. change only the data variable; record the size of each resulting set
  4. train at a small proxy scale with \(r\) seeds each (formula 187.10)
  5. evaluate on a broad, decontaminated suite: zero-shot and few-shot, captioning, VQA, OCR, retrieval, and text-only tasks
  6. re-run the top 2–3 options at one larger scale to check that the ranking holds // Figure 3: it may not
  7. output: the chosen recipe, with the effect size and its uncertainty

How many seeds? Two runs that differ only in the random seed still give different scores. If the seed-to-seed standard deviation of a benchmark is \(\sigma\), the difference between two recipes averaged over \(r\) seeds each has standard error \(\sigma\sqrt{2/r}\). To detect a true difference \(\delta\) at about 95% confidence you need

\[ \mathrm{SE}(\Delta) = \sigma \sqrt{\frac{2}{r}}, \qquad r \;\ge\; 2 \left( \frac{1.96\, \sigma}{\delta} \right)^{2} \tag{187.10} \]

Example (illustrative numbers): with \(\sigma = 0.3\) points and a hoped-for gain of \(\delta = 0.5\) points, \(r \ge 2 \times (1.96 \times 0.3 / 0.5)^2 = 2.77\), so 3 seeds per recipe. One seed each gives a standard error of 0.42 points, almost as large as the effect. Measure \(\sigma\) on your own setup first; it differs a lot between benchmarks.

Code: a small curation pipeline in PyTorch

The code below builds a toy pool of 1,000 image–text pairs as CLIP-like unit vectors, filters it by CLIP score (formulas 187.1 and 187.2), runs SemDeDup (Algorithm 2) after adding 200 near-copies, and samples a mixture (formula 187.8). It is deterministic; the printed numbers are the ones we got.

import torch, torch.nn.functional as F
torch.manual_seed(0)

# ---- 1) a toy pool: 1,000 image-text pairs as CLIP-like unit embeddings (d = 64)
N, d = 1000, 64
concept = F.normalize(torch.randn(N, d), dim=-1)            # what each image shows
img = F.normalize(concept + 1.2 * F.normalize(torch.randn(N, d), dim=-1), dim=-1)
rel = torch.rand(N)                                          # how much each alt-text is about its image
good = rel > 0.5                                             # call it a "good" caption if rel > 0.5
txt = F.normalize(rel[:, None] * concept + 1.2 * F.normalize(torch.randn(N, d), dim=-1), dim=-1)

# ---- 2) CLIP-score filter, formulas (187.1) and (187.2)
score = (img * txt).sum(-1)                                  # cosine, because both are unit vectors
def keep_top(score, q):                                      # DataComp style: keep the top q fraction
    k = int(round(q * len(score)))
    return torch.topk(score, k).indices
def keep_threshold(score, tau):                              # LAION style: fixed threshold
    return torch.nonzero(score >= tau).squeeze(1)
idx = keep_top(score, 0.3)
print("tau for top 30%:", round(score[idx].min().item(), 3),
      "| precision:", round(good[idx].float().mean().item(), 3),
      "| base rate:", round(good.float().mean().item(), 3))
# tau for top 30%: 0.33 | precision: 0.883 | base rate: 0.495

# ---- 3) SemDeDup: k-means, then remove near-duplicates inside each cluster (187.5, 187.6)
def kmeans(x, k, iters=20, seed=0):
    g = torch.Generator().manual_seed(seed)
    c = x[torch.randperm(len(x), generator=g)[:k]].clone()
    for _ in range(iters):
        a = (x @ c.T).argmax(1)                              # nearest centroid by cosine
        for j in range(k):
            m = a == j
            if m.any(): c[j] = F.normalize(x[m].mean(0), dim=0)
    return a, c

def semdedup(x, k, eps):
    a, c = kmeans(x, k)
    keep = torch.ones(len(x), dtype=torch.bool); pairs = 0
    for j in range(k):
        ids = torch.nonzero(a == j).squeeze(1)
        if len(ids) < 2: continue
        sim_c = x[ids] @ c[j]
        ids = ids[torch.argsort(sim_c)]                      # farthest from the centroid first
        S = x[ids] @ x[ids].T
        S = torch.triu(S, diagonal=1)                        # only compare with earlier items
        pairs += len(ids) * (len(ids) - 1) // 2
        dup = S.max(0).values >= 1 - eps                     # a near-copy of an earlier item?
        keep[ids[dup]] = False
    return keep, pairs

# add 200 near-copies of random images (re-uploads, resizes, crops)
src = torch.randint(0, N, (200,))
x = torch.cat([img, F.normalize(img[src] + 0.02 * torch.randn(200, d) / d**0.5, dim=-1)])
keep, pairs = semdedup(x, k=10, eps=0.01)
print("kept", keep.sum().item(), "of", len(x),
      "| pair checks", pairs, "vs all pairs", len(x) * (len(x) - 1) // 2)
# kept 1000 of 1200 | pair checks 72107 vs all pairs 719400

# ---- 4) a mixture sampler, formula (187.8)
w = torch.tensor([0.45, 0.45, 0.10])                         # caption, interleaved, text-only
g = torch.Generator().manual_seed(0)
draws = torch.multinomial(w, 100_000, replacement=True, generator=g)
print("sampled shares:", (torch.bincount(draws, minlength=3) / 100_000).tolist())
# sampled shares: about [0.450, 0.449, 0.101]

Three things to notice. The top-30% filter raises the share of good captions from 49.5% to 88.3%, but it also drops many good pairs (only 30% of the pool is kept). The dedup removes exactly the 200 added copies while checking only 10% of all pairs. And a sampler with fixed seeds makes a mixture reproducible, which matters when you compare ablations.

Where this is used

WhereWhat data work is doneWhy
CLIP and SigLIP-style encodersbillions of filtered pairs (LAION, DataComp, DFN), dedup, eval decontaminationcontrastive training needs scale, and noise directly becomes wrong positives (187.3)
VLM pretraining (MM1, Idefics2, LLaVA-OneVision)pairs + interleaved + text-only mixture, re-captioned subsetszero-shot from pairs, few-shot and multi-image skills from documents, language kept by text
Text-to-image generationre-captioning with dense captions (RECAP, Recap-DataComp-1B), aesthetic and watermark filtersthe model follows the prompt only as well as captions describe the images
Video retrieval and Video-RAGclip segmentation, ASR alignment, model-written captions (Panda-70M, InternVid)search quality depends on how well each clip's text matches what is on screen
Action recognition and key-moment detectiontemporal captions at several time scales, removal of static clipsthe model must link words to when things happen, not only what
Video fingerprinting and near-duplicate detectionperceptual and embedding hashes of frames, as in dedup (187.4–187.6)the same tools that clean datasets also find re-uploads and copies in production
Domain VLMs (documents, medical, retail)small curated or synthetic sets mixed with general data (temperature sampling, 187.9)a small high-value source needs a high weight, but not so high that it repeats until overfit

Common mistakes

  • Comparing datasets at different compute. A filtered set trained for fewer steps looks worse only because it trained less. Fix the samples seen, as DataComp does.
  • Trusting small-scale rankings. In DataComp, the best filter at the small scale is not the best at large scales. Confirm the winners one scale up.
  • Filtering too hard. The top 5% by CLIP score is clean but small and narrow. Diversity loss shows up later as poor performance on rare concepts and new domains.
  • Forgetting to decontaminate. If test images (or near-copies) are in the training set, benchmark gains are memorization. Search every training pool against every eval set.
  • Using only one score. CLIP score misses text-spotting pairs and copies CLIP's blind spots. Combine it with other signals: image clusters, text rules, masked-text scores, a learned filter.
  • Replacing all web text with synthetic captions. You gain detail and lose names, facts and diversity, and you import the captioner's hallucinations. Mix both kinds of text.
  • Ignoring text-only data. Without it, the LLM inside the VLM forgets language skills during multimodal training. MM1 keeps 10% text-only.
  • Not storing scores and embeddings. Every new filter idea then needs another pass of a large model over billions of images. Compute once, keep everything.

Interview questions

Why not train on all web image–alt-text pairs without filtering?basic
Most alt-text does not describe its image (file names, boilerplate, SEO text, descriptions of something else). These pairs waste compute and, in a contrastive loss, act as wrong positives that pull unrelated images and texts together. In DataComp, every filter beat no filtering at every scale; at the largest scale the best filter gave 79.2% ImageNet zero-shot against 72.3% for the unfiltered 12.8B pool, with the same model and compute. Filtering also removes unsafe content and test-set copies.
How does CLIP-score filtering work, and what are its failure modes?core
Embed the image and the text with a trained CLIP model and compute their cosine similarity (187.1). Keep pairs above a threshold (LAION: 0.28 with ViT-B/32 for English) or the top fraction (DataComp: top 30% by ViT-L/14). Failure modes: it copies CLIP's blind spots and biases (rare languages, domains CLIP does not know); it gives top scores to images whose caption just repeats the visible text (nearly 40% of LAION images have strong text overlap, per T-MARS); thresholds do not transfer between CLIP models; and too strict a cut removes diversity, so intermediate subset sizes work best.
Why does DataComp fix the number of samples seen instead of the number of epochs?core
So that every dataset gets the same compute. If a filter keeps 30% of the pool, the model sees each kept sample about 3.3 times; it is not trained for less time. This isolates the question “is this a better set of examples for a fixed budget?”, which is the question a practitioner with a fixed GPU budget has. Comparing at fixed epochs would mix data quality with training length.
Why is keeping the top 30% by CLIP score better than keeping the top 5% or 100%?core
It is a trade-off between quality and diversity at fixed compute. With 100%, many samples are noise. With 5%, the set is clean but small, so it is repeated many times and covers fewer concepts; repetition gives less new information per step, and the set is biased toward content that CLIP scores easily. DataComp found the best CLIP-score subset at roughly 30% of the pool, while random subsets were always better when larger. The best fraction depends on the compute: with more samples seen, you can afford, and need, a larger set.
What do interleaved documents give that image–text pairs do not?core
Long text, several images in one context, and references across images: the format of few-shot prompts, multi-image chat and documents. MM1 found that at least 50% interleaved data was needed to keep 8-shot performance above 61% (it fell to 45% without), while caption data drove zero-shot performance up (25.8% to 39.3% as its share grew). OBELICS found interleaved data reached the same performance with about ten times fewer images. Pairs remain better for captioning, classification and OCR, so the two are mixed.
Explain SemDeDup. Why does it cluster first, and what does that cost?advanced
Embed every image (CLIP), run k-means, and inside each cluster remove items whose cosine similarity to an earlier item is at least \(1 - \varepsilon\), keeping the item farthest from the centroid. All-pairs comparison costs about \(n^2\) (\(1.9 \times 10^{17}\) for 440M items); clustering reduces it to \(\sum_c n_c^2/2\), about \(n^2/2k\) for equal clusters (paper: \(4.6 \times 10^{12}\) with \(k = 50{,}000\)). The cost is that duplicates on different sides of a cluster border are never compared, so some are missed. On LAION-440M it removed 50% of the data with under 0.5% performance loss and halved training time.
Are synthetic dense captions a good idea? What are the risks?core
Yes, with care. ShareGPT4V kept 558K images fixed and replaced short BLIP captions with dense ones: MME perception gain rose from +4.7 to +22.9 and MMBench from +0.9 to +3.0. RECAP improved text-to-image FID from 17.87 to 14.84. Risks: the captioner hallucinates and the student learns it; captions lose proper nouns and world knowledge (CapsFusion); style becomes uniform, so diversity drops; and errors are systematic, not random, so they do not average out. Mitigations: keep the original alt-text and mix or fuse both, check samples by hand and with human preference tests, and use synthetic data as a share of the mix (MM1: 7% of caption data gave several points of few-shot gain).
Design an experiment to choose the share of text-only data in VLM pretraining.advanced
Fix the model, optimizer, steps and samples seen. Sweep the text-only share (for example 0%, 5%, 10%, 20%, 30%) while keeping the caption : interleaved ratio fixed. Train at a small proxy scale with enough seeds (formula 187.10: measure seed variance first). Evaluate on multimodal zero-shot and few-shot tasks and on text-only benchmarks, because the trade-off is between them. Plot both against the share, pick the knee, and re-run the top two or three at a larger scale to confirm the ranking. Report the epochs each source receives (187.8) so repetition is visible. MM1's answer was a 5:5:1 caption : interleaved : text ratio.
A safety classifier catches 96% of unsafe images with an 8% false-positive rate. Is it good enough?core
It depends on the base rate. If 3% of the pool is unsafe, per million images it catches about 28,900 unsafe images but also flags about 77,200 safe ones, so only 27% of flagged images are unsafe, 10.6% of the data is removed, and about 1,100 unsafe images per million still pass. So: tune the threshold to the cost of each error, stack filters (image, text, hash lists for illegal content), look at samples by hand, and accept that some content must be removed with very high recall even at a large false-positive cost.
What makes video–text data harder than image–text data?core
Alignment in time. ASR narration (HowTo100M) is often about what the speaker will do or did, not what is on screen now, and much of it is not visual at all. Long videos must be cut into clips that each show one scene (Panda-70M splits 3.8M videos into consistent clips), and captions must describe events at the right time scale (InternVid uses several scales). Video is also far more expensive: storage, decoding, and visual tokens per frame. Dedup needs video fingerprints because re-uploads are re-encoded, trimmed or cropped.
Exact, near- and semantic duplicates: how do you find each, and why remove duplicates of the test set?basic
Exact: a cryptographic hash of the file bytes. Near: a perceptual hash (pHash, dHash) and a small Hamming distance (187.4), which survives resizing and re-compression. Semantic: embeddings plus a cosine threshold, made affordable by clustering (SemDeDup). Test-set duplicates must go because otherwise the model is tested on images it has memorized and the benchmark overstates its ability; DataComp removed about 3% of its 16.8B downloaded images as near-duplicates of evaluation images.

Exercises

  1. A pool has 12.8M pairs. You keep the top 20% by CLIP score and train for 12.8M samples seen. How many times is each kept pair seen on average?
  2. LAION removed about 90% of roughly 50B candidates with its CLIP filter. How many pairs did it keep? Compare with the 5.85B it reports.
  3. For SemDeDup with \(n = 10^9\) items and \(k = 100{,}000\) equal clusters, how many pair checks are needed, and how many times fewer than all pairs (\(n^2/2\))?
  4. Two unit vectors have cosine 0.995. What is their distance? Are they duplicates at \(\varepsilon = 0.01\)?
  5. Three sources have 1B, 100M and 10M items. Compute the sampling probabilities with temperature \(\alpha = 0.5\).
  6. In MM1's setup (102.4M sequences), you give a 5M-image dense-caption set a weight of 10%, packed 16 per sequence. How many epochs of that set does the run see? Is that a problem?
Answerscheck
1. 20% of 12.8M is 2.56M; \(12.8 / 2.56 = 5\) times. 2. About \(0.1 \times 50\text{B} = 5\text{B}\), consistent with the reported “just short of 6 billion” (5.85B), since 90% is approximate. 3. Each cluster has \(10^4\) items: \(10^5 \times 10^4 \times (10^4 - 1)/2 \approx 5 \times 10^{12}\) checks, against \(n^2/2 = 5 \times 10^{17}\): about \(10^5\) times fewer (exactly a factor \(k\)). 4. \(\lVert a - b \rVert = \sqrt{2(1 - 0.995)} = \sqrt{0.01} = 0.1\). They are duplicates if \(\cos \ge 1 - 0.01 = 0.99\): yes. 5. \(\sqrt{N}\): 31,623, 10,000 and 3,162; the sum is 44,785, so \(p = 0.706, 0.223, 0.071\). 6. \(0.1 \times 102.4\text{M} \times 16 = 163.8\)M images, \(163.8 / 5 = 32.8\) epochs. Yes: a 33-times repeated set will be memorized; lower its weight, or make the set larger.

Key takeaways

  • With the model and compute fixed, data choices alone moved DataComp's ImageNet zero-shot from 72.3% to 79.2%. Data is a first-class design choice.
  • Large datasets are funnels: tens of billions of crawled candidates, cheap filters first, then CLIP-score filtering (187.1, 187.2), safety filters, eval decontamination and dedup. DataComp-1B is 1.59% of its raw pool.
  • CLIP-score filtering works but has blind spots (text spotting, CLIP's biases); intermediate subset sizes (about 30%) are best at fixed compute.
  • Dedup works at three levels: exact hashes, perceptual hashes (187.4) and semantic embeddings with clustering (187.5, 187.6); SemDeDup halved LAION-440M with under 0.5% loss.
  • Pairs give zero-shot naming skill; interleaved documents give few-shot and multi-image skill; text-only data protects language (MM1: 45 / 45 / 10). Video adds time alignment problems.
  • Dense re-captions give large gains (ShareGPT4V, RECAP) but can hallucinate and lose world knowledge; mix them with original text.
  • The mixture (187.8, 187.9) is a hidden hyperparameter. Tune it with controlled ablations at fixed samples seen, several seeds (187.10), broad evaluation, and a check at larger scale.

Key formulas

\[ s = \frac{f(x)^\top g(y)}{\lVert f(x)\rVert \lVert g(y)\rVert}, \qquad \cos \ge 1-\varepsilon \iff \lVert \mathbf{a}-\mathbf{b}\rVert \le \sqrt{2\varepsilon} \]
\[ p_i = \frac{N_i^{\alpha}}{\sum_j N_j^{\alpha}}, \qquad \text{epochs}_i = \frac{p_i S u_i}{N_i}, \qquad r \ge 2\left(\frac{1.96\,\sigma}{\delta}\right)^2 \]

References for this session

  1. Schuhmann et al. (2022). LAION-5B: An open large-scale dataset for training next generation image-text models
  2. Gadre et al. (2023). DataComp: In search of the next generation of multimodal datasets
  3. Laurençon et al. (2023). OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
  4. Chen et al. (2023). ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
  5. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 4: Training Data and Preprocessing for VLMs
  6. Noyan, Farré, Marafioti, Zohar, Vision Language Models (O’Reilly, 2026), chapter 4: Training Data and Preprocessing for VLMs
  7. Schuhmann et al. (2022). LAION-5B: An open large-scale dataset for training next generation image-text models
  8. Gadre et al. (2023). DataComp: In search of the next generation of multimodal datasets
  9. Laurençon et al. (2023). OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
  10. Chen et al. (2023). ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
  11. McKinzie et al. (2024). MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  12. Abbas, Tirumala, Simig, Ganguli & Morcos (2023). SemDeDup: Data-efficient learning at web-scale through semantic deduplication
  13. Fang, Jose, Jain, Schmidt, Toshev & Shankar (2023). Data Filtering Networks
  14. Maini, Goyal, Lipton, Kolter & Raghunathan (2023). T-MARS: Improving Visual Representations by Circumventing Text Feature Learning
  15. Zhu et al. (2023). Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
  16. Awadalla et al. (2024). MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
  17. Miech et al. (2019). HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
  18. Bain, Nagrani, Varol & Zisserman (2021). Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
  19. Wang et al. (2023). InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
  20. Chen et al. (2024). Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
  21. Segalis, Valevski, Lumen, Matias & Leviathan (2023). A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  22. Li et al. (2024). What If We Recaption Billions of Web Images with LLaMA-3?
  23. Yu et al. (2023). CapsFusion: Rethinking Image-Text Data at Scale
  24. LAION (2024). Releasing Re-LAION-5B: transparent iteration on LAION-5B with additional safety fixes
  25. Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision
  26. Alayrac et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning