LLM training data providers and sources explained

Language-model development uses data at several stages, and each stage values different material. Pretraining exposes a model to very large amounts of broad text and code so that it learns language, facts and patterns. Post-training, which includes supervised fine-tuning and preference tuning, uses smaller sets of carefully constructed examples to shape how the model responds. Evaluation uses held-out material to measure performance, and it must be kept out of training to remain meaningful.

A data owner should know which stage its collection could serve. A large, varied archive of professional writing might suit pretraining or domain adaptation. A set of expert-reviewed question-and-answer pairs is closer to post-training material. A carefully constructed, never-published test set may be most valuable as evaluation data, under a licence that forbids training on it.

Broad web text is widely available, so the text that draws interest tends to offer something the open web does not: depth in a specialised field, careful editing, reliable structure, or language and register that are under-represented. Long-form technical writing, edited reference material, professional correspondence with appropriate permissions and well-organised archives of reports can all fall into that category.

Value also depends on cleanliness. Text with consistent formatting, little duplication, clear document boundaries and useful metadata such as date, author role or subject is easier to filter and use. A smaller, well-curated corpus can be more useful to a developer than a much larger export full of boilerplate and repeated templates.

Source code teaches models syntax, structure and problem solving. Collections that pair code with context are especially useful: commit histories that show how code changed, review comments that explain why, tests that define correct behaviour, and documentation that links intent to implementation.

Code also carries particular risks. Repositories can contain credentials, customer data in fixtures, third-party code under open-source licences with conditions, and confidential business logic. Before offering code for licensing, an owner should establish which repositories are wholly owned, scan for secrets, record the open-source components included and decide whether any internal systems must be excluded.

Domain data covers specialised fields such as law, medicine, finance, engineering, science or industrial operations. It helps models handle terminology and reasoning that general text covers thinly. It also tends to involve the strictest rights questions, because it often contains client, patient or customer information subject to confidentiality duties or regulation.

Multimodal data combines text with images, audio, video or structured records: illustrated manuals, annotated diagrams, recorded meetings with transcripts, or scanned documents with extracted text. Language models that accept several input types need aligned examples, where the text genuinely describes or accompanies the other medium. Alignment quality, not just volume, determines usefulness.

Language-model developers increasingly ask where each part of a corpus came from and on what basis it may be used for training. An owner should be able to explain whether material was created by employees, commissioned from contractors, contributed by users or customers, or licensed in from others, and what each source’s agreement allows.

Holding a copy of text is not the same as holding the right to license it. Contributor agreements, customer contracts, publisher licences and privacy commitments can all limit secondary use. Where permissions are unclear for part of a collection, it is usually better to exclude that part than to offer it with caveats. Our provenance and licensing-model guides explain how to record permissions and define the uses you grant.

LLM training data reaches a model through a chain. At one end are the creators and holders of material: publishers, enterprises, professional firms, research groups and individual experts. In the middle are intermediaries that aggregate, clean, annotate or package data. At the other end is the developer that trains the model and the products built on it.

An owner can enter at different points. It can license directly to a developer, through an aggregator that combines collections, or into a programme that pays for expert contributions rather than stored records. Each position carries different obligations for documentation, delivery and acceptance, and a different share of control over how the data is used downstream.

With that groundwork, the first conversation with an LLM training data provider or developer can focus on fit and terms instead of basic questions about what the collection is.