Data corpus

A corpus is the specific set of files a model will use, named as a set rather than as a location. Defining one means writing the filter that produces it: this project, these formats, files read within this period, excluding these owners and these paths. A folder is not a corpus, because its contents change without anyone deciding.

Key takeaways

A corpus is chosen on purpose, through a written filter.
It is defined by a filter rather than a folder, so it stays stable as storage changes.
Training data provenance records where each file came from and when it joined the set.

What is training data provenance?

Provenance is the record of where each file in a corpus came from, who owned it, and when it entered the set. It lets an organization answer questions about a model’s output months later. Provenance has to be captured at selection time, because it cannot be reconstructed afterward.

How do you build a corpus using metadata?

By writing the selection as a filter over a metadata catalog, then checking the result before anything moves. The usual steps:

Start from the use case. Write down what the model must learn and which data could teach it, such as one product line, one study, or one period.
Turn it into a filter. Express the selection with attributes such as project, file type, date range, owner, and tags.
Remove what does not belong. Exclude duplicates, scratch and temporary files, and anything tagged as sensitive or under legal hold.
Review the result. Check the file count, total size, and the largest folders and owners with the people who know the data, and adjust the filter.
Save it and deliver it. Save the filter, record the file list with its provenance, and copy or point the selected files to the AI pipeline.

Do you need to open files to build a corpus?

Not to select them. Path, file type, size, dates, owner, and tags already say which files belong to a project, a period, or a format, so the decision can be made from metadata alone. Only the files that are selected need to be opened, and that happens later, inside the AI pipeline. Opening every file just to decide whether it qualifies is slow and costly on large estates, and it exposes content that was never needed for the model.

How do you keep a corpus the same over time?

By recording the exact file list, not only the filter. A saved filter returns a different set as files are added, changed, or deleted, so the corpus used for one training run needs its own list of paths, sizes, and modified dates at the time it was built. That list lets a team rebuild the same set, compare it with a newer one, and show which files a model was trained on.

How does application metadata help choose AI training data?

It lets teams select data by what happened when it was created, without opening the files. An autonomous vehicle program can join its drive logs, which record location, weather, time of day, and sensor setup, to the recordings they describe, and pick every night drive in rain in minutes. A drug discovery team can join instrument and lab records to microscope images, and select every image from one compound or assay. In both cases, the application already knew the context, and the catalog makes it searchable.

How does Diskover help?

Diskover lets your team define a corpus as a saved search over the enriched catalog, so the same set can be reviewed, rebuilt, and traced back to its source files.

Explore AI and BI pipelines

Book a demo