Data curation

Data curation is the ongoing work of selecting, organizing, describing, and maintaining data so the right people and systems can find it, trust it, and use it. For unstructured data, it starts from metadata: deciding which files matter, adding the context they lack, and keeping that description current as the estate changes.

Key takeaways

Curation is ongoing, while preparing data for one project is a single pass.
Curation turns large, mixed file collections into sets that people and AI can trust.
Metadata is what lets curation scale, because files are selected and described without opening them one by one.

Why does data curation matter?

Because analytics and AI are only as good as the data they are given. An uncurated estate is full of duplicates, outdated versions, test runs, and files nobody can explain, so every new project starts by searching and cleaning again. Curated data is found quickly, trusted by the people who use it, and reused from one project to the next.

What is an example of data curation?

A research group curates its microscope images. It removes test runs and duplicates, tags each image with the compound, assay, and date from the lab system, and marks the sets a scientist has approved. New images join the right set automatically by rule, so the curated collection stays current without anyone sorting files by hand.

What is a curated dataset?

A curated dataset is a set of files that has been selected for a purpose, cleaned of duplicates and outdated versions, described with the context people need, and approved by someone who knows the data. A curated set of training images, for example, holds only usable images, each tagged with its source and labels. It is also kept current: new files that meet the rules join it, and files that no longer belong leave it.

How do you keep a curated dataset together?

Usually without moving the files. A curated dataset can stay spread across several storage systems and still act as one set, held together in one of three ways, often combined:

Tags: each file carries a tag with the dataset’s name and version, so a search for that tag returns the whole set, wherever the files sit.
Saved searches: the set is defined as a rule, such as a project, a date range, and an approval status, so new files that match join automatically.
File lists: for AI training and audits, the exact list of files, with their paths and dates, is recorded at a point in time, so the same set can be rebuilt later.

Tags and saved searches keep the set current, and file lists keep a record of exactly what was used.

What is the difference between data curation and data preparation?

Data curation is ongoing and covers the whole estate: it keeps data selected, described, and trustworthy over time. Data preparation is the set of steps that gets one chosen set ready for one use, such as converting formats or splitting data for training. Good curation makes preparation faster, because the right data is already found and described.

What is the difference between data curation and data filtering?

Filtering is one step: selecting the files that match certain conditions, such as a project, a date range, or a file type. Curation is the whole practice around it: choosing the right conditions, removing what does not belong, describing what remains, having someone approve it, and keeping it current over time. Every curated set starts with a filter, but a filter alone does not make data trustworthy.

Is data curation part of data lifecycle management?

They are closely related but answer different questions. Data lifecycle management decides where data sits and how long it is kept. Data curation decides which data is valuable and how it is described so people can use it. Both work from the same metadata, and each makes the other better: curation tells lifecycle rules which files are worth protecting, and lifecycle rules keep curated data on the right storage over time.

Can you move or tier data without curating it first?

Yes, but it usually costs more than it saves. A rule based only on file age can move data without any curation, yet it may send files people still need to slow storage, and it carries duplicates, temporary files, and outdated versions along to the new location, where you pay to store them again. A light round of curation first, such as removing obvious waste and tagging the data that must stay close, makes each move smaller, cheaper, and safer.

What does a data curator do?

A data curator decides what is kept, how it is described, and which sets are ready for use. Day to day, that means reviewing new data, fixing missing or wrong descriptions, removing what no longer belongs, and publishing approved sets for teams and AI projects. The role often sits with data stewards, research data managers, or librarians, rather than with the storage team.

How do you curate unstructured data at scale?

By working from metadata rather than file by file:

Index every storage system, so curation starts from what actually exists.
Remove waste, such as duplicates, temporary files, and outdated versions.
Add context with enrichment and tags, such as project, owner, sensitivity, and approval status.
Define curated sets as saved searches, so each set is a rule rather than a copy.
Keep sets current by running the rules after every scan.

Can AI help curate data?

Yes, as an assistant. AI can suggest tags, flag files that look out of place, and answer questions about the data in plain language, which speeds up review. The decisions stay with people: a curator approves suggested tags before they are applied, so the curated set reflects human judgment rather than a model’s guess.

How does data analytics help with data curation?

It shows where to start and whether curation is working. File analytics reveals the largest and oldest groups of data, where duplicates pile up, who owns what, and which folders nobody can explain, so curators know which areas need attention first. Over time, the same reports track progress, such as the share of data with a known owner, the share that is tagged, and how much waste has been removed.

How does Diskover help?

Diskover curates unstructured data from metadata: it indexes every storage system, enriches files with business context, and lets teams save curated sets that stay current with every scan.

Explore data curation

Book a demo