Preparing unstructured data for AI.
How to find, clean, and curate the files an AI project needs, so models learn from the right data, at a lower cost.
Read the full guide.
Eight chapters, a cost estimate, a readiness score, and a checklist, open to everyone.
What uncurated data does to an AI project.
It lowers the quality of results first, and adds cost and risk along the way.
Three problems, one cause.
Quality.
Most of the files in an enterprise were never written for AI. They are copies, drafts, old versions, logs, and exports, saved across many storage systems over many years. A model does not know which file is the latest. Duplicates give one source too much weight, and old versions teach it facts that are no longer true.
Risk.
Files with personal or confidential data end up in places they were never meant to go, and a model can repeat what it saw. Varonis based its finding on 1,000 real IT environments.
Money.
Point a project at a whole share, and you pay to move, store, and process every file in it, often more than once.
A smaller, better dataset, chosen on purpose, gives better answers. It also costs less to store, move, and process.
Estimate what uncurated data costs you.
Five numbers. Two minutes. A figure you can take to your next AI project review.
Enter your own numbers in the white boxes. The example shows the method, not a benchmark. Your numbers stay in your browser.
Every file you leave out in step 3 lowers D before the pipeline runs, and every rule you write lowers E for the next project. Curation pays twice: once in cost, once in quality.
What AI-ready means for file data.
A definition you can check, and a score that shows where you stand.
AI-ready data is a defined set of files that are current, described by metadata, free of content the model should not see, and traceable back to their source.
Only 20% of unstructured data is AI ready.
Few organizations are there yet.
In a survey of 2,050 leaders published by Snowflake and conducted by Omdia, only 20% of unstructured data was considered AI ready.
Each part of the definition maps to one step in this guide: see the files, describe them, leave out what does not belong, curate the set, and keep a record. Related term: AI-ready data.
Score your readiness.
Rate each statement from 1 (never true) to 5 (always true). Your score adds up as you go.
Five steps from raw files to a trusted dataset.
The same five steps work for a chatbot, a training run, or a retrieval-augmented generation (RAG) pipeline.
- STEP 1See
- STEP 2Describe
- STEP 3Leave out
- STEP 4Curate
- STEP 5Deliver and record
See every file.
You cannot prepare files you cannot find. File data sits on network attached storage (NAS) from several vendors, in object storage, in the cloud, and at remote sites, and each system reports only on itself. Thales reports that only 34% of organizations know where all their data is. Build one inventory across all of them. File metadata, such as name, path, size, type, owner, and dates, is enough to see what exists, where it lives, and who owns it. Related: metadata catalog, data silos.
One view of every file a project could use, on every system.
Which systems hold project files that nobody has looked at in years?
Describe files with metadata.
System metadata tells you what a file is. Business metadata tells you why it matters. Add tags for project, owner, sensitivity, and retention. Tags can come from rules that read paths and file names, from people who know the data, and from AI suggestions that a person reviews and approves. Related: metadata enrichment, data tagging.
Files you can select by project, owner, sensitivity, and approval.
Which tag would tell you a file is approved for AI use?
Leave out what does not belong.
Remove duplicates (find them by size and name, then confirm with a checksum), old versions, and ROT data. Flag the folders that usually hold personal data, such as HR, finance, and legal shares, and have your security team scan them before anything is approved. Related: duplicate file detection, sensitive data discovery.
A smaller set, with less cost and less risk.
Is each exclusion written down as a rule the next project can reuse?
Curate the dataset.
A curated dataset, often called a data corpus, is a named set of files chosen for one purpose, with an owner, a rule, and a version. Example: every PDF under /docs/manuals/, latest approved version only, tagged “public,” owned by the documentation team, version 3. Related: data curation.
A named dataset you can open and check when a result looks wrong.
Who owns this dataset, and what rule defines it?
Deliver it and keep a record.
Copy or move the curated set to where the pipeline reads it, such as object storage, a data lake, or a data lakehouse. Record the file list, the rule, the version, the date, and who approved it. That record is your data lineage, and it matters most for RAG, where the model quotes your files back to users.
Results you can repeat, and answers you can give an auditor.
Could you remove one file from the next version of the dataset today?
A one-time cleanup helps one project. Data keeps arriving, so write each step down as a rule. The next dataset then starts from rules you already trust.
Check a dataset before it reaches a model.
Eight questions. Tick each one you can answer yes to today.
Before a dataset reaches a model
0 of 8 ready
Use it at every review.
Run the checklist when a dataset is first built, and again before each new version goes to the pipeline. A “no” on any line is a reason to stop and fix the rule, not the file.
Lines 1 to 4 come from steps 1 to 3. Lines 5 to 8 come from steps 3 to 5. If you scored under 15 in chapter 3, start with lines 1 and 2.
Print this page, or save it as a PDF from your browser, and bring the checklist to your next AI project review.
What good looks like.
Teams that curate first, with very different kinds of data.
Cancer research center
The pipelines kept running throughout, and the team now sees which runs produce results and which leave data behind.
$180B entertainment leader
The old lakehouse workflow for media was retired, and business users now ask questions in plain language instead of filing tickets.
Autonomous driving automaker
From a test scan of open self-driving datasets. The plan: keep recordings where they land, and send models only the data they lack.
Biopharma company
Indexes instrument and research data across on-premises storage, the cloud, and an archive, at several sites, and tags the datasets that train its drug discovery models. Scientists find the right files from one search.
None of these teams pointed a model at a whole share. They started by seeing what they had, described it with metadata, and sent only the files that belonged.
How Diskover helps.
Diskover builds one metadata index across all your storage, from every vendor and every location, so you can find, understand, and act on your file data.
See
Index file metadata across NAS, object storage, cloud, and remote sites, and search it from one place.
Steps 1 and 3
Curate
Tag files with rules and plugins. Your own AI tools, connected through the AI Connector, suggest tags, and people approve them before they apply.
Steps 2 and 4
Deliver
Move the curated set to the AI pipeline with a policy, and keep the record of what went where.
Step 5
No tag decides what a model learns from until a person checks it. Diskover works from file metadata, so selecting a dataset does not mean reading every file.
See how Diskover feeds AI and BI pipelines with curated file data.
Explore AI-ready dataQuestions people ask.
Short answers to the questions AI, data, and storage teams ask most.
How long does it take to prepare unstructured data for AI?
It depends on how many storage systems hold the data and how well it is already described. Building the first inventory and tags takes the longest. After that, the inventory and tags are reused, so each new dataset takes far less time than the first.
Do you need to read file contents to prepare data for AI?
Not to find and select it. File metadata, such as name, path, type, size, owner, and dates, plus tags that add business context, such as project and approval status, is enough to build an inventory, remove duplicates and old versions, and choose a dataset. Content tools, such as sensitive data scanners or document parsers, can then run on the smaller set you selected, which saves time and cost.
What is the difference between a data corpus and a data lake?
A data lake stores large amounts of raw data for many uses. A data corpus is a curated set of files chosen for one purpose, with an owner, a rule, and a version. A corpus can live inside a data lake.
How do you keep sensitive files out of a training set?
Tag the folders and file types that usually hold personal or confidential data, have your security team scan those files, and make “approved for AI use” a required tag. Files without that approval stay out of the dataset.
Who should own AI data preparation?
It is usually shared. IT and storage teams own the inventory, data owners in each business team, often called data stewards, approve tags and datasets, and the AI or data team defines what the model needs. One named owner per dataset keeps decisions clear.
See every file before you choose the ones your model learns from.
Book a demo to see Diskover index your storage and build an AI dataset from file metadata.