eGuide | AI data preparation

Preparing unstructured data for AI.

How to find, clean, and curate the files an AI project needs, so models learn from the right data, at a lower cost.

Read the full guide.

Eight chapters, a cost estimate, a readiness score, and a checklist, open to everyone.

Curate first. Then feed the model.

Most AI projects start by pointing a model at a whole share. This guide shows how to prepare data for AI instead: see every file, describe it with metadata, leave out what does not belong, curate a dataset with an owner, and keep a record of what went in. Curated data gives better answers, and it costs less to store, move, and process. It works on any storage, from any vendor, and it starts from file metadata, not file contents.

  1. 01
    What uncurated data does to an AI project.
  2. 02
    Estimate what it costs you.
  3. 03
    What AI-ready means for file data.
  4. 04
    Five steps from raw files to a trusted dataset.
  5. 05
    Check a dataset before it reaches a model.
  6. 06
    What good looks like.
  7. 07
    How Diskover helps.
  8. 08
    Questions people ask.
Key takeaways
  • Most enterprise files were never written for AI. Duplicates, old versions, and sensitive files lower the quality of results, add risk, and raise costs when they reach a model.
  • Preparation starts with metadata. A file’s name, path, type, size, owner, and dates describe it before anyone opens it.
  • Curate a defined dataset instead of pointing a model at a whole share.
  • Keep a record of what went in, so every result can be traced and repeated.
01
Chapter 1 | Why it matters

What uncurated data does to an AI project.

It lowers the quality of results first, and adds cost and risk along the way.

60%
of AI projects unsupported by AI-ready data will be abandoned through 2026, Gartner predicts
$108B
in AI investment wasted on weak data foundations
99%
of organizations have sensitive information exposed to AI
38%
of AI failures in infrastructure and operations were caused by poor data quality or limited data availability

Three problems, one cause.

Quality.

Most of the files in an enterprise were never written for AI. They are copies, drafts, old versions, logs, and exports, saved across many storage systems over many years. A model does not know which file is the latest. Duplicates give one source too much weight, and old versions teach it facts that are no longer true.

Risk.

Files with personal or confidential data end up in places they were never meant to go, and a model can repeat what it saw. Varonis based its finding on 1,000 real IT environments.

Money.

Point a project at a whole share, and you pay to move, store, and process every file in it, often more than once.

Duplicates

Answers that give one source too much weight, and storage, transfer, and processing paid more than once.

Old versions

Answers built on facts that are no longer true. See ROT data.

Sensitive files

Compliance exposure, and a model that can repeat what it saw. See PII.

What this means

A smaller, better dataset, chosen on purpose, gives better answers. It also costs less to store, move, and process.

02
Chapter 2 | Cost estimate

Estimate what uncurated data costs you.

Five numbers. Two minutes. A figure you can take to your next AI project review.

Enter your own numbers in the white boxes. The example shows the method, not a benchmark. Your numbers stay in your browser.

A
Data in scope for the AI project
Everything the project would read if nobody curated it: shares, buckets, and archives.
TB
B
Yearly cost per TB in the pipeline
Storage for the copy, plus transfer and processing such as parsing and embedding. Use your own cloud bills and contracts.
$
C
Share that adds nothing
Duplicates, old versions, and ROT data. Start with 30%. A 2016 Veritas study put ROT data at 33% of what organizations store.
%
D
Yearly spend on files that add nothing
A × B × C
$90,000
E
Hours spent checking and cleaning AI data manually
Number of people who find, check, or clean files for AI projects × hours each spends per week.
×
hrs
$
Hourly cost of their time
Fully loaded hourly cost of the people in E.
$
F
Yearly cost of that time
18 hrs × $75 × 52 weeks
$70,200
Where the savings come from

Every file you leave out in step 3 lowers D before the pipeline runs, and every rule you write lowers E for the next project. Curation pays twice: once in cost, once in quality.

03
Chapter 3 | Readiness

What AI-ready means for file data.

A definition you can check, and a score that shows where you stand.

Definition

AI-ready data is a defined set of files that are current, described by metadata, free of content the model should not see, and traceable back to their source.

Few organizations are there yet.

In a survey of 2,050 leaders published by Snowflake and conducted by Omdia, only 20% of unstructured data was considered AI ready.

Each part of the definition maps to one step in this guide: see the files, describe them, leave out what does not belong, curate the set, and keep a record. Related term: AI-ready data.

Score your readiness.

Rate each statement from 1 (never true) to 5 (always true). Your score adds up as you go.

1 never · 5 always
04
Chapter 4 | Method

Five steps from raw files to a trusted dataset.

The same five steps work for a chatbot, a training run, or a retrieval-augmented generation (RAG) pipeline.

  1. STEP 1See
  2. STEP 2Describe
  3. STEP 3Leave out
  4. STEP 4Curate
  5. STEP 5Deliver and record
Figure 1. Five steps take raw files from every storage system to a dataset a model can use, with a record of what went in.
Step 1

See every file.

You cannot prepare files you cannot find. File data sits on network attached storage (NAS) from several vendors, in object storage, in the cloud, and at remote sites, and each system reports only on itself. Thales reports that only 34% of organizations know where all their data is. Build one inventory across all of them. File metadata, such as name, path, size, type, owner, and dates, is enough to see what exists, where it lives, and who owns it. Related: metadata catalog, data silos.

You get

One view of every file a project could use, on every system.

Ask yourself

Which systems hold project files that nobody has looked at in years?

Step 2

Describe files with metadata.

System metadata tells you what a file is. Business metadata tells you why it matters. Add tags for project, owner, sensitivity, and retention. Tags can come from rules that read paths and file names, from people who know the data, and from AI suggestions that a person reviews and approves. Related: metadata enrichment, data tagging.

You get

Files you can select by project, owner, sensitivity, and approval.

Ask yourself

Which tag would tell you a file is approved for AI use?

Step 3

Leave out what does not belong.

Remove duplicates (find them by size and name, then confirm with a checksum), old versions, and ROT data. Flag the folders that usually hold personal data, such as HR, finance, and legal shares, and have your security team scan them before anything is approved. Related: duplicate file detection, sensitive data discovery.

You get

A smaller set, with less cost and less risk.

Ask yourself

Is each exclusion written down as a rule the next project can reuse?

Step 4

Curate the dataset.

A curated dataset, often called a data corpus, is a named set of files chosen for one purpose, with an owner, a rule, and a version. Example: every PDF under /docs/manuals/, latest approved version only, tagged “public,” owned by the documentation team, version 3. Related: data curation.

You get

A named dataset you can open and check when a result looks wrong.

Ask yourself

Who owns this dataset, and what rule defines it?

Step 5

Deliver it and keep a record.

Copy or move the curated set to where the pipeline reads it, such as object storage, a data lake, or a data lakehouse. Record the file list, the rule, the version, the date, and who approved it. That record is your data lineage, and it matters most for RAG, where the model quotes your files back to users.

You get

Results you can repeat, and answers you can give an auditor.

Ask yourself

Could you remove one file from the next version of the dataset today?

Rules, not cleanups

A one-time cleanup helps one project. Data keeps arriving, so write each step down as a rule. The next dataset then starts from rules you already trust.

05
Chapter 5 | Checklist

Check a dataset before it reaches a model.

Eight questions. Tick each one you can answer yes to today.

Before a dataset reaches a model

0 of 8 ready

Use it at every review.

Run the checklist when a dataset is first built, and again before each new version goes to the pipeline. A “no” on any line is a reason to stop and fix the rule, not the file.

Lines 1 to 4 come from steps 1 to 3. Lines 5 to 8 come from steps 3 to 5. If you scored under 15 in chapter 3, start with lines 1 and 2.

Tip

Print this page, or save it as a PDF from your browser, and bring the checklist to your next AI project review.

06
Chapter 6 | Proof

What good looks like.

Teams that curate first, with very different kinds of data.

Cancer research center

27Mfiles indexed across 3 sites
0.5 PBof temporary and training data its AI pipelines left behind, reclaimed
$250Kin storage overspend avoided, while keeping its AI data pipelines flowing

The pipelines kept running throughout, and the team now sees which runs produce results and which leave data behind.

$180B entertainment leader

260 PBindexed across 100B+ files
1metadata catalog, curated into Snowflake for analytics and AI
30+data and operational users asking questions of the whole estate

The old lakehouse workflow for media was retired, and business users now ask questions in plain language instead of filing tickets.

Autonomous driving automaker

500 TBa day expected from a planned 100-vehicle fleet
<9%of 2,067 test recordings matched one training need: rain
1,000recordings with a suspect clock, flagged before any training

From a test scan of open self-driving datasets. The plan: keep recordings where they land, and send models only the data they lack.

Biopharma company

Indexes instrument and research data across on-premises storage, the cloud, and an archive, at several sites, and tags the datasets that train its drug discovery models. Scientists find the right files from one search.

The pattern

None of these teams pointed a model at a whole share. They started by seeing what they had, described it with metadata, and sent only the files that belonged.

07
Chapter 7 | Diskover

How Diskover helps.

Diskover builds one metadata index across all your storage, from every vendor and every location, so you can find, understand, and act on your file data.

Deliver

Move the curated set to the AI pipeline with a policy, and keep the record of what went where.

Step 5

Explore data mobility →

AI-assisted, human-approved

No tag decides what a model learns from until a person checks it. Diskover works from file metadata, so selecting a dataset does not mean reading every file.

See how Diskover feeds AI and BI pipelines with curated file data.

Explore AI-ready data
08
Chapter 8 | Questions

Questions people ask.

Short answers to the questions AI, data, and storage teams ask most.

How long does it take to prepare unstructured data for AI?

It depends on how many storage systems hold the data and how well it is already described. Building the first inventory and tags takes the longest. After that, the inventory and tags are reused, so each new dataset takes far less time than the first.

Do you need to read file contents to prepare data for AI?

Not to find and select it. File metadata, such as name, path, type, size, owner, and dates, plus tags that add business context, such as project and approval status, is enough to build an inventory, remove duplicates and old versions, and choose a dataset. Content tools, such as sensitive data scanners or document parsers, can then run on the smaller set you selected, which saves time and cost.

What is the difference between a data corpus and a data lake?

A data lake stores large amounts of raw data for many uses. A data corpus is a curated set of files chosen for one purpose, with an owner, a rule, and a version. A corpus can live inside a data lake.

How do you keep sensitive files out of a training set?

Tag the folders and file types that usually hold personal or confidential data, have your security team scan those files, and make “approved for AI use” a required tag. Files without that approval stay out of the dataset.

Who should own AI data preparation?

It is usually shared. IT and storage teams own the inventory, data owners in each business team, often called data stewards, approve tags and datasets, and the AI or data team defines what the model needs. One named owner per dataset keeps decisions clear.

Next step

See every file before you choose the ones your model learns from.

Book a demo to see Diskover index your storage and build an AI dataset from file metadata.

Talk to usWeb diskoverdata.comEmail sales@diskoverdata.comPhone (800) 560-5853
Scroll to Top