GLOSSARY
What is data discovery for unstructured data?
Data discovery for unstructured data is the process of finding out what an organization holds across every storage system, using file metadata rather than file contents. It produces a named list of files, not an estimate. This page covers file analytics, duplicates and orphaned data, classification from metadata, reclaimable capacity, and how saved searches keep the index current.
Home | Resources | Glossary | Data Discovery and File Analytics
DEFINITION
Data discovery for unstructured data is the process of finding out what an organization actually holds, across every storage system, without opening the files. It reads file attributes into one index, then answers questions about that index: what exists, how old it is, who owns it, what it duplicates, and what has not been read in years. The output is a list, not an estimate.
KEY TAKEAWAYS
52%
of stored data is dark, meaning its value is unknown
30 to 50%
of an estate turns out to be reclaimable once analyzed
36%
of stored data is what organizations themselves report as dark or unclassified
What data discovery is.
Discovery is the work of replacing an assumption with a measurement. Our blog post on file-level data intelligence explains why the unit of measurement has to be the file.
Storage vendor reports compared with cross-vendor discovery:
| STORAGE VENDOR REPORTS | CROSS-VENDOR DISCOVERY | |
|---|---|---|
| Scope | One storage platform at a time | Every storage system in one index |
| Source of facts | The platform’s own records | File attributes read into one index, without opening files |
| Duplicates across systems | A copy on another platform is invisible | Identified across storage systems |
| Orphaned data | Rarely appears in a capacity report | Found from the owner attribute the index holds |
| Output | A capacity view of one platform | A named list of files |
| Decisions it supports | Decisions about one platform | Estate decisions, such as what to tier or what to retire |
What is file analytics?
File analytics is the analysis of file metadata to answer questions about a storage estate. Typical questions are how much capacity each project consumes, what share has not been read in three years, where duplicates sit, and which owners account for the most growth.
What is file system observability?
File system observability is continuous visibility into what is happening across file systems: what is growing, what is being read, what has gone quiet, and where capacity is heading. It is the operational half of discovery, run on a schedule rather than as a one-time assessment.
What is the difference between storage vendor reporting and cross-vendor discovery?
A storage vendor tool reports accurately on its own platform and is blind to everything else. Cross-vendor discovery reads PowerScale, ONTAP, S3, Azure Blob, and the rest into one index. Most decisions, such as what to tier or what to retire, are estate decisions, so a per-vendor view answers the wrong question.
What is metadata analytics?
Metadata analytics is the analysis of file metadata, such as size, owner, age, and location, to answer questions about a data estate without opening any file. It is another name for file analytics. Typical results are capacity by project, the share of data not read in years, and the owners behind the most growth.
What is the difference between content analytics and file analytics?
Content analytics reads what is inside files, such as the text of a document, to classify them. File analytics reads only file metadata, such as name, path, owner, and dates, and leaves the contents untouched. File analytics can cover a whole data estate quickly because it never opens a file, and it narrows the set of files that any later content review needs to look at.
Duplicates, orphaned data, and sensitive files.
Each of these questions starts from an attribute the index already holds, which is also how unstructured data governance gets its evidence.
How do you find files nobody has opened in years?
Last access date is the mechanism. An index that records last access for every file across every system turns the question into a filter. The practical difficulties are that some storage systems update last access on backup reads, and that mount options can disable the attribute, so both need checking first.
How do you find duplicate files across storage systems?
Comparison can start with the least costly signals, such as identical size and name. Confirmation uses a content hash, which computes a fingerprint from the file itself and is the only method that proves two files are identical. Working across vendors matters here, because the same file copied between a NAS share and an object store is invisible to either system alone.
What is orphaned or abandoned data?
Orphaned data is data whose owner account no longer exists, usually because someone left. Abandoned data still has a valid owner but no longer has a purpose, such as debug output and core dumps that a workflow generated and never cleared. Both accumulate quietly, and neither shows up in a capacity report.
What is sensitive data discovery?
Sensitive data discovery is the work of locating files likely to hold regulated or confidential material. A metadata-only approach surfaces candidates from name, path, format, and location, then leaves the judgment to a person. It answers where to look first. After an incident, that is the question everything else waits on, and an index turns it into a search rather than an investigation.
What is the difference between data deduplication and duplicate file detection?
Data deduplication is a storage system feature that stores repeated blocks of data once, inside one system, without changing what users see. Duplicate file detection finds whole files that exist more than once, often on different storage systems, so a person can decide which copies to keep. Deduplication saves space inside one system automatically, and duplicate file detection shows the copies that sit across systems, where no single system can see them.
Data classification and reclaimable capacity.
A count only becomes useful when it turns into a named list of files. The five steps in 5 steps to cut storage costs show what that list is then used for.
How do you measure how much of an estate is reclaimable?
Combine three filters: files not read for a defined period, duplicates, and files belonging to closed projects or departed owners. Across Diskover deployments, the reclaimable share runs 30 to 50%, with top performers cutting their footprint by half or more.
How is data classification done without reading file contents?
Classification from metadata uses signals the file system and the index already hold: name, extension, path, location, age, owner, permissions, and structural detail supplied by format-aware plugins. It groups files by what they are and where they sit. It does not detect what a document says, and it is faster and less invasive because of that.
What is the difference between file metadata and a storage audit log?
File metadata records the owner of a file and the dates it was created, modified, and last accessed, so a catalog can show whose file it is and when activity happened. A storage audit log records which person performed each action, which the catalog does not hold. The two answer different halves of the same question, so they are read together when you need to know both what changed and who changed it.
How do you measure a data footprint?
A data footprint is the total capacity an organization’s data occupies, every copy included, across every storage system and tier. It is measured by indexing file metadata across all systems and adding up size by system, project, owner, and age. The breakdown is the useful part, because it shows which share is active, which share is cold, and which share is duplicated.
Saved searches and index freshness.
An index is accurate as of its last crawl, so discovery becomes monitoring only when it repeats. From there, it feeds data lifecycle management rather than a one-time cleanup.
What is a saved search?
A saved search is a query stored and re-run on a schedule, so a question asked once keeps being answered. It is how discovery becomes monitoring: a search for files over a size threshold in a project directory becomes a standing alert rather than an occasional audit.
How current is a discovery index?
An index is current as of its last crawl, so the crawl schedule is the real answer. Incremental crawls that read only what changed make a daily or faster cadence practical on large estates. Any figure taken from an index should be read with its crawl date attached.
Terms in this category.
Data discovery >
Data discovery is the work of building an accurate picture of what an organization holds, across every system. For unstructured data, it works from file attributes rather than contents, which is what lets it cover a petabyte data estate at the same speed as a small one.
Duplicate file detection >
Duplicate file detection identifies identical files across storage systems. Size and name narrow the search cheaply, and a content hash, which computes a fingerprint from the file itself, is what proves two files are the same. Working across vendors matters, because a file copied from a NAS share to an object store is invisible to either system alone.
File analytics >
File analytics is the analysis of file metadata to answer questions about a storage estate: how much capacity each project consumes, what share has not been read in three years, where duplicates sit, and which owners account for the most growth.
Orphaned data >
Orphaned data is data whose owner account no longer exists, usually because the person left. It keeps consuming capacity, and it has no one to ask about it, which is why it accumulates quietly and rarely appears in a capacity report.
File system observability >
File system observability is continuous visibility into growth, access, and quiet areas across file systems. It is the operational half of discovery, run on a schedule rather than as a one-time assessment, so a change shows up as it happens rather than at the next audit.
Abandoned data >
Abandoned data is data with a valid owner and no remaining purpose, such as debug output, render intermediates, and core dumps that a workflow produced and never cleared. It differs from orphaned data in that someone can still be asked about it.
Data classification >
Data classification is the grouping of files by what they are, from metadata signals rather than content: name, extension, path, location, age, owner, permissions, and structural detail. It groups files so a rule can be written once and applied to the group.
Last access date >
Last access date, often referred to as atime, is the file system attribute recording when a file was last read. It is the strongest single signal for cold data, with two cautions: some systems update it on backup and antivirus reads, and some mount options switch it off entirely.
Sensitive data discovery >
Sensitive data discovery is the work of locating files likely to hold regulated or confidential material. A metadata-only approach surfaces candidates from name, path, format, and location and leaves the judgment to a person, which is what makes it fast enough to run across a whole estate.
Saved search >
A saved search is a stored query re-run on a schedule, turning discovery into monitoring. A question asked once keeps being answered, so a search for oversized files in a project directory becomes a standing alert rather than an occasional audit.
Data footprint >
A data footprint is the total capacity an organization’s data occupies across every storage system and tier, copies included. Broken down by project, owner, and age, it shows which share is active and which share could move or be removed.
Diskover for data discovery.
Discovery is not a report you run once. Here is where to start.
One search across the whole estate.
Search by any attribute the index holds, across every vendor, tier, and location at once. The data stays in place while the catalog is built, and it moves only when a workflow calls for it.
Likely-sensitive data from metadata signals.
Diskover surfaces likely-sensitive files from name, path, format, and location, without reading file contents. Rules applied with AutoTag turn a one-time search into a standing tag. The policy decision stays with a person.
Counts you can act on.
Ask how much is duplicated or stale, and the answer names the specific files behind the number, ready to select and act on from the result.
Discovery that keeps running.
Saved searches re-run on a schedule, so the picture stays current as the estate changes. The same search can then drive a policy that acts on what it finds.
Related categories.
SOURCES
- Veritas, Global Databerg Report, March 2016: 52% of all information stored and processed by organizations worldwide is dark data, whose value is unknown.
- Ponemon Institute, The State of Cyber Resilience, 2025, sponsored by Pure Storage: Organizations report an average of 36% of stored data as dark or unclassified.
- Diskover deployments, 2026: 30 to 50% of a data estate proves reclaimable once analyzed.
Method note. The reclaimable range is Diskover’s own, measured across customer deployments, and it varies with how long an organization has been accumulating data. The two dark data figures measure different things: the 2016 Veritas number is an analyst estimate, and the 2025 Ponemon number is what organizations report about themselves in a study sponsored by a storage vendor. That is why they differ, and why both are quoted with their year.
Last updated: October 30, 2026.
See it on your own data.
We’ll index a sample of your environment and show you
what’s recoverable against your current storage spend.