GLOSSARY
What is data discovery for unstructured data?
Finding and measuring what is already there: file analytics, duplicate detection, classification from metadata, and the difference between a vendor report and an estate view.
Home | Resources | Glossary | Data Discovery and File Analytics
DEFINITION
Data discovery for unstructured data is the process of finding out what an organization actually holds, across every storage system, without opening the files. It reads file attributes into one index, then answers questions about that index: what exists, how old it is, who owns it, what it duplicates, and what has not been read in years. The output is a list, not an estimate.
KEY TAKEAWAYS
52%
of stored data is dark, meaning its value is unknown
30 to 50%
of an estate turns out to be reclaimable once analyzed
36%
is what organizations themselves estimate is dark or unclassified
QUESTIONS THIS PAGE ANSWERS
What is data discovery for unstructured data?
Data discovery is the process of building an accurate picture of what an organization holds across every storage system. For unstructured data, it works from file attributes rather than content, which lets it cover petabyte estates and restricted environments at the same speed.
What is file analytics?
File analytics is the analysis of file metadata to answer questions about a storage estate. Typical questions are how much capacity each project consumes, what share has not been read in three years, where duplicates sit, and which owners account for the most growth.
How do you find files nobody has opened in years?
Last access date is the mechanism. An index that records last access for every file across every system turns the question into a filter. The practical difficulties are that some storage systems update last access on backup reads, and that mount options can disable the attribute, so both need checking first.
What is file system observability?
File system observability is continuous visibility into what is happening across file systems: what is growing, what is being read, what has gone quiet, and where capacity is heading. It is the operational half of discovery, run on a schedule rather than as a one-time assessment.
How do you find duplicate files across storage systems?
Comparison can start with the least costly signals, such as identical size and name. Confirmation uses a content hash, which computes a fingerprint from the file itself and is the only method that proves two files are identical. Working across vendors matters here, because the same file copied between a NAS share and an object store is invisible to either system alone.
What is orphaned or abandoned data?
Orphaned data is data whose owner account no longer exists, usually because someone left. Abandoned data still has a valid owner but no longer has a purpose, such as debug output and core dumps that a workflow generated and never cleared. Both accumulate quietly, and neither shows up in a capacity report.
How is data classification done without reading file contents?
Classification from metadata uses signals the file system and the index already hold: name, extension, path, location, age, owner, permissions, and structural detail supplied by format-aware plugins. It groups files by what they are and where they sit. It does not detect what a document says, and it is faster and less invasive because of that.
What is sensitive data discovery?
Sensitive data discovery is the work of locating files likely to hold regulated or confidential material. A metadata-only approach surfaces candidates from name, path, format, and location, then leaves the judgment to a person. It answers where to look first. After an incident, that is the question everything else waits on, and an index turns it into a search rather than an investigation.
How do you measure how much of an estate is reclaimable?
Combine three filters: files not read for a defined period, duplicates, and files belonging to closed projects or departed owners. Across Diskover deployments, the reclaimable share runs 30 to 50%, with top performers cutting their footprint by half or more.
What is the difference between storage vendor reporting and cross-vendor discovery?
A storage vendor tool reports accurately on its own platform and is blind to everything else. Cross-vendor discovery reads PowerScale, ONTAP, S3, Azure Blob, and the rest into one index. Most decisions, such as what to tier or what to retire, are estate decisions, so a per-vendor view answers the wrong question.
What is a saved search?
A saved search is a query stored and re-run on a schedule, so a question asked once keeps being answered. It is how discovery becomes monitoring: a search for files over a size threshold in a project directory becomes a standing alert rather than an occasional audit.
How current is a discovery index?
An index is current as of its last crawl, so the crawl schedule is the real answer. Incremental crawls that read only what changed make a daily or faster cadence practical on large estates. Any figure taken from an index should be read with its crawl date attached.
What is the difference between file metadata and a storage audit log?
File metadata records the owner of a file and the dates it was created, modified, and last accessed, so a catalog can show whose file it is and when activity happened. A storage audit log records which person performed each action, which the catalog does not hold. The two answer different halves of the same question, so they are read together when you need to know both what changed and who changed it.
TERMS IN THIS CATEGORY
Data discovery >
Data discovery is the work of building an accurate picture of what an organization holds, across every system. For unstructured data, it works from file attributes rather than contents, which is what lets it cover a petabyte data estate at the same speed as a small one.
Duplicate file detection >
Duplicate file detection identifies identical files across storage systems. Size and name narrow the search cheaply, and a content hash, which computes a fingerprint from the file itself, is what proves two files are the same. Working across vendors matters, because a file copied from a NAS share to an object store is invisible to either system alone.
File analytics >
File analytics is the analysis of file metadata to answer questions about a storage estate: how much capacity each project consumes, what share has not been read in three years, where duplicates sit, and which owners account for the most growth.
Orphaned data >
Orphaned data is data whose owner account no longer exists, usually because the person left. It keeps consuming capacity, and it has no one to ask about it, which is why it accumulates quietly and rarely appears in a capacity report.
File system observability >
File system observability is continuous visibility into growth, access, and quiet areas across file systems. It is the operational half of discovery, run on a schedule rather than as a one-time assessment, so a change shows up as it happens rather than at the next audit.
Abandoned data >
Abandoned data is data with a valid owner and no remaining purpose, such as debug output, render intermediates, and core dumps that a workflow produced and never cleared. It differs from orphaned data in that someone can still be asked about it.
Data classification >
Data classification is the grouping of files by what they are, from metadata signals rather than content: name, extension, path, location, age, owner, permissions, and structural detail. It groups files so a rule can be written once and applied to the group.
Last access date >
Last access date, often referred to as atime, is the file system attribute recording when a file was last read. It is the strongest single signal for cold data, with two cautions: some systems update it on backup and antivirus reads, and some mount options switch it off entirely.
Sensitive data discovery >
Sensitive data discovery is the work of locating files likely to hold regulated or confidential material. A metadata-only approach surfaces candidates from name, path, format, and location and leaves the judgment to a person, which is what makes it fast enough to run across a whole estate.
Saved search >
A saved search is a stored query re-run on a schedule, turning discovery into monitoring. A question asked once keeps being answered, so a search for oversized files in a project directory becomes a standing alert rather than an occasional audit.
Diskover for data discovery.
Discovery is not a report you run once. Here is where to start.
One search across the whole estate.
Search by any attribute the index holds, across every vendor, tier, and location at once. The data stays in place while the catalog is built, and it moves only when a workflow calls for it.
Likely-sensitive data from metadata signals.
Diskover surfaces likely-sensitive files from name, path, format, and location, without reading file contents. Rules applied with AutoTag turn a one-time search into a standing tag. The policy decision stays with a person.
Counts you can act on.
Ask how much is duplicated or stale, and the answer names the specific files behind the number, ready to select and act on from the result.
Discovery that keeps running.
Saved searches re-run on a schedule, so the picture stays current as the estate changes. The same search can then drive a policy that acts on what it finds.
RELATED CATEGORIES
SOURCES
- Veritas, Global Databerg Report, March 2016: 52% of all information stored and processed by organizations worldwide is dark data, whose value is unknown.
- Ponemon Institute, The State of Cyber Resilience, 2025, sponsored by Pure Storage: Organizations report an average of 36% of stored data as dark or unclassified.
- Diskover deployments, 2026: 30 to 50% of a data estate proves reclaimable once analyzed.
Method note. The reclaimable range is Diskover’s own, measured across customer deployments, and it varies with how long an organization has been accumulating data. The two dark data figures measure different things: the 2016 Veritas number is an analyst estimate, and the 2025 Ponemon number is what organizations report about themselves in a study sponsored by a storage vendor. That is why they differ, and why both are quoted with their year.
Last updated: September 24, 2026.