GLOSSARY
What is unstructured data?
The starting vocabulary: what unstructured data is, how it differs from database data, and the words for the parts of it that cost the most.
Home | Resources | Glossary | Unstructured Data Basics
DEFINITION
Unstructured data is information stored as files rather than as rows in a database. It includes documents, images, video, audio, sensor output, log files, design files, and instrument output: anything held in a file system or object store where the content does not follow a fixed schema. The name describes the storage, not the content. A DICOM scan and a BAM sequencing file are both highly structured internally, and both count as unstructured data.
KEY TAKEAWAYS
90%
of the data organizations generate is unstructured
52%
of stored data is dark, meaning its value is unknown
up to 50%
is redundant, obsolete, or trivial, and much of that sits on fast, expensive storage
QUESTIONS THIS PAGE ANSWERS
What is unstructured data?
Unstructured data is information stored as files rather than as database records. Documents, images, video, audio, log files, design files, and instrument output all qualify. The defining feature is that no database schema describes the content, so the file system holds everything known about it.
What is the difference between structured and unstructured data?
Structured data fits into rows and columns, like a database table of customer records or a spreadsheet of transactions. Every field has a defined type and a defined place, which is what makes it straightforward to query. Unstructured data has no such arrangement. Video files, images, research output, design files, logs, and documents all carry meaning, but that meaning lives inside the file rather than in a schema. Most organizations hold far more of the second kind than the first, and it is the kind that grows fastest. The practical consequence is that a database query can answer a question about structured data, while answering the same question about unstructured data means reading file attributes from every storage system into one catalog first.
What are examples of unstructured data?
Unstructured data is any file whose contents do not sit in rows and columns. In media and entertainment, that means camera footage, render output, and project files. In life sciences, it means microscope images, sequencing output, and instrument logs. In semiconductor design, it means simulation results and verification logs. Across every industry, it also means the ordinary working files: documents, presentations, spreadsheets, email attachments, PDFs, design files, photographs, audio recordings, backups, and virtual machine images. A single production or research project can generate millions of them, which is why a data estate is usually counted in files rather than in records.
What is the difference between structured, semi-structured, and unstructured data?
Structured data lives in a defined schema, usually database rows and columns. Semi-structured data carries its own internal markers, as JSON and XML do. Unstructured data is held as files, where the storage system records the path, size, owner, and dates, and anything about the contents has to be described separately. That last difference is the one that drives everything else on this page.
How much of an organization’s data is unstructured?
Around 90% of the data organizations generate is unstructured, according to IDC research published in 2023. Some estates are almost entirely unstructured, because instruments and render farms produce files continuously. That pattern shows up in media and entertainment, life sciences and healthcare, semiconductor design, and energy.
What counts as unstructured data in practice?
Office documents and PDFs, images and video, audio, email archives, CAD and EDA design files, simulation and render output, log and telemetry files, backups, virtual machine images, and scientific formats such as BAM, FASTQ, DICOM, and MXF. If it arrives as a file on an NFS or SMB share or as an object in S3 or Azure Blob, it counts.
Why is unstructured data harder to manage than database data?
A database describes itself. Every row sits in a named column with a defined type, so a rule can be written straight against the schema. Files carry no schema of their own, so a rule has to be written against attributes assembled from outside the file. That is a solvable problem, and it is why a file estate needs a catalog where a database needs only a query.
What is file metadata?
File metadata is the description a storage system keeps about a file rather than the file’s content. It includes the path, name, extension, size, owner and group, permissions, the created, modified, and last accessed dates, and identifiers the file system uses internally, such as the inode number that tracks a file across renames. This is the raw material every catalog, tiering rule, and retention decision is built on.
What is the difference between file storage and object storage?
File storage organizes data in a directory tree and is reached over NFS or SMB. Object storage holds flat objects with a key, reached over an HTTP API such as S3. Object storage lets you attach custom metadata to each object, which file systems generally do not, so some description can travel with the object itself. For a file on a NAS share, that description has to live somewhere else, which is what a catalog is for.
What is dark data?
Dark data is data an organization holds without knowing what it contains or what it is worth. It is a visibility problem rather than a storage problem: the files are present and perfectly readable, and what they are for was never written down. That is why dark data cannot be reduced by buying capacity. It shrinks only as the data estate gets described.
What is ROT data?
ROT stands for redundant, obsolete, and trivial. Redundant is duplicated, obsolete has passed its useful life, and trivial never had value. It is the part of an estate that can be removed with the least argument, which makes it the first capacity most teams reclaim.
What is the difference between hot, warm, and cold data?
The three terms describe how often data is read. Hot data is in active use and belongs on the fastest tier. Warm data is read occasionally. Cold data has not been touched for a long period. The dividing lines are set by each organization, and the last access date is what measures them.
Why does unstructured data grow faster than structured data?
Database growth is bounded by what applications write. File growth is bounded by what instruments, cameras, sensors, simulations, and people produce, and none of those has a natural stopping point. Higher resolution, larger models, and more sensors all multiply file size and file count at the same time.
Who owns unstructured data inside an organization?
Ownership is usually recorded as the account that created the file, which drifts as people change roles and leave. That is why departed-employee data accumulates. Useful ownership has to be derived from the path, the project, the share, or an enrichment rule, rather than read from the file system. Until that is done, every question about who should decide on a file ends in a guess.
TERMS IN THIS CATEGORY
Unstructured data >
Unstructured data is information stored as files rather than as database records. Documents, images, video, audio, log files, design files, and instrument output all count. The name describes how the data is stored, not how orderly it is inside.
ROT data >
ROT data is redundant, obsolete, and trivial data: the portion of an estate with no remaining use. Redundant means duplicated, obsolete means past its useful life, and trivial means it never had value. It is usually the first capacity a team reclaims, because it is the easiest to defend removing.
Semi-structured data >
Semi-structured data carries its own internal markers rather than a fixed schema, as JSON and XML do. A program can find the fields inside it without being told the layout in advance, which puts it between database rows and plain files.
Cold data >
Cold data is data that has not been read for a long period, measured by the last access date. Each organization sets its own dividing line, commonly somewhere between six months and three years, and cold data is the usual candidate for a move to less costly storage.
Structured data >
Structured data is information held in a defined schema, usually database rows and columns. Every value sits in a named field with a declared type, which is what lets a query find it without anything being described separately.
Hot data >
Hot data is data in active use, which belongs on the fastest storage tier. It is defined by how often it is read rather than by how new it is, so a five-year-old reference file that everyone opens daily is hot.
File metadata >
File metadata is the description a storage system keeps about a file, separate from its content: path, name, extension, size, owner, permissions, and the created, modified, and last accessed dates. It is what every catalog, tiering rule, and retention decision is built from.
File storage >
File storage is storage organized as a directory tree and reached over NFS or SMB. Files have paths, folders, and permissions, and the system records a fixed set of attributes about each one. It is the most common way unstructured data is held.
Dark data >
Dark data is data an organization holds without knowing its content or value. It is not hidden or inaccessible. It has simply never been described, so no one can say whether it should be kept, moved, or removed.
Data estate >
A data estate is all the data an organization holds, across every system, site, and tier. It is the scope a decision has to cover: a question answered for one storage system has been answered for part of the estate, not for the estate.
Diskover for unstructured data.
This vocabulary describes files you already hold. Here is where to start looking at them.
See every file across every system.
One index across storage vendors, tiers, and locations, built without moving or copying anything. Search the whole estate at once by any attribute the index holds.
Find the cold and the redundant.
Last access date across the whole estate turns an assumption about cold data into a list, with file-level actions attached.
Describe what the file system leaves blank.
Enrichment rules add project, owner, retention class, and format details as files are indexed, so the catalog answers questions the storage system cannot.
Work at petabyte scale.
File count is what makes an estate hard to manage, not capacity. Diskover has indexed 1.4 billion files in 2 hours on three nodes, about 200,000 files per second.
RELATED CATEGORIES
SOURCES
- IDC, 2023: 90% of the data organizations generated in 2022 was unstructured.
- Veritas, Global Databerg Report, March 2016: 52% of all information stored and processed by organizations worldwide is dark data, whose value is unknown.
- Cadence Group, 2025: in large enterprises, redundant, obsolete, or trivial data can account for up to 50% of stored data.
Method note. The three figures on this page come from published third-party research, each named with its publisher and year in the sources above. They measure different things and are not additive. The IDC figure counts data that organizations generated in a year, while the Veritas and Cadence Group figures describe data already stored, so the 52% and the up to 50% overlap with each other rather than dividing the 90%. The Veritas study dates from 2016 and is used here as an order of magnitude, because the condition it measured, files accumulating faster than anyone describes them, has not changed. The Cadence Group figure is a ceiling for large enterprises rather than a typical result, and where an estate falls below it depends on its age and its retention practice. The indexing figure in the Diskover section is Diskover’s own, measured on a three-node deployment, and it varies with hardware, protocol, and file size, so it needs confirming at each review.
Last updated: September 24, 2026.