What is unstructured data?

Home | Resources | Glossary | Unstructured Data Basics

KEY TAKEAWAYS

What unstructured data is.

Unstructured data is the material held on network-attached storage, object storage, and parallel file systems, rather than in database rows. These four questions set the boundaries of the term, including where unstructured data storage differs from a database.

Structured, semi-structured, and unstructured data compared:

	STRUCTURED DATA	SEMI-STRUCTURED DATA	UNSTRUCTURED DATA
How it is held	Rows and columns in a defined schema	Records that carry their own internal markers	Files in a file system or object store
What describes it	The schema itself	Markers inside the data, such as JSON keys or XML tags	File metadata kept by the storage system
Common examples	Database tables	JSON and XML files	Documents, images, video, and instrument output
How a value is found	A query against a named field	A program that reads the markers	A search of file metadata in a catalog

What is the difference between structured and unstructured data?

Structured data fits into rows and columns, like a database table of customer records or a spreadsheet of transactions. Every field has a defined type and a defined place, which is what makes it straightforward to query. Unstructured data has no such arrangement. Video files, images, research output, design files, logs, and documents all carry meaning, but that meaning lives inside the file rather than in a schema. Most organizations hold far more of the second kind than the first, and it is the kind that grows fastest. The practical consequence is that a database query can answer a question about structured data, while answering the same question about unstructured data means reading file attributes from every storage system into one catalog first.

What is the difference between structured, semi-structured, and unstructured data?

The three terms describe how much structure travels with the data. Structured data lives in a defined schema, usually database rows and columns. Semi-structured data carries its own internal markers, as JSON and XML do. Unstructured data is held as files, where the storage system records the path, size, owner, and dates, and anything about the contents has to be described separately. That last difference is the one that drives everything else on this page.

What are examples of unstructured data?

Unstructured data is any file whose contents do not sit in rows and columns. In media and entertainment, that means camera footage, render output, and project files. In life sciences, it means microscope images, sequencing output, and instrument logs. In semiconductor design, it means simulation results and verification logs. Across every industry, it also means the ordinary working files: documents, presentations, spreadsheets, email attachments, PDFs, design files, photographs, audio recordings, backups, and virtual machine images. A single production or research project can generate millions of them, which is why a data estate is usually counted in files rather than in records.

What counts as unstructured data in practice?

Office documents and PDFs, images and video, audio, email archives, CAD and EDA design files, simulation and render output, log and telemetry files, backups, virtual machine images, and scientific formats such as BAM, FASTQ, DICOM, and MXF. If it arrives as a file on an NFS or SMB share or as an object in S3 or Azure Blob, it counts.

Unstructured data volume and growth.

Three questions about volume: what share of an organization’s data is unstructured, how fast it is growing, and why that share keeps rising. Our blog post on what actually breaks at a billion files covers what that growth does to everyday storage work.

How much of an organization’s data is unstructured?

Around 90% of the data organizations generate is unstructured, according to IDC research published in 2023. Some estates are almost entirely unstructured, because instruments and render farms produce files continuously. That pattern shows up in media and entertainment, life sciences and healthcare, semiconductor design, and energy.

Why does unstructured data grow faster than structured data?

Database growth is bounded by what applications write. File growth is bounded by what instruments, cameras, sensors, simulations, and people produce, and none of those has a natural stopping point. Higher resolution, larger models, and more sensors all multiply file size and file count at the same time.

How fast is unstructured data growing?

IDC predicted that organizations worldwide would generate more than 73,000 exabytes of unstructured data in 2023 alone, in a white paper sponsored by Box and published in August 2023. That volume rises every year with higher-resolution instruments, larger models, and more sensors. For any one organization, the useful measure is its own growth rate by project and team, which a metadata catalog reports from file sizes and dates.

What is hiding inside unstructured data.

Not all of what an organization stores is known, and not all of it is worth keeping. These three terms name the parts that are neither, and each one is where storage cost optimization begins.

What is dark data?

Dark data is data an organization holds without knowing what it contains or what it is worth. It is a visibility problem rather than a storage problem: the files are present and perfectly readable, and what they are for was never written down. That is why dark data cannot be reduced by buying capacity. It shrinks only as the data estate gets described.

What is ROT data?

ROT stands for redundant, obsolete, and trivial. Redundant is duplicated, obsolete has passed its useful life, and trivial never had value. It is the part of an estate that can be removed with the least argument, which makes it the first capacity most teams reclaim.

What is the difference between hot, warm, and cold data?

The three terms describe how often data is read. Hot data is in active use and belongs on the fastest tier. Warm data is read occasionally. Cold data has not been touched for a long period. The dividing lines are set by each organization, and the last access date is what measures them.

Why unstructured data is hard to manage.

A database describes itself, and a file system does not. These five questions cover what a file system records, what it leaves blank, why files end up in silos, and why metadata management exists to fill the gap.

Why is unstructured data harder to manage than database data?

A database describes itself. Every row sits in a named column with a defined type, so a rule can be written straight against the schema. Files carry no schema of their own, so a rule has to be written against attributes assembled from outside the file. That is a solvable problem, and it is why a file estate needs a catalog where a database needs only a query.

What is file metadata?

File metadata is the description a storage system keeps about a file rather than the file’s content. It includes the path, name, extension, size, owner and group, permissions, the created, modified, and last accessed dates, and identifiers the file system uses internally, such as the inode number that tracks a file across renames. This is the raw material every catalog, tiering rule, and retention decision is built on.

What is the difference between file storage and object storage?

File storage organizes data in a directory tree and is reached over NFS or SMB. Object storage holds flat objects with a key, reached over an HTTP API such as S3. Object storage lets you attach custom metadata to each object, which file systems generally do not, so some description can travel with the object itself. For a file on a NAS share, that description has to live somewhere else, which is what a catalog is for.

Who owns unstructured data inside an organization?

Ownership is usually recorded as the account that created the file, which drifts as people change roles and leave. That is why departed-employee data accumulates. Useful ownership has to be derived from the path, the project, the share, or an enrichment rule, rather than read from the file system. Until that is done, every question about who should decide on a file ends in a guess.

What is a data silo?

A data silo is a store of data that only one team, application, or storage system can see. For files, silos form when each department buys its own storage and each vendor reports only on its own platform. The data is still there, but no one can answer a question across the whole data estate. A single index of file metadata across every system ends the silo for reporting, while the files stay where they are.