What is unstructured data?

Home | Resources | Glossary | Unstructured Data Basics

KEY TAKEAWAYS

What unstructured data is.

Unstructured data is the material held on network-attached storage, object storage, and parallel file systems, rather than in database rows. These five questions set the boundaries of the term, including where unstructured data storage differs from a database.

What is unstructured data?

Unstructured data is information stored as files rather than as database records. Documents, images, video, audio, log files, design files, and instrument output all qualify. The defining feature is that no database schema describes the content, so the file system holds everything known about it.

What is the difference between structured and unstructured data?

Structured data fits into rows and columns, like a database table of customer records or a spreadsheet of transactions. Every field has a defined type and a defined place, which is what makes it straightforward to query. Unstructured data has no such arrangement. Video files, images, research output, design files, logs, and documents all carry meaning, but that meaning lives inside the file rather than in a schema. Most organizations hold far more of the second kind than the first, and it is the kind that grows fastest. The practical consequence is that a database query can answer a question about structured data, while answering the same question about unstructured data means reading file attributes from every storage system into one catalog first.

What is the difference between structured, semi-structured, and unstructured data?

The three terms describe how much structure travels with the data. Structured data lives in a defined schema, usually database rows and columns. Semi-structured data carries its own internal markers, as JSON and XML do. Unstructured data is held as files, where the storage system records the path, size, owner, and dates, and anything about the contents has to be described separately. That last difference is the one that drives everything else on this page.

What are examples of unstructured data?

Unstructured data is any file whose contents do not sit in rows and columns. In media and entertainment, that means camera footage, render output, and project files. In life sciences, it means microscope images, sequencing output, and instrument logs. In semiconductor design, it means simulation results and verification logs. Across every industry, it also means the ordinary working files: documents, presentations, spreadsheets, email attachments, PDFs, design files, photographs, audio recordings, backups, and virtual machine images. A single production or research project can generate millions of them, which is why a data estate is usually counted in files rather than in records.

What counts as unstructured data in practice?

Office documents and PDFs, images and video, audio, email archives, CAD and EDA design files, simulation and render output, log and telemetry files, backups, virtual machine images, and scientific formats such as BAM, FASTQ, DICOM, and MXF. If it arrives as a file on an NFS or SMB share or as an object in S3 or Azure Blob, it counts.

Unstructured data volume and growth.

Two questions about volume: what share of an organization’s data is unstructured, and why that share keeps rising.

How much of an organization’s data is unstructured?

Around 90% of the data organizations generate is unstructured, according to IDC research published in 2023. Some estates are almost entirely unstructured, because instruments and render farms produce files continuously. That pattern shows up in media and entertainment, life sciences and healthcare, semiconductor design, and energy.

Why does unstructured data grow faster than structured data?

Database growth is bounded by what applications write. File growth is bounded by what instruments, cameras, sensors, simulations, and people produce, and none of those has a natural stopping point. Higher resolution, larger models, and more sensors all multiply file size and file count at the same time.

What is hiding inside unstructured data.

Not all of what an organization stores is known, and not all of it is worth keeping. These three terms name the parts that are neither, and each one is where storage cost optimization begins. The storage bill is the visible cost of keeping all three, and our blog post on the storage you are paying for but no longer need works through what that looks like in a real data estate.

What is dark data?

Dark data is data an organization holds without knowing what it contains or what it is worth. It is a visibility problem rather than a storage problem: the files are present and perfectly readable, and what they are for was never written down. That is why dark data cannot be reduced by buying capacity. It shrinks only as the data estate gets described.

What is ROT data?

ROT stands for redundant, obsolete, and trivial. Redundant is duplicated, obsolete has passed its useful life, and trivial never had value. It is the part of an estate that can be removed with the least argument, which makes it the first capacity most teams reclaim.

What is the difference between hot, warm, and cold data?

The three terms describe how often data is read. Hot data is in active use and belongs on the fastest tier. Warm data is read occasionally. Cold data has not been touched for a long period. The dividing lines are set by each organization, and the last access date is what measures them.

Why unstructured data is hard to manage.

A database describes itself, and a file system does not. These four questions cover what a file system records, what it leaves blank, and why metadata management exists to fill the gap.

Why is unstructured data harder to manage than database data?

A database describes itself. Every row sits in a named column with a defined type, so a rule can be written straight against the schema. Files carry no schema of their own, so a rule has to be written against attributes assembled from outside the file. That is a solvable problem, and it is why a file estate needs a catalog where a database needs only a query.

What is file metadata?

File metadata is the description a storage system keeps about a file rather than the file’s content. It includes the path, name, extension, size, owner and group, permissions, the created, modified, and last accessed dates, and identifiers the file system uses internally, such as the inode number that tracks a file across renames. This is the raw material every catalog, tiering rule, and retention decision is built on.

What is the difference between file storage and object storage?

File storage organizes data in a directory tree and is reached over NFS or SMB. Object storage holds flat objects with a key, reached over an HTTP API such as S3. Object storage lets you attach custom metadata to each object, which file systems generally do not, so some description can travel with the object itself. For a file on a NAS share, that description has to live somewhere else, which is what a catalog is for.

Who owns unstructured data inside an organization?

Ownership is usually recorded as the account that created the file, which drifts as people change roles and leave. That is why departed-employee data accumulates. Useful ownership has to be derived from the path, the project, the share, or an enrichment rule, rather than read from the file system. Until that is done, every question about who should decide on a file ends in a guess.