Glossary›Storage Systems, Protocols, and Scale›Unstructured data storage

Unstructured data storage

What storage architectures support unstructured data at scale?

Three architectures carry most unstructured data:

• Scale-out network-attached storage serves files to many users over NFS and SMB.
• Object storage holds very large numbers of objects behind an HTTP interface, on premises or in the cloud.
• Parallel file systems serve high-performance work, such as simulation and AI training, at high aggregate speed.

Most large organizations run several of them, plus archive tiers, which is why one index across all of them matters.

What is the difference between block, file, and object storage?

Each one organizes data in a different way:

• Block storage splits data into fixed-size blocks that an operating system or database manages directly, which suits databases and virtual machines.
• File storage organizes data as files in folders, reached over NFS or SMB, which suits shared work.
• Object storage keeps data as objects with a key and metadata in a flat namespace, reached over an HTTP interface such as S3, which suits very large volumes.

Unstructured data mostly lives in file and object storage.

What is a parallel file system?

A parallel file system spreads a single file system across many servers and storage devices so that many clients can read and write at once at high aggregate speed. Lustre, IBM Storage Scale, and BeeGFS are common examples. They are built for throughput in high-performance computing and AI training, and they hold very large file counts.

What is the storage hierarchy?

The storage hierarchy is the ordering of storage from fastest and most costly per terabyte to slowest and least costly: memory, then solid-state drives, then hard disk, then tape and cloud archive tiers. Each step down trades speed for price. Data management is largely the work of keeping each file at the right step, which depends on how often it is read.

What is edge data?

Edge data is data created and stored outside the main data center or cloud, close to where it is produced: on film sets, in factories, in vehicles, at remote labs and field sites, and in branch offices. Cameras, sensors, instruments, and local servers write it first, and much of it later travels to a central site or the cloud, or never does.

Why is edge data hard to manage?

Because it is spread across many small sites that each work a little differently:

• Few or no IT staff on site to organize or clean up data.
• Limited network bandwidth, so sending everything to a central site is slow or costly.
• Copies everywhere, as data is copied to drives, servers, and the cloud on its way to the center.
• No central view, so nobody can say what each site holds until it is shipped back.

How do you manage data at the edge?

Start by seeing it where it is. A scanner at each site indexes local storage and sends only the metadata, which is small, to a central catalog, so the whole organization can search every site from one place. Rules then decide what moves to the core or the cloud and when, such as finished projects or data a central team needs, and local copies are removed only once the data is safe elsewhere.

What is a global namespace?

A global namespace presents storage from several systems or locations as one directory tree, so users see one path regardless of where a file physically sits. It simplifies access. It does not by itself describe the files, so a namespace and a metadata catalog answer different questions.

How do you choose storage for unstructured data?

Match each kind of data to how it is used:

• Active, shared work such as editing and research suits fast file storage.
• Large-scale processing such as simulation and AI training often needs a parallel file system.
• Large data that is read rarely, such as archives and backups, suits object storage or the cloud.
• Records under strict rules may need immutable storage.

Most organizations use several of these, often split between on premises and the cloud, which is called hybrid cloud storage. That is why an inventory across all of them matters.

← Back to Storage Systems, Protocols, and Scale

Last updated: November 13, 2026.