What is unstructured data storage?

Home | Resources | Glossary | Storage Systems, Protocols, and Scale

KEY TAKEAWAYS

NAS, object, and parallel file systems.

These four systems hold the same thing in different shapes, and reading across all of them at once is what metadata management makes possible.

NAS, object storage, and parallel file systems compared:

	NETWORK-ATTACHED STORAGE (NAS)	OBJECT STORAGE	PARALLEL FILE SYSTEM
How data is organized	A directory tree of files	Flat objects identified by a key	A file system spread across many servers and storage devices
How it is reached	NFS or SMB	An HTTP API, such as S3 or Azure Blob	Many clients reading and writing at once
What it records about data	Standard file attributes, such as path, size, owner, and dates	Custom metadata as key and value pairs	File attributes for very large file counts
Common examples	Shared file storage for teams	S3 and Azure Blob	Lustre, IBM Storage Scale, and BeeGFS
Typical use	Everyday shared files	Large stores reached by applications over HTTP	High-performance computing and AI training

What is network-attached storage?

Network-attached storage, usually shortened to NAS, is storage presented to many clients over a network as a shared file system, accessed over NFS or SMB. It organizes data in a directory tree, enforces permissions, and records the standard file attributes: path, size, owner, permissions, and dates.

What is object storage, and how does its metadata differ?

Object storage holds flat objects identified by a key, reached over an HTTP API such as S3 or Azure Blob. Objects can carry custom metadata as key-value pairs, so some description travels with the object itself. Some scale-out file systems offer the same on files, and many do not, which is why a catalog that reads both keeps one consistent description across file and object storage.

What is a parallel file system?

A parallel file system spreads a single file system across many servers and storage devices so that many clients can read and write at once at high aggregate speed. Lustre, IBM Storage Scale, and BeeGFS are common examples. They are built for throughput in high-performance computing and AI training, and they hold very large file counts.

What is the storage hierarchy?

The storage hierarchy is the ordering of storage from fastest and most costly per terabyte to slowest and least costly: memory, then solid-state drives, then hard disk, then tape and cloud archive tiers. Each step down trades speed for price. Data management is largely the work of keeping each file at the right step, which depends on how often it is read.

NFS, SMB, S3, and POSIX.

A protocol decides what a client can ask of a file system, and what the file system will answer. These three questions cover NFS, SMB, S3, and POSIX, and the limits they set on what data discovery for unstructured data can see.

What is the difference between NFS and SMB?

NFS came from the Unix world, and SMB from the Windows world, and both present a remote file system over a network. They differ in their treatment of permissions, locking, and identity mapping. For anything that crawls a file system, the practical differences are speed and how each one reports the last access date.

What is POSIX, and why does it matter for data management?

POSIX is the standard that defines how programs interact with a file system, including the attributes a file carries and how operations behave. It matters because it lets a crawler read storage from many vendors the same way, and because departures from it make cross-vendor behavior unpredictable.

What does S3 expose as metadata, and what does it leave out?

S3 exposes the key, size, last modified time, storage class, entity tag, and any user metadata set at upload. It does not record a last access time in the way a file system does, so access patterns have to come from access logging or from an inventory report rather than the object itself.

Petabytes, small files, and inodes.

At a billion files, the cost of management scales with file count rather than with capacity. File count is also what sets the ceiling on a storage estate, and what makes storage cost optimization a file-level question rather than a capacity question. These five questions run from petabytes to exabytes, and from small files to how stored data decays.

What is a petabyte, in practical terms?

A petabyte is one thousand terabytes. The useful way to think about it is file count rather than capacity: a petabyte of video might hold a few hundred thousand files, while a petabyte of simulation or sequencing output can hold hundreds of millions. The second one is much harder to manage, at the same capacity.

What is the small file problem?

The small file problem is that the cost of managing files scales with their number rather than their size. A billion small files consume metadata operations, index space, and crawl time out of all proportion to the capacity they occupy, which is why two data estates of the same size can be days apart in how long they take to describe.

What is an inode, and why does it run out?

An inode is the record a file system keeps for each file, holding its attributes and the location of its data. Many file systems allocate a fixed number of inodes when formatted, so an estate with many small files can exhaust them while capacity remains. The symptom is a disk that reports space and refuses writes.

What is an exabyte?

An exabyte is one thousand petabytes, or one million terabytes. At that scale, file count matters even more than capacity, because every management job, from indexing to tiering, works one file at a time. An exabyte of small files can hold hundreds of billions of them, which is why scale is planned in files as well as in bytes.

What is bit rot?

Bit rot is the slow corruption of stored data as storage media degrade, so a file that has not been read for years may no longer read correctly. It is also called data decay. Storage systems guard against it with checksums and scrubbing, which reread data and repair it from redundant copies. A catalog of last access dates shows which files have gone longest without being read, which is a useful starting point for deciding what to verify.

Global namespace and crawling.

Reading a whole estate without slowing it down is an architecture problem. Diskover’s cached scanning is how it is handled here.

What is a global namespace?

A global namespace presents storage from several systems or locations as one directory tree, so users see one path regardless of where a file physically sits. It simplifies access. It does not by itself describe the files, so a namespace and a metadata catalog answer different questions.

How does a crawler read a file system without slowing it down?

By reading metadata rather than content, running many workers in parallel across separate directory branches, preferring incremental crawls that read only what changed, and pacing requests so storage keeps serving its users. Content reads are what cost, which is why a metadata-only approach can cover petabytes.

What is hybrid cloud storage?

Hybrid cloud storage is an estate that combines on-premises storage with one or more public clouds, with data placed in each according to cost, performance, and rules about location. The difficulty is that each side reports only on itself. A global namespace can present both as one directory tree, and a metadata catalog can describe both in one index, which is what lets one question cover the whole estate.

Why does a crawler need Active Directory or LDAP?

File systems record owners as identifiers, such as user and group IDs, rather than as names. Active Directory, Microsoft’s directory service, and the Lightweight Directory Access Protocol (LDAP) map those identifiers to people and groups. A crawler looks the identifiers up so the catalog can show owners by name and by department, which is what turns an owner field into something a person can act on.