GLOSSARY
What is unstructured data storage?
The systems the files actually sit on: NAS and object storage, NFS and SMB, parallel file systems, and what changes when a single estate passes a billion files.
Home | Resources | Glossary | Storage Systems, Protocols, and Scale
DEFINITION
Unstructured data storage is the set of systems that hold files rather than database records: network-attached storage reached over NFS and SMB, object storage reached over an HTTP API such as S3, parallel file systems built for high throughput, and archive tiers including tape. Each records a different set of attributes about the files it holds, which is why a description that spans them has to be built above them.
KEY TAKEAWAYS
1.4B
files indexed by Diskover in 2 hours on 3 nodes, ~200,000 files per second
10B+
files held by one semiconductor customer, with one inventory serving over 4,000 engineers
435+
Diskover platform features, including plugins, connectors, and API endpoints
QUESTIONS THIS PAGE ANSWERS
What is unstructured data storage?
Unstructured data storage is any system holding files rather than database records: network-attached storage (NAS), object storage, parallel file systems, and archive tiers. What they share is that the storage system records where a file sits and how large it is. Knowing what the file is for takes a separate record: its owner, its project, and timestamps.
What is network-attached storage?
Network-attached storage, usually shortened to NAS, is storage presented to many clients over a network as a shared file system, accessed over NFS or SMB. It organizes data in a directory tree, enforces permissions, and records the standard file attributes: path, size, owner, permissions, and dates.
What is object storage, and how does its metadata differ?
Object storage holds flat objects identified by a key, reached over an HTTP API such as S3 or Azure Blob. The important difference for management is that objects can carry custom metadata as key-value pairs, which file systems generally cannot, so some description can travel with the object itself.
What is the difference between NFS and SMB?
NFS came from the Unix world, and SMB from the Windows world, and both present a remote file system over a network. They differ in their treatment of permissions, locking, and identity mapping. For anything that crawls a file system, the practical differences are speed and how each one reports the last access date.
What is a parallel file system?
A parallel file system spreads a single file system across many servers and storage devices so that many clients can read and write at once at high aggregate speed. Lustre, IBM Storage Scale, and BeeGFS are common examples. They are built for throughput in high-performance computing and AI training, and they hold very large file counts.
What is a global namespace?
A global namespace presents storage from several systems or locations as one directory tree, so users see one path regardless of where a file physically sits. It simplifies access. It does not by itself describe the files, so a namespace and a metadata catalog answer different questions.
What is a petabyte, in practical terms?
A petabyte is one thousand terabytes. The useful way to think about it is file count rather than capacity: a petabyte of video might hold a few hundred thousand files, while a petabyte of simulation or sequencing output can hold hundreds of millions. The second one is much harder to manage, at the same capacity.
What is the small file problem?
The small file problem is that the cost of managing files scales with their number rather than their size. A billion small files consume metadata operations, index space, and crawl time out of all proportion to the capacity they occupy, which is why two data estates of the same size can be days apart in how long they take to describe.
What is an inode, and why does it run out?
An inode is the record a file system keeps for each file, holding its attributes and the location of its data. Many file systems allocate a fixed number of inodes when formatted, so an estate with many small files can exhaust them while capacity remains. The symptom is a disk that reports space and refuses writes.
How does a crawler read a file system without slowing it down?
By reading metadata rather than content, running many workers in parallel across separate directory branches, preferring incremental crawls that read only what changed, and pacing requests so storage keeps serving its users. Content reads are what cost, which is why a metadata-only approach can cover petabytes.
What does S3 expose as metadata, and what does it leave out?
S3 exposes the key, size, last modified time, storage class, entity tag, and any user metadata set at upload. It does not record a last access time in the way a file system does, so access patterns have to come from access logging or from an inventory report rather than the object itself.
What is POSIX, and why does it matter for data management?
POSIX is the standard that defines how programs interact with a file system, including the attributes a file carries and how operations behave. It matters because it lets a crawler read storage from many vendors the same way, and because departures from it make cross-vendor behavior unpredictable.
TERMS IN THIS CATEGORY
Unstructured data storage >
Unstructured data storage is any system holding files rather than database records: network-attached storage, object storage, parallel file systems, and archive tiers including tape. Each records a different set of attributes about what it holds, which is why a description spanning all of them has to be built above them.
Parallel file system >
A parallel file system is a file system spread across many servers and storage devices so that many clients can read and write at once at high aggregate speed. Lustre, IBM Storage Scale, and BeeGFS are common examples, used in high-performance computing and AI training, and they hold very large file counts.
Network attached storage >
Network attached storage, usually shortened to NAS, is shared file storage reached over a network by NFS or SMB. It organizes data in a directory tree, enforces permissions, and records the standard file attributes: path, size, owner, permissions, and dates.
Global namespace >
A global namespace presents storage from several systems or locations as one directory tree, so users see one path wherever a file physically sits. It simplifies access and does not describe the files, which is why a namespace and a catalog answer different questions.
Object storage >
Object storage holds flat objects identified by a key, accessed over an HTTP API such as S3 or Azure Blob. Objects can carry custom metadata as key and value pairs, which file systems generally cannot, so some description travels with the object itself.
Petabyte >
A petabyte is one thousand terabytes. The useful way to think about one is file count rather than capacity: a petabyte of video might hold a few hundred thousand files, and a petabyte of sequencing output can hold hundreds of millions. The second is far harder to manage at the same capacity.
NFS >
NFS stands for Network File System and is a protocol for reaching a remote file system over a network, originating in the Unix world. It is one of the two ways most network-attached storage is accessed, the other being SMB.
Inode >
An inode is the record a file system keeps for each file, holding its attributes and the location of its data. Many file systems allocate a fixed number when formatted, so an estate of many small files can exhaust them while capacity remains: the symptom is a disk that reports free space and refuses writes.
SMB >
SMB stands for Server Message Block and is a network file sharing protocol originating in the Windows world. It differs from NFS in how it handles permissions, locking, and identity mapping, which matters to anything crawling a file system.
POSIX >
POSIX stands for Portable Operating System Interface (the X on the end was added because it sounded better). It is the standard defining how programs interact with a file system, including which attributes a file carries and how operations behave. It matters because it lets one crawler read storage from many vendors the same way.
Diskover for storage estates.
Data estates run on many vendors at once. Here is where to start.
One index across every vendor.
PowerScale, ONTAP, S3, Azure Blob, and more, read into one searchable index while the data stays in place.
Format-aware enrichment at index time.
37 enrichment plugins read structural details without reading contents, so each format gives up the attributes it holds.
Built for file count, not just capacity.
File count is what makes an estate hard to manage. Diskover has indexed 1.4 billion files in 2 hours on three nodes, about 200,000 files per second, and one semiconductor customer holds more than 10 billion files in a single inventory.
Standard interfaces throughout.
10 or more storage connectors and 40 or more API endpoints, so the index is reachable by whatever already runs in the environment, from a scheduler to a business intelligence tool.
RELATED CATEGORIES
SOURCES
- Diskover product measurement, 2026: 1.4 billion files indexed in two hours on three nodes, about 200,000 files per second.
- Diskover customer deployment, semiconductor industry: more than 10 billion files in one inventory serving over 4,000 engineers. Reported by industry rather than by name.
- Diskover platform, 2026: 435+ platform features, including more than 10 storage connectors and more than 40 API endpoints.
Method note. The three figures on this page are Diskover’s own rather than third-party research. The indexing figure is a product measurement on a three-node deployment and varies with file count and storage response time rather than with capacity. The customer figures describe one semiconductor deployment, reported by industry rather than by name, and describe that estate rather than a typical result. The feature count is self-reported and changes with each release, so it needs confirming at each review.
Last updated: September 24, 2026.