Glossary›Metadata and Cataloging›Metadata indexing

Metadata indexing

Why does metadata indexing matter?

Because every later step depends on it. Search, reports, tiering, retention, and AI data selection all work from the index, so a file that was never indexed is invisible to all of them. Indexing is also the only practical way to see across many storage systems at once, because it turns billions of scattered file records into one place to ask questions.

What is an example of metadata indexing?

A media company points an indexer at its file servers and cloud buckets overnight. By morning, every file has a record with its path, size, owner, type, and dates, and the team can answer questions that used to take weeks, such as how much of the archive nobody has opened since a show ended, or which projects hold the most data on fast storage.

What are examples of file metadata fields?

An index usually records three kinds of fields for every file:

• File system fields, such as path, name, extension, size, owner, permissions, and the created, modified, and last accessed dates.
• Storage fields, such as the storage system, volume, or bucket a file sits on, and its storage tier or class.
• Added fields, such as header details from enrichment, or a project, department, and retention class assigned by rules and tags.

The first two come with every scan. The third is what turns a list of files into answers about the business.

How does a crawler read a file system without slowing it down?

By reading metadata rather than content, running many workers in parallel across separate directory branches, preferring incremental crawls that read only what changed, and pacing requests so storage keeps serving its users. Content reads are what cost, which is why a metadata-only approach can cover petabytes.

Why does a crawler need Active Directory or LDAP?

File systems record owners as identifiers, such as user and group IDs, rather than as names. Active Directory, Microsoft’s directory service, and the Lightweight Directory Access Protocol (LDAP) map those identifiers to people and groups. A crawler looks the identifiers up so the catalog can show owners by name and by department, which is what turns an owner field into something a person can act on.

How current is a discovery index?

An index is current as of its last crawl, so the crawl schedule is the real answer. Incremental crawls that read only what changed make a daily or faster cadence practical on large estates. Any figure taken from an index should be read with its crawl date attached.

What matters more for metadata indexing: the number of files or the amount of storage?

The number of files. Indexing writes one record for every file and folder, and reads each one’s metadata separately, so time, hardware, and index size all grow with the count, not with capacity. Diskover’s sizing guide gives an example: 500 TB of large video files can need a smaller system than 50 TB holding millions of small documents.

What is the difference between a crawler and a scanner?

In practice, very little. Both terms describe the component that reads a storage system and records the metadata of each file, and many tools use them as synonyms. “Crawler” stresses the method: walking a directory tree, folder by folder. “Scanner” is the broader term, and also covers storage that is not read as a tree, such as object storage listed through an API. Diskover’s documentation uses “scanner.”

What is the difference between a full scan and an incremental scan?

A full scan reads the metadata of every file, every time. An incremental scan reads only the folders that changed since the last scan, usually found by comparing folder modification dates, and reuses the records for everything else. The first scan of a system is always full. After that, incremental scans keep the index current in a fraction of the time: Diskover reindexes at 500,000 to 600,000 files per second by skipping folders that have not changed.

What is a cached scan?

A cached scan is one way to run an incremental scan. The scanner keeps a local copy, or cache, of each folder’s listing and file details from the previous scan. On the next scan, it compares each folder’s modification date with the cached one: unchanged folders are served from the cache, and only changed folders are read again from storage. That makes repeat scans much faster and takes load off busy or network storage. The first scan fills the cache, so it runs at normal speed. Diskover offers this through its directory caching scanner.

Does metadata indexing need AI?

No. Indexing reads attributes the file system already holds, such as name, size, owner, and dates, which is fast, exact, and repeatable without AI. Running an AI model on every file would be far slower and more costly, and it could give different answers each time. AI is most useful after indexing: it helps people ask the index questions in plain language and suggests tags for a person to approve. The index gives AI the facts, and AI gives people an easier way to reach them.

← Back to Metadata and Cataloging

Last updated: November 13, 2026.