Glossary›Metadata and Cataloging›Metadata catalog

Metadata catalog

Why does an organization need a metadata catalog?

Because without one, nobody can answer simple questions about the data estate, such as how much data a project holds, who owns it, or what has not been opened in years. Each storage system reports only on itself. A metadata catalog puts every system in one place, so cost, governance, and AI decisions are made from facts instead of estimates.

What is a global file index?

A global file index is one searchable record covering every storage system, tier, and location an organization uses. It is the same idea as a metadata catalog, described from the perspective of coverage rather than content: one place to ask a question, instead of one place per vendor.

What is a metadatabase?

A metadatabase is a database whose records describe other data rather than holding it. A metadata catalog for files is a metadatabase in that sense: each record describes one file, such as its path, size, owner, and dates, while the file itself stays on storage. The term is also used in research and libraries for databases that describe datasets or collections.

Does a metadata catalog store copies of files?

No. The catalog stores descriptions, and the files stay exactly where they are. That is what lets one catalog span storage systems which could not otherwise be queried together, and it is why building a catalog does not require a migration.

Can one catalog span several storage vendors?

Yes, and that is the main reason to build one. A storage vendor tool reports accurately on its own platform and is blind to everything else. Read PowerScale, ONTAP, S3, Azure Blob, and the rest into one index, and a single search reaches the whole data estate at once. That is the only way to decide about the estate rather than one system at a time.

What does a metadata catalog not do?

It does not read file contents, so it cannot classify a document by what it says. It does not set retention policy. It supplies the evidence that a policy was applied. And it does not improve data quality, because a catalog of poorly organized data accurately describes poorly organized data.

Does a metadata catalog replace backup?

No. A backup holds copies of files so they can be restored after loss or damage. A metadata catalog holds a description of files, such as path, size, owner, and age, and no copies of the files themselves. The two work together: the catalog shows what exists and where, which helps decide what needs protecting, and the backup holds the protected copies.

How do you build a metadata catalog?

A metadata catalog is built by reading the storage you already have, without moving or opening files. The usual steps:

1. Connect the storage. Point crawlers at each file system, NAS share, and object storage bucket, with read access to metadata.
2. Crawl the metadata. The crawlers record attributes such as path, size, owner, dates, and file type for every file.
3. Resolve owners. Translate user and group IDs into names through Active Directory or LDAP.
4. Add business context. Tag files with attributes such as project, department, or retention class, using rules based on path, type, or owner.
5. Schedule the scans. Set recurring scans so the catalog stays current.

Most organizations start with one storage system and add the rest once the first results are trusted.

Where does a metadata catalog live?

In its own index, separate from the storage it describes. The index usually runs on servers or virtual machines the organization controls, in its own data center or its own cloud account, and the crawlers run close to the storage they read. Because it holds descriptions rather than files, the index is small compared with the data estate, and it can be secured and backed up like any other system.

What technology does a metadata catalog usually run on?

Most catalogs have three parts:

• Scanners that read each storage system and collect the metadata of every file.
• A store for the records, most often a search engine such as Elasticsearch or OpenSearch, built to search and filter billions of records in seconds. Some catalogs also keep their records in open table formats that analytics engines and data warehouses can query directly.
• Interfaces for people and tools, such as a web interface, an API, and an AI connector.

The store matters most at scale: it decides how fast searches return, how much hardware the catalog needs, and which other tools can read the records.

How much space does a metadata catalog take?

Very little compared with the data it describes, because it holds a short record for each file rather than the file itself. Size depends on two things: how many files are indexed, and how many past scans are kept for comparison. Diskover’s sizing guide estimates about 1 GB of index for every 5 million files, for each saved scan. A billion files therefore needs about 200 GB per scan, and keeping six months of weekly scans multiplies that by 24, still a small fraction of the petabytes being described.

How do people access a metadata catalog?

Through a search interface, dashboards, and reports for people, and through an API for other tools. A storage administrator can search for every file over a certain age on one system, a finance team can read storage cost by department, and an application can query the API to select files for an AI pipeline. Access can be set by role, so each team sees what it needs.

Can you add your own information to a metadata catalog?

Yes, and that is where most of its value comes from. System metadata says where a file is and how large it is, but not which project it belongs to or whether it can be deleted. Teams add that knowledge as tags and custom fields, such as project, cost center, retention class, or review status. Tags can be applied by hand, by rules based on path or file type, or from other systems such as an asset management or project tracking tool.

How do you keep a metadata catalog up to date?

With scheduled scans. A catalog is current as of its last scan, so the schedule sets how fresh it is, and incremental scans that read only what changed make daily updates practical even on large estates. Tagging rules should run on the same schedule, so new files are described as soon as they appear, and files that were moved or deleted stop showing up in results.

Can ransomware affect a metadata catalog?

It can, like any other system, but the damage is limited. The catalog holds descriptions of files, not the files themselves, so encrypting it does not touch the data it describes. Because the storage remains the source of truth, a damaged catalog can be rebuilt by scanning again. The catalog servers should still be backed up and protected like any other production system, so teams have it available when they need it most: right after an incident.

What happens if someone gains access to a metadata catalog?

They cannot open files through the metadata alone, but they can learn a lot. File names, paths, owners, and sizes can reveal project names, client names, where sensitive data sits, and who works on what. That is why access to a catalog should be protected like access to the storage it describes: single sign-on, roles and per-index permissions that limit what each user can see, and activity logs. If the catalog can also run actions on files, such as moving or deleting them, those permissions need the same care as storage administrator rights.

← Back to Metadata and Cataloging

Last updated: November 13, 2026.