Duplicate file detection

Duplicate file detection finds identical copies of the same file across storage systems, so teams can decide which copy to keep. It works across vendors, where each system alone sees only its own copies.

Key takeaways

A copy moved from a NAS share to an object store is invisible to each system alone.
Size and name narrow the search, and a content hash confirms a true match.
Deduplication saves space inside one system, while duplicate detection finds copies across systems.

Why do duplicate files matter?

Because every copy multiplies cost and risk. Each duplicate uses primary storage, is copied into every backup, and is carried through every migration. Copies of sensitive files are harder to protect, since each one has its own access settings. Duplicates also cause confusion when people edit different copies of the same document.

How do you find duplicate files across storage systems?

Comparison can start with the least costly signals, such as identical size and name. Confirmation uses a content hash, which computes a fingerprint from the file itself and is the only method that proves two files are identical. Working across vendors matters here, because the same file copied between a NAS share and an object store is invisible to either system alone.

What is the difference between matching duplicates by name and size and matching by checksum?

Matching by name, size, and dates uses metadata, so it is fast and works across billions of files, but it only finds likely duplicates. Two different files can share a size, and a renamed copy of the same file is missed. A checksum, also called a hash, is a fingerprint calculated from the full contents of a file. Two files with the same checksum are identical, whatever their names, so it is the method that proves a duplicate. It takes longer, because every byte has to be read. The usual approach works in stages: matching sizes narrows the list to likely duplicates, a quick checksum of the first part of each file removes more, and a full checksum confirms only the files that are left. On large estates, that means only a small share of files ever needs to be read in full.

What is the difference between data deduplication and duplicate file detection?

Data deduplication is a storage system feature that stores repeated blocks of data once, inside one system, without changing what users see. Duplicate file detection finds whole files that exist more than once, often on different storage systems, so a person can decide which copies to keep. Deduplication saves space inside one system automatically, and duplicate file detection shows the copies that sit across systems, where no single system can see them.

Is it safe to delete duplicate files?

Usually, after a short check. Confirm which copy is the original, keep the one in the right location, and check that no application opens the copy by its path. Copies kept on purpose, such as backups or replicas for another site, are not waste and should be left out of any cleanup.

Why is it hard to decide which duplicate file to delete?

Because identical files are not always equal copies:

The original is not obvious. Copying a file can change its dates, so the oldest-looking copy is not always the first.
Copies may be in use. An application, a script, or a shortcut may open one copy by its path.
Owners differ. Each copy may belong to a different team, and each team wants to keep its own.
Some copies are on purpose. Backups, replicas for another site, and data under legal hold are meant to exist more than once.

How do you choose which duplicate file to keep?

With a simple rule agreed in advance, so nobody has to decide file by file. A common rule keeps the copy in the official location, such as the project or archive folder, owned by the project rather than a person, and removes copies in home folders, scratch areas, and downloads. Before removal, check for legal holds and for applications that use the copy’s path, and tell the owners which copies will go and where the kept copy lives.