Diskover architecture
How Diskover indexes and moves data at scale.
DataDiscovery reads metadata where your data already lives, builds one enriched catalog, and scales out as your estate grows. DataDelivery moves files by policy. This page shows how the parts fit together.
Drawing 01 / How it fits together
From the storage you run to the tools that use the data.
Drawing 02 / The components
Six parts, one catalog.
A / Storages and volumes
Add a storage once.
Enter each NFS server, SMB server, or S3 endpoint once, with its credentials. DataDiscovery finds its exports, shares, and buckets, and scan tasks target those volumes.
B / Scanner
Fast, parallel, and cached.
The scanner reads local, NFS, SMB, S3, and S3-compatible storage. A built-in cache keeps rescans fast, and every file records the storage and volume it came from.
C / Catalog
Rich metadata, ready to search.
Standard metadata, such as path, owner, size, and dates, plus what plugins, tags, and rules add. Insights statistics are computed after every scan.
C2 / Task workers
Work runs close to the data.
Task workers run scans, enrichment, and file actions. Linux workers mount storages and volumes; Windows and macOS workers scan the paths you give them.
D / Access
One interface, open APIs.
A web interface for search, dashboards, and reports, plus REST APIs and the AI Connector for your own tools.
E / Integrations
Open at every edge.
Run file actions such as move, rename, and delete, export curated file lists for analytics and AI, and move data with DataDelivery or tools such as Rclone and Dell SmartSync.
Drawing 03 / Scale
Scale out, one node at a time.
You grow the catalog the way you grow storage: add a node when the file count grows. Each deployment is sized by file count, not terabytes.
1.4B
files indexed in 2 hours on 3 nodes
500K–600K
files per second when reindexing
103B+
files cataloged at one customer, across 4 data centers
260 PB
of media indexed at one customer, across 6 locations
Scale out
Add indexers as you grow.
More indexers scan more systems at the same time, while the catalog stays one.
Stay current
Rescans without the wait.
Scheduled rescans use the built-in cache, so the catalog stays current without slowing your storage.
Drawing 04 / Deployment
Runs where your data runs.
Our team sizes and plans each deployment with you, based on your file count and your sites.
On premises
Install on Linux, such as Rocky Linux or Red Hat Enterprise Linux, with an automated deployment.
In the cloud
Run DataDiscovery in your own cloud account, such as AWS, close to your cloud storage.
Hybrid
Place task workers near each site and cloud, and keep one catalog for the whole estate.
Drawing 05 / DataDelivery
Policies decide. Agents move.
DataDelivery, the data mover in Diskover, has three parts. It runs on its own or next to DataDiscovery, and each product runs its own jobs.
Controller
Holds your policies and jobs. It runs as a cloud service or as a virtual machine you host.
Agents
Run close to your storage and copy, move, or sync files at up to 50 Gbps with a single agent. Several agents can share one job, least-loaded agents first.
Accessors
Secure connection points to each storage system: NFS, SMB, S3, Azure Blob, Google Cloud Storage, Wasabi, and Dell PowerScale.
- 1SelectFiles by name, path, size, or age
- 2Dry runSee the scope and cost first
- 3MoveCopy, move, or sync, at up to 50 Gbps per agent
- 4ProveChecksum verified, recorded in the activity log
Drawing 06 / Security
Metadata in, your data stays put.
DataDiscovery is designed to have no operational impact on your storage. It reads metadata, inside your network.
- Volumes are mounted read-only by default, right before each scan.
- Storage credentials are stored encrypted, passed in memory, and never written to disk.
- Role-based access across the web interface and the APIs.
- Sign-in with local accounts, LDAP and Active Directory, Okta, or Microsoft Entra ID.
- DataDelivery sign-in works with Okta, Microsoft Entra ID, and SAML, and every DataDelivery job is recorded in the activity log.
- Searching is read-only: actions on files need the right role.
Drawing 07 / Open foundation
Open by design, built to last.
Open-source core
Built on open-source technologies proven at enterprise scale, with fast search across billions of files.
No vendor lock-in
The storage you buy next year plugs into the same catalog, on a schedule you set.
Built to connect
Connect your own tools through the APIs, and add plugins for the work your teams do.
Specifications
At a glance.
| Indexing | 1.4B files in 2 hours on 3 nodes |
| Reindexing | 500K to 600K files per second with cached rescans |
| Sources | NFS, SMB, local file systems, Amazon S3, and S3-compatible storage, plus connectors for cloud and archive systems |
| Setup | Guided setup from license to first scan |
| Access | Web interface, REST APIs, AI Connector (optional) |
| Sign-in | Local accounts, LDAP and Active Directory, Okta, Microsoft Entra ID |
| Task workers | Linux, Windows, macOS, from 8 cores and 8 GB of memory |
| Deployment | On premises, cloud, or hybrid |
| Hardware | Standard servers or cloud instances, with no special appliances. Our team sizes each deployment with you. |
| Sizing | By file count, not terabytes |
| DataDelivery storage | NFS, SMB, S3, Azure Blob, Google Cloud Storage, Wasabi, and Dell PowerScale |
| DataDelivery transfers | Copy, move, and sync, on a schedule or on demand, with a dry run first. Up to 50 Gbps with a single agent |
| DataDelivery sign-in | Okta, Microsoft Entra ID, and SAML |
| Streaming | DataDelivery catalogs publish file metadata to Kafka topics |
FAQ
Questions about the architecture.
See it on your own data.
We’ll index a sample of your environment and show you
what’s recoverable against your current storage spend.