INTEGRATIONS | PLUGINS
Supercharged tools for limitless innovation.
Diskover’s open-source infrastructure provides unparalleled extensibility, allowing plugins to be developed by Diskover, third parties, or even end users to address specific requirements.
Our comprehensive data management solution features a diverse array of integrated, configurable tools, with many more under development. One standout feature is file action, which enables authorized users to execute file operations—directly from the user interface. This capability streamlines workflows, enhances efficiency, and ensures greater control over data management processes.
PLUGINS FOR METADATA CATALOG ENRICHMENT
Metadata harvest.
Diskover sets the standard in the data management market with its extensive range of metadata harvest plugins, which enrich data with a comprehensive set of attributes.
This business context metadata is an invaluable asset for locating, organizing, and analyzing specific data, as well as for designing workflows tailored to precise data management tasks. To ensure optimal performance, these plugins can be executed post-indexing and at scheduled intervals, preserving the rapid speed of regular indexing while still providing detailed metadata insights.
CORE
Breadcrumb.
Overview:
Designed to extract/add metadata from files’ breadcrumbs to the Elasticsearch index as a post-index process.
Technical documentation:
File kind.
Overview:
Allows users to categorize file types by groups and adds the extra metadata to the Elasticsearch index during the indexing process, useful for reporting purposes.
Technical documentation:
Filesystems contextual metadata harvested from various platforms.
Overview:
Out of the box, Diskover efficiently scans generic filesystems. However, in today’s complex IT architectures, files are often stored across a variety of repositories. To address this, Diskover provides a robust foundation for building alternate indexers, aka scanners, enabling comprehensive indexing of any file storage location.
These repositories often contain relevant additional metadata which can be leveraged by adding it to the index. This enhanced metadata inclusion allows for more precise data management workflows and analysis, with the ultimate goal of reusing and optimizing your data storage solutions.
Learn more:
Technical documentation:
First index time/arrival time.
Overview:
Triggers the creation of an additional attribute when Diskover first detects a new file in a given location and adds the extra metadata to the Elasticsearch index during the indexing process.
Technical documentation:
Hash values/checksums.
Overview:
Adds xxhash, md5, sha1, and sha256 hash values to files in Elasticsearch indices during indexing or as a post-index process to use for checksums/data integrity. Hash values are like fingerprints; they are unique to each file. They are the results of a cryptographic algorithm, which is a mathematical equation with different complexity and security levels used to scramble the plain text and make it unreadable. They are used for data encryption, authentication, and digital signatures.
Technical documentation:
Technical documentation:
Video | Use case example:
Hash values/checksums for S3.
Overview:
Adds md5 and sha1 hash values for files and directories to the Elasticsearch index as a post-index process, using AWS Lambda/Fixity to files in Elasticsearch indices built using S3 alt scanner. Hash values are like fingerprints; they are unique to each file. They are the results of a cryptographic algorithm, which is a mathematical equation with different complexity and security levels, used to scramble the plain text and make it unreadable. They are used for data encryption, authentication, and digital signatures.
Technical documentation:
Video | Use case example:
Image info.
Overview:
Designed to add Image EXIF info metadata from your image files to the Elasticsearch index during the indexing process. Exchangeable Image File Format is a standardized way of storing useful metadata in digital image files. It holds a wealth of technical information about how the image was created, including the time and date it was taken, the camera and lens that was used, and the shooting settings.
Technical documentation:
Path tokens.
Overview:
Designed to break down concatenated directory/file names and add the tokenized metadata to the Elasticsearch index during the indexing process.
Technical documentation:
PDF info.
Overview:
Designed to add several metadata fields from your PDF files to the Elasticsearch index during the indexing process.
Technical documentation:
Unix permissions.
Overview:
Adds the Unix permission attributes of each file and directory to the Elasticsearch data catalog during indexing. Two tags are added, unixperms-plugin and ugo+rwx, if a file or directory is found with fully open permissions (777 or 666).
Technical documentation:
Windows attributes.
Overview:
Adds Windows file owner, primary group, and DACL info to files and directories in Elasticsearch indices.
Technical documentation:
Windows owner.
Overview:
Adds the Windows file owner and primary group of each file and directory to the Elasticsearch index during the indexing process.
Technical documentation:
MEDIA
Autodesk Flow Production Tracking (formerly ShotGrid).
Overview:
The Flow Production Tracking (formerly ShotGrid) Plugin is designed to enhance basic metadata with detailed production status information, aligning data management with production schedules. The Diskover Flow Production Tracking Plugin can harvest over a hundred attributes from the Autodesk Flow Production Tracking platform for every shotdirectory located on storage. These attributes become properties of the shot directories and include status information such as finaled, out-of-picture, multiple project tracking dates, and many more, totaling around one hundred indexable fields.
User documentation:
Media Info.
Overview:
Adds business context and searchability via additional media file attributes (resolution, codec, pixel format, etc.). The enriched metadata is key for granular analysis, workflow automation, and overall data curation.
Technical documentation:
User documentation:
Xytech asset creation.
Overview:
The Xytech Asset Creation Plugin is designed to find and index newly restored online assets from LTO tapes, removable hard drives, etc., making them available, findable, and searchable within the Xytech Media Operations Platform as well as within Diskover. This plugin adds business context following assets rehydration via asset ID attributes. The enriched metadata is key for granular analysis, workflow automation, and overall data curation.
Technical documentation:
User documentation:
Video:
Xytech order status.
Overview:
The Xytech Order Status Plugin is designed to automate the correlation of the order management system and the storage system, by harvesting key business context from Xytech and applying that context within the AJA Diskover Media Edition. In turn, this business context metadata (order phase, invoice date, etc.) can be used to automate workflows, curate data, monitor costs, create highly customized reports, and search granularly.
Learn more:
Technical documentation:
User documentation:
Video:
LIFE SCIENCE
BAM and SAM.
Overview:
The BAM plugin enables additional metadata collection for BAM (Binary Alignment Map) and SAM (Sequence Alignment Map) file formats. These extra fields become searchable, reportable for analysis, and actionable, allowing for potential upstream file management, manually or via automated scheduled tasks.
Learn more:
Technical documentation:
User documentation:
Video:
PLUGINS FOR SUSTAINABLE DATA MANAGEMENT
Data curation.
These plugins are central to achieving sustainable data management, serving as powerful tools that streamline both manual and automated data maintenance.
By addressing a variety of use cases, they enable efficient data clean-up, organization, and ongoing optimization, ensuring that your data environment remains healthy, compliant, and cost-effective. Whether it’s automating routine maintenance tasks or executing complex data management workflows, these plugins are essential for maintaining the integrity and usability of your data over time.
CORE
AutoClean.
Overview:
Designed to move, copy, delete, rename, or run custom commands on files and/or directories based on a set of highly configurable criteria. Any Elasticsearch query (tags, age, size, path, filename, extra metadata collected during post-index, etc.) can be used for the criteria providing very granular actions.
Learn more:
Technical documentation:
AutoTag.
Overview:
Allows for automated tags to be applied/added to the Elasticsearch index as a post-index process. Tags can be applied automatically via a series of rules to directories or files. The criteria can be very powerful based on a combination of base and business-context metadata.
Learn more:
Technical documentation:
Costs.
Overview:
Adds costs per GB for files and directories to the Elasticsearch index as a post-index process. Note that this feature can also be run at index time.
Learn more:
Technical documentation:
Duplicates finder.
Overview:
Precise duplicate files’ checksums across a single or all indices using xxhash, md5, sha1, and sha256 hash values. The plugin is designed for multiple use cases:
- To check for duplicate files across a single or all file systems (single or multiple indices) and index the file docs in index/indices that are dupes.
- To calculate the file checksums/hashes for all duplicate files or all files and index hashes to file docs in an index.
- Reports can be customized to easily locate all duplicate files for clean-up efforts and/or to secure sensitive information.
Calculating file hash checksums is an expensive CPU/disk operation. The dupes finder provides configuration options to control what files in the index get a hash calculated and marked as a dupe. The duplicates plugin harvests hash values that can be stored only for duplicates or for all files.
In addition, the dupes finder provides additional optimization mechanisms as described in the Diskover Configuration and Administration Guide.
Technical documentation:
Elasticsearch field copier.
Overview:
Migrates Elasticsearch field data from one index to another as a post-index process.
Technical documentation:
Elasticsearch query report.
Overview:
Sends email reports and notifications based on Elasticsearch search queries.
Technical documentation:
Grafana.
Overview:
Provides the ability to visualize and trend data metrics over time using Grafana. The plugin rolls up summary data and creates Grafana-specific indices within Elasticsearch. These indices use time series @timestamp metrics to separate logstash- indices, indexes directory size, counts up to N dir depths (default 2). Elasticsearch can then use these summary indexes as a data source for viewing these logstash indices from Grafana.
Technical documentation:
Grafana cloud.
Overview:
Provides the ability to visualize and trend data metrics over time using Grafana Cloud. The plugin rolls up summary data and creates Grafana-specific indices within Elasticsearch. These indices use time series @timestamp metrics to separate logstash- indices, indexes directory size, counts up to N dir depths (default 2). Elasticsearch can then use these summary indexes as a data source for viewing these logstash indices from Grafana.
Technical documentation:
Tag copier.
Overview:
Designed to migrate tags (manual or autotags) from one index to the next during indexing or as a post-index process.
Technical documentation:
Technical documentation:
PLUGINS FOR DATA MOVEMENT
Data movers.
These plugins could easily fall under both data curation and file actions; however, the rapid growth of our plugin library and increasing demand for specialized tools have inspired us to create a dedicated category. This distinction highlights their significance and reaffirms our commitment to meeting the evolving needs of our users in data optimization and organization.
These plugins play a pivotal role in optimizing data placement and timing, ensuring that data is stored where it’s most needed, precisely when it’s needed.
CORE
Ngenea data orchestrator by PixitMedia.
Overview:
With the Ngenea Data Orchestrator File Action, authorized users can quickly and securely transport data, directly from the Diskover UI, to and from globally distributed cloud, object storage, traditional NAS files, and tape resources, automatically moving data into the ‘right cost’ resource according to value and usage as your work teams and business needs demand.
Technical documentation:
Solution brief:
Video:
Qumulo data mover.
Overview:
The Qumulo Data Mover File Action allows authorized users to quickly and securely transport data, directly from the Diskover UI, from Qumulo to Qumulo, Qumulo to AWS, and AWS to Qumulo.
Video:
Rclone data mover.
Overview:
Using the Rclone open-source technology (https://rclone.org/), the Diskover Rclone Data Mover Plugin provides the framework for reliable data movement based on pre-configured source and destination profiles. The plugin provides two mechanisms within Diskover to trigger data movement: 1) on-demand user-initiated file action directly from the Diskover interface, and 2) scheduled automated workflow based on file attributes meeting predetermined criteria.
Technical documentation:
Vcinity high-speed data transfer.
Overview:
Regardless of distance and latency, the high-speed data transfer Vcinity Plugin provides the framework for reliable and fast data movement based on pre-configured source and destination profiles. The plugin can move NFS, SMB, and S3 to any NFS, SMB, and S3 vendor, no matter the brand, ex: Dell, NetApp, HPE, etc.
The Vcinity High-Speed Data Transfer Plugin provides two mechanisms within Diskover to trigger data movement: 1) on-demand user-initiated file action directly from the Diskover interface, and 2) scheduled automated workflow based on file attributes meeting predetermined criteria.
Technical documentation:
Solution brief:
Video:
PLUGINS FOR CLEAN AND ACCESSIBLE DATA
Data integrity.
Creating data is expensive, and recreating lost or corrupted data can be even costlier. Our plugins provide peace of mind by ensuring the integrity of your files with precise validation tools, safeguarding your data against errors or corruption.
They also help maintain compliance and accessibility by automatically correcting file names with illegal characters or excessive lengths, streamlining data management and reducing the risk of costly mistakes. With these plugins, you can trust that your data remains accurate, compliant, and easy to manage, saving time and resources.
CORE
Hash differential checksums.
Overview:
DescriptionDesigned for precise data movement monitoring, the plugin checksums xxhash, md5, sha1, and sha256 hash values between the original file and the resulting file once it reaches its transfer destination, catching any possible file corruption in the process. The plugin alerts on areas where the file checksum of the source location does not match the file checksum of the destination location, which would then require a retransfer of these suspect files. A manifest is generated to provide insurance upon completion that all files arrived uncorrupted.
Technical documentation:
Video:
Illegal and long filename.
Overview:
Analyzes the index of all directories and file names for illegal characters, and long filenames or file paths to proactively find potential files with names that can break applications. Offending filenames are tagged with the corresponding non-conformance and the list of illegal filenames can then be sent via email reports. The plugin can be configured to remediate these issues with automatic renaming or character replacement.
Technical documentation:
Video:
Index differential.
Overview:
Designed to provide a list of file differences between two indices (or points in time). The differential list can be used to feed synchronization tools or identify deltas where two repositories should be identical. Outputs a CSV file containing the differences between the two indices.
Technical documentation:
PLUGINS FOR USER-INITIATED ACTIONS
File actions.
Authorized users, following role-based access controls, can initiate File Actions directly from the Diskover interface, providing a versatile range of use cases.
Whether it’s taking targeted actions on data or seamlessly integrating with third-party platforms, Diskover’s open-source architecture empowers users to develop custom File Action Plugins, enabling the automation of unique in-house workflows with ease and flexibility.
CORE
Export.
Overview:
Allows authorized users to preview and create a formatted CSV file, enabling integration with other applications that monitor and trigger workflows based on the file’s arrival.
Technical documentation:
Fix Unix permissions.
Overview:
Enables authorized users to change the Unix permissions of selected files or folders to a configured value.
Technical documentation:
Live view.
Overview:
Provides authorized users with a live view of a filesystem between indexing intervals in real-time.
Technical documentation:
User documentation:
Video:
Make links.
Overview:
Allows authorized users to create symlinks and hard links for selected files and directories.
Technical documentation:
PDF viewer.
Overview:
Enables authorized users to view and validate PDF files without accessing the source files.
Technical documentation:
Spectra.
Overview:
On-demand menu option that queries the Spectra API to verify additional status, such as whether the desired tape is in the library or on the shelf.
Technical documentation:
Develop your own file action.
Overview:
Due to its open-source architecture, Diskover allows users to easily develop their own File Action plugins to automate their custom in-house workflows. These plugins can be launched in one click using Diskover’s File Action feature.
Technical documentation:
MEDIA
CineViewer Player by CineSys.
Overview:
The CineViewer Player is designed to securely view high resolution media from a remote browser, without giving users access to the source files. Using the File Action feature from the Diskover user interface, CineViewer allows for seamless validation of media assets, therefore increasing productivity, while safeguarding your production network.
Learn more:
Technical documentation:
User documentation:
Video:
EDL check.
Overview:
Allows authorized users without read/write access to production storage to verify the validity of EDL (Edit Decision List) files.
Technical documentation:
EDL download.
Overview:
Allows authorized users without read/write access to production storage to upload or download specific lists of pre-defined file types and sizes, eliminating the need for data management group involvement.
Technical documentation:
Find file sequences.
Overview:
Designed to find/list any file sequences in a directory from a single file in a sequence.
Technical documentation:
User documentation:
IMF change report.
Overview:
Generates a list of IMF image changes in both human-readable format and machine-readable EDL, referencing the updated media.
Technical documentation:
IMF package validator by Oxagile.
Overview:
This plugin allows users to scan and validate IMF packages before delivery—from a remote platform and regardless of the location of the IMF Package data—saving immense amounts of man-hours over the course of a business year. IMF stands for the Interoperable Master Format, a technical standard used in the media and entertainment industry for exchanging digital content between different platforms and systems. This service helps content creators and providers validate their content and ensure it’s ready to share with their distribution partner or client by meeting IMF packaging requirements.
Learn more:
Technical documentation:
User documentation:
Video:
Telestream GLIM.
Overview:
This plugin results in a seamless integration with GLIM, allowing end-users to safely view and validate media files, while safeguarding your source assets and production network. Diskover allows users to do advanced searches of media assets, and then launch GLIM in one click via our File Actions.
Technical documentation:
User documentation:
Video:
Telestream Vantage.
Overview:
The Diskover Telestream Vantage Plugin allows end-users to submit files for transcoding directly from the Diskover user interface.
Learn more:
Technical documentation:
User documentation:
Video:
LIFE SCIENCE
Grant.
Overview:
The Grant Plugin has a dual purpose with 1) assisting research institutes in managing their grants/members/storage costs internally, and 2) fulfilling the requirements for the new NIH DMS Policy.
The Grant Plugin collects and parses grants’ metadata (grant number, group ID, etc.) to curated datasets. In turn, staff associated with a specific grant has visibility/searchability of their limited data/grant without access to the source files or other grants. That extra metadata is also available to use for further workflow automation if needed.
Learn more:
Technical documentation:
GET STARTED WITH
DISKOVER
Ready to manage your data everywhere from anywhere?