
500+ TB
of temporary and training data identified and cleared
$250K
in storage overspend avoided
4.7 PB
research storage estate indexed in place
27M
pathology files in one catalog across three sites
At a glance.
Only 7 percent of respondents say their organization’s data is completely ready for AI, according to Cloudera and Harvard Business Review Analytic Services. This center faced the same challenge. Its bioinformatics and research IT teams ran data-intensive AI pipelines across a 4.7 PB research storage estate of 27 million files, spread over three geographic locations, and as those workloads scaled, storage costs climbed with them.
What the team could not see was why. There was no way to tell whether the pipelines were running efficiently or simply accumulating specialized data on the side. Diskover indexed the full estate into one searchable catalog of files, owners, and ages, then classified what was piling up and pinpointed the temporary, transient, and training data consuming capacity across all three sites.
The center cleared 0.5 PB of temporary, transient, and training data and avoided $250K in storage overspend, while keeping its AI pipelines running throughout. The same index also reclaimed $542K of the data estate within days. The team came away with something it did not have before: ongoing visibility into which pipelines produce results and which ones produce leftovers.
CUSTOMER PROFILE
A world-renowned cancer center on the US East Coast that combines a specialty hospital, a research institute with more than 100 laboratories, and a pathology department holding over a century of records.
Its pathology and research teams run imaging scanners, image management platforms, and machine learning pipelines that generate petabytes of data a year, spread across several file systems and storage clusters in three locations.
A small infrastructure team keeps that storage available for thousands of researchers, clinicians, and administrators.
Data
AI training data, temporary pipeline output, and specialized research datasets.
Storage indexed
4.7 PB and 27 million files across the center’s research storage estate.
Users
Bioinformatics, research IT, and lab operations teams.
Locations
Three geographic locations.
Challenges.
Uncontrolled storage growth.
AI workloads generated data faster than the storage estate could absorb, driving costs up across three sites.
Unknown data mix.
Specialized research data accumulated across the estate with little insight into what it was or whether it was still needed.
Pipeline blind spots.
The team had no way to tell whether its AI data pipelines ran efficiently or simply piled up data on the side.
No single owner.
Pipelines crossed team boundaries, so no one group could say which data was still needed or authorize clearing it.
Solutions.
Indexed the full estate.
Diskover crawled 4.7 PB and 27 million files across three locations into one searchable catalog of files, owners, and ages, with the data staying in place.
Optimized the AI pipelines.
Reports by application and location showed which pipelines left the most data behind, so the team could fix the biggest producers first.
Pinpointed wasted capacity.
Searches on fields such as path, age, and owner found temporary and training data unread since the run that created it.
Classified the data mix.
Tags on fields such as path, file type, and age separated critical research data from pipeline leftovers, so the team knew what to keep.
“Every AI pipeline produces two things: the result somebody asked for, and the working files nobody planned for. At petabyte scale, that second pile quietly turns into your next storage purchase. Reading the metadata is how you tell the two apart before you buy more disk.”
Paul Honrud, CPO, Diskover
The pipeline produced results. It also produced half a petabyte.
AI pipelines are built to produce results, not to clean up after themselves.
0.5 PB
cleared and $250K overspend avoided, through global visibility and data curation
Zero
interruption to a running pipeline
Ongoing
view of which pipelines leave data behind
Outcomes.
$250K overspend avoided.
The center cleared 0.5 PB and avoided $250K in storage overspend, while keeping its AI pipelines running throughout.
Clear pipeline visibility.
The team gained ongoing insight into pipeline efficiency and the data each run leaves behind.
Half a petabyte reclaimed.
Diskover identified and cleared over 500 TB of temporary and training data, freeing storage resources.
Research kept moving.
Data management stayed efficient and uninterrupted throughout, supporting the center’s research mission.
Questions people ask.
Why do AI pipelines fill research storage so quickly?
Every training run, checkpoint, test set, and intermediate step writes new files, and pipelines are built to produce results, not to clean up after themselves. Across several sites and file systems, those working files grow faster than anyone can track them, and they end up driving the next storage purchase.
What kind of data piles up in life science AI pipelines?
Mostly temporary and transient output between pipeline stages, copies and checkpoints from training runs, and old test sets. It sits next to the research datasets that teams still need, so the challenge is telling the two apart. File metadata, such as path, file type, owner, and age, separates them at scale.
How did this cancer research center avoid $250K in storage overspend?
Diskover indexed its 4.7 PB research storage estate, 27 million files across three locations, into one catalog. Searches and tags on path, age, and owner surfaced more than 500 TB of temporary and training data that had not been read since the run that created it. The center cleared 0.5 PB and avoided $250K in storage overspend, and the same index helped reclaim $542K of the data estate within days.
Does Diskover read the files to classify them?
No. Diskover works from metadata, such as name, path, owner, file type, and age, plus the tags teams add and enriched metadata from plugins. It never reads file contents, which keeps indexing fast and non-invasive, even across petabytes of research data.
Can the cleanup happen without stopping research pipelines?
Yes. Diskover indexes the data in place, so files stay where the pipelines expect them. The team decided what to keep and what to clear based on the catalog, and its AI pipelines kept running throughout the project.
Related resources.
See it on your own data.
We’ll index a sample of your environment and show you
what’s recoverable against your current storage spend.
