CUSTOMER SUCCESS STORY | LIFE SCIENCE

$250K in overspend avoided by unclogging the AI pipeline.


Pages of the Life Science AI Pipeline Success Story

500+ TB

of temporary and training data identified and cleared

$250K

in storage overspend avoided

4.7 PB

research storage estate indexed in place

27M

pathology files in one catalog across three sites

Only 7 percent of respondents say their organization’s data is completely ready for AI, according to Cloudera and Harvard Business Review Analytic Services. This center faced the same challenge. Its bioinformatics and research IT teams ran data-intensive AI pipelines across a 4.7 PB research storage estate of 27 million files, spread over three geographic locations, and as those workloads scaled, storage costs climbed with them.

What the team could not see was why. There was no way to tell whether the pipelines were running efficiently or simply accumulating specialized data on the side. Diskover indexed the full estate into one searchable catalog of files, owners, and ages, then classified what was piling up and pinpointed the temporary, transient, and training data consuming capacity across all three sites.

The center cleared 0.5 PB of temporary, transient, and training data and avoided $250K in storage overspend, while keeping its AI pipelines running throughout. The same index also reclaimed $542K of the data estate within days. The team came away with something it did not have before: ongoing visibility into which pipelines produce results and which ones produce leftovers.

A world-renowned cancer center on the US East Coast that combines a specialty hospital, a research institute with more than 100 laboratories, and a pathology department holding over a century of records.

Its pathology and research teams run imaging scanners, image management platforms, and machine learning pipelines that generate petabytes of data a year, spread across several file systems and storage clusters in three locations.

A small infrastructure team keeps that storage available for thousands of researchers, clinicians, and administrators.

Data

AI training data, temporary pipeline output, and specialized research datasets.

Storage indexed

4.7 PB and 27 million files across the center’s research storage estate.

Users

Bioinformatics, research IT, and lab operations teams.

Locations

Three geographic locations.

AI workloads generated data faster than the storage estate could absorb, driving costs up across three sites.

Specialized research data accumulated across the estate with little insight into what it was or whether it was still needed.

The team had no way to tell whether its AI data pipelines ran efficiently or simply piled up data on the side.

Pipelines crossed team boundaries, so no one group could say which data was still needed or authorize clearing it.

Diskover crawled 4.7 PB and 27 million files across three locations into one searchable catalog of files, owners, and ages, with the data staying in place.

Reports by application and location showed which pipelines left the most data behind, so the team could fix the biggest producers first.

Searches on fields such as path, age, and owner found temporary and training data unread since the run that created it.

Tags on fields such as path, file type, and age separated critical research data from pipeline leftovers, so the team knew what to keep.

“Every AI pipeline produces two things: the result somebody asked for, and the working files nobody planned for. At petabyte scale, that second pile quietly turns into your next storage purchase. Reading the metadata is how you tell the two apart before you buy more disk.”

Paul Honrud, CPO, Diskover

The pipeline produced results. It also produced half a petabyte.

AI pipelines are built to produce results, not to clean up after themselves.

0.5 PB

cleared and $250K overspend avoided, through global visibility and data curation

Zero

interruption to a running pipeline

Ongoing

view of which pipelines leave data behind

Outcomes.

The center cleared 0.5 PB and avoided $250K in storage overspend, while keeping its AI pipelines running throughout.

The team gained ongoing insight into pipeline efficiency and the data each run leaves behind.

Diskover identified and cleared over 500 TB of temporary and training data, freeing storage resources.

Data management stayed efficient and uninterrupted throughout, supporting the center’s research mission.

Questions people ask.

Why do AI pipelines fill research storage so quickly?

Every training run, checkpoint, test set, and intermediate step writes new files, and pipelines are built to produce results, not to clean up after themselves. Across several sites and file systems, those working files grow faster than anyone can track them, and they end up driving the next storage purchase.

What kind of data piles up in life science AI pipelines?

Mostly temporary and transient output between pipeline stages, copies and checkpoints from training runs, and old test sets. It sits next to the research datasets that teams still need, so the challenge is telling the two apart. File metadata, such as path, file type, owner, and age, separates them at scale.

How did this cancer research center avoid $250K in storage overspend?

Diskover indexed its 4.7 PB research storage estate, 27 million files across three locations, into one catalog. Searches and tags on path, age, and owner surfaced more than 500 TB of temporary and training data that had not been read since the run that created it. The center cleared 0.5 PB and avoided $250K in storage overspend, and the same index helped reclaim $542K of the data estate within days.

Does Diskover read the files to classify them?

No. Diskover works from metadata, such as name, path, owner, file type, and age, plus the tags teams add and enriched metadata from plugins. It never reads file contents, which keeps indexing fast and non-invasive, even across petabytes of research data.

Can the cleanup happen without stopping research pipelines?

Yes. Diskover indexes the data in place, so files stay where the pipelines expect them. The team decided what to keep and what to clear based on the catalog, and its AI pipelines kept running throughout the project.

Scroll to Top