Which autonomous vehicle training data do your models still need?

An autonomous test fleet comes back from a day of driving with terabytes of camera, light detection and ranging (LiDAR), radar, and Global Positioning System (GPS) data sitting at the site where it drove.

Autonomous vehicle training data is the subset of those recordings a perception model learns from, and choosing it well starts with knowing what each recording captured before any of it travels.

Bill Burns, Director of Digital Archaeology at Diskover, set out to show that a fleet’s own metadata can make recordings at every site, and on every storage system, searchable against the criteria an AI team defines, so the team can rank candidates before paying to move them.

The question behind the bandwidth bill.

Bill was in the room with the infrastructure and artificial intelligence (AI) teams of a global automaker building autonomous vehicles. Their default plan shipped every recording from every test site back to headquarters, where the models are trained. The infrastructure team counted the cost in bandwidth, transfer windows, and duplicate storage. The AI team counted it in weeks spent waiting for data the model might not need.

Their question was simple to say and hard to answer: which recordings do our models still need? If the training set already holds plenty of one driving condition, the next recording of that condition can stay at the edge. The answer lives inside the files, though, and opening a fleet’s worth of recordings to find it is a project in itself.

The pressure is real across the industry. Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data. Bill’s bet was that the gaps in the autonomous vehicle training data could be found from metadata, before a single recording moved.

Teaching the catalog to read a recording.

Bill started with the file format. MCAP is an open container format for robotics and sensor data: a file commonly holds a recording session, or part of one, with timestamped messages from every sensor grouped into topics such as LiDAR point clouds, camera frames, GPS fixes, and vehicle state. To a file system, that same file is a name, a size, and a date.

So Bill wrote a plugin that reads each MCAP file as Diskover scans it and writes what the file already knows about itself into the catalog:

• The dataset and sequence the recording belongs to
• The vehicle or platform that captured it
• The sensors present
• Start time, end time, and duration
• The GPS position of the capture

The plugin is extensible, so fields are added as a team’s questions change. Each field answers something an AI team asks every week: where was this captured, when, with which sensors, and for how long? Once the fields land in the catalog, a request like “every LiDAR recording across every dataset” becomes one search.

A fleet’s recordings on one page.

The automaker’s recordings stayed with the automaker, so Bill gathered public ones: nine open robotics and self-driving datasets, from full-size test vehicles to drones and portable capture rigs, all stored in or converted to MCAP format. He put them on a file share, ran one scan, and built a fleet dashboard from the catalog.

The summary alone tells a story: 2,067 recordings, 2.07 terabytes (TB), 18.7 hours of recorded time, and 336 kilometers (km) of driving and flight. One dataset holds more than half the bytes. One page of metadata shows an AI team what it has before anyone opens a file.

The sensor chart shows the first gap. Every recording carries LiDAR, and nearly every one carries camera and GPS, but only half carry radar or an inertial measurement unit (IMU). A team assembling autonomous vehicle training data for a radar-heavy model learns that from one bar chart.

The dashboard also flagged problems that usually surface weeks into training. About half the recordings carried a suspect clock: one dataset publishes time relative to the GPS epoch, so its real capture dates, and the weather on those dates, are unrecoverable. Older recordings were still uncompressed, hundreds of gigabytes (GB) of recompression candidates.

Adding the weather the files never recorded.

For this AI team, one attribute mattered more than the rest: the weather at capture time. A perception model that has seen endless clear afternoons and almost no heavy rain has a known gap, and the quickest way to close it is to find the rain recordings already sitting at the edge.

MCAP can carry any metadata a team chooses to record, but these recordings carried no weather labels. Every recording in the catalog now carried a GPS position and a start time, though, and that pair works as a key into outside data.

Bill added a workflow step, separate from the plugin, that takes each recording’s capture position and start hour, looks up the historical weather for that place and hour, and attaches the condition, temperature, wind, and precipitation to each recording’s record in the dashboard.

This is metadata enrichment: fields from an outside source attached to the file’s own record. 1,047 recordings had a trustworthy time and a position, and every one received its weather. The result is a label from a weather archive rather than a measurement from the vehicle, and one hourly value at the start of a recording can miss a change in conditions along the route. Bill treats it as a way to find candidates, which is what the team needed.

Bill’s point reaches well past weather. Place and time also join a test schedule, a road closure list, a vehicle maintenance log, or a map of operational design domains (ODD). Every join adds a field the AI team can filter on, while the recordings stay on the storage where they were captured.

What the charts said about autonomous vehicle training data.

With weather attached to each recording, the dashboard could answer the original question, using criteria that come from the AI team. One chart plots every recording at its capture point. Another stacks each site’s recordings by weather condition, and a third plots the air temperature of every recording by site.

Boston showed the full range: clear, partly cloudy, overcast, drizzle, and rain. On the afternoon of September 18, 2018, one day’s driving produced 179 recordings, just under an hour of footage, while the weather archive showed rain over the site. Those 179 recordings became rain candidates, labeled with their capture place and time. If the models need more rain, those are the recordings to prioritize, and the dashboard names each one. For that one need, the selection is 179 of 2,067 recordings, under 9% of the set.

The automaker’s AI data architect read the chart at once. Looking for drizzle, he named the sites himself: Boston, Singapore, and Hamilton. Using the metadata as the key for what to bring home, he said, is “a game changer” for keeping storage costs down, and the metadata is “almost more important than the data itself.”

Istanbul, the Southern District of Hong Kong, and Vedi in Armenia recorded clear weather in every matched recording. If the training set already holds enough clear-weather driving, a rule focused on rain leaves those recordings where they are. A clear-weather recording can still hold a rare object or a failure case worth keeping, so the rule sets priority for one need. Recordings outside the selection stay at the edge, available for the next need, until their retention period ends.

The temperature chart exposed a gap the team had not asked about.

Boston drives span 19–35 °C, the Singapore scenes sit between 26 and 30 °C, and the single near-freezing capture is one drive in Hamilton, Ontario, at 0.5 °C. Among the recordings with weather attached, there is no snow and no sub-zero driving.

What would your model do on its first icy morning?

A shortage like this is a gap in coverage, and only the AI team’s own model evaluation can say whether it is also a gap in performance. The dashboard makes the question searchable: if winter matters, winter recordings become the next candidates.

From a proposed tag to a moved file.

A chart that names files is half the workflow. The other half moves them, without someone copying file paths into a script. Bill’s dashboard proposed tags from the enriched fields: recordings captured in a wet hour, in high wind, or above 30 °C, plus the files to recompress. Each tag stayed a proposal until a person approved it.

That approval step is also how Diskover works with AI more broadly. A team can ask its question in plain language through the AI Connector, get the matching recordings with a recommended tag, and apply it only after a person says yes: AI-assisted, human-approved.

Teams can also tag directly in the Diskover interface. From there, a Diskover policy copies the tagged recordings to the training store, and every other recording stays where it is until its retention period ends.

The demo itself ran on one file share, covering indexing, the weather lookup, and proposed tags. The deployment the automaker’s infrastructure team cared about extends that across every site and every storage system.

One central Diskover instance holds the catalog and the dashboards. Diskover indexes the storage at each remote site, close to where the recordings land. Metadata travels to the center on every scan, and recordings travel only when a tag and a policy call for them.

Their infrastructure lead described it as one Diskover across every site, with the decision made centrally and the bytes kept at the edge until a model needs them.

The scale explains why. Each of the automaker’s vehicles creates about 5 TB of recordings a day, and the team is designing for a fleet of 100 vehicles: 500 TB a day, with growth planned beyond that. In the team’s working numbers, that means on the order of a million MCAP files per site. At that scale, the metadata is the difference between a bandwidth plan and a bandwidth problem. As the automaker’s AI data architect put it: “The more we grow, the more important these capabilities become.”

What the AI architects asked next.

The infrastructure team saw the transfer savings right away. The data and AI architects pushed further, and their questions are the ones any AI team will ask.

Can it read our recordings, or do we send our own pipeline’s output? Either path works. A plugin can read the format during the scan, or a team can bring the fields its existing extraction pipeline already produces, and both land in the same catalog.

Our value sits deep inside the file. Can we bring our own extraction models? Yes. Fields produced by a team’s own models can be added to the catalog alongside the plugin’s, and from there they drive search, tags, and movement like any other field.

Can we take the catalog out? Yes. The index, tags, and jobs are available through a documented representational state transfer (REST) API, the catalog runs on Elasticsearch for teams that want to query it directly, and metadata exports to Parquet files that Snowflake and Databricks ingest, so the catalog can join everything else in the data lake.

Can we see how the data drifts over time? Each scan is a snapshot, and comparing scans shows what changed. Tracking how one site’s conditions shift as its operating domain grows is an analysis Bill would build with a team on top of those snapshots.

What other dense formats can it read? Diskover offers plugins for many industries, including genomics files, media containers, images, and PDFs, and teams build their own for formats unique to their work.

Every file has a where and a when.

This example comes from autonomous driving, but the same workflow applies well beyond vehicles. Strip the labels off and it is four steps.

1. Read what each file already carries.
2. Add the context it lacks from a system the business already runs.
3. Let the people who own the question set the selection criteria.
4. Move only the files that meet them, and leave the rest where they are.

The pattern fits any field where the data is large, spread across sites, and judged by criteria that live outside the file. A life sciences team joins instrument runs to the sample records in its laboratory information management system (LIMS) to decide which runs move to analysis. A media team joins dailies to the production schedule and camera metadata to decide which footage moves to active storage and which stays in the archive. A manufacturer joins test logs to maintenance records to find the runs that preceded a failure. Any company joins its shared drives to project and personnel systems to answer who owns each file, whether it is still needed, and where it should live. Automotive teams already catalog drive test and sensor data by campaign, vehicle, and date. The weather join adds one more field to that record.

In each case, the catalog does the job it did for the fleet. It makes the business’s own criteria searchable across every file, on every storage system, so the selection happens before anything moves. That is what Diskover’s Metadata Control Plane is for: it sits above the storage layer, reads every file where it is, enriches it from the systems around it, and moves only what a rule and a person call for.

Diskover lets a company use its metadata to decide how it wants to work with its data. In each case, the catalog does the job it did for the fleet: it makes the business’s own criteria searchable across every file, on every storage system, so the selection happens before anything moves. 

“If your team has a version of the automaker’s question, tell me what it is. The answer is almost always in metadata you already have, and finding it is the part of this job I like best.”

— Bill Burns, Director of Digital Archaeology, Diskover