Data lakehouse

A data lakehouse combines the low-cost storage of a data lake with the table structure and transactions of a data warehouse, so analytics and machine learning work from one place. It changes how data is read, not how it was chosen.

Key takeaways

Lakehouses organize tables, and files still need their own description.
Much of the data AI needs still sits in files around and inside the lake.
A file catalog lets those files be selected by business context.

What is the difference between a data lake and a data lakehouse?

A data lake stores raw files in their original formats and leaves structure to whatever reads them. A lakehouse adds table structure and transaction support over that storage, so analytics and machine learning can query it directly. Both hold unstructured data, and both work from what has already landed in them. Deciding which files land, and describing them first, is a separate job.

Can an analytics engine query file metadata?

Yes. A metadata catalog can be exported in a standard table format or queried as a source, so an analytics engine can join file descriptions with business data in one query. Teams use it to answer questions such as how much storage each project uses, or which approved files belong in a training set. Diskover supports both approaches, so the files stay in place while their descriptions join the analytics work.

What is an AI-ready data lake?

An AI-ready data lake is a data lake where every file is described well enough for a model to use it on purpose. Each file needs details such as its source, owner, project, and sensitivity recorded, so teams can select a defined set instead of pointing a model at everything. In practice, that description comes from a metadata catalog that covers the lake and the storage around it.

Why does a lakehouse need a file catalog?

A lakehouse organizes tables well, but much of the data AI and analytics need still sits in files, such as documents, images, video, and instrument output. A file catalog describes those files, so they can be found, filtered, and joined to table data by shared fields such as a project or order number. With it, files can be selected alongside tables instead of staying out of view.

How do you get unstructured data into a lakehouse?

By selecting it first, rather than copying everything:

Find the files with a metadata catalog, filtered by attributes such as project, type, date, and tags.
Check them for duplicates, outdated versions, and sensitive content.
Load or reference them in the lakehouse, with their metadata attached as a table.
Keep the link between each record in the lakehouse and the source file, so results can be traced back.