Glossary Storage Systems, Protocols, and ScaleObject storage

Object storage

Object storage holds flat objects identified by a key, reached over an HTTP API such as S3 or Azure Blob. Objects can carry custom metadata as key-value pairs, so some description travels with the object itself. Some scale-out file systems offer the same on files, and many do not, which is why a catalog that reads both keeps one consistent description across file and object storage.

Key takeaways

Object storage scales to very large volumes in a flat namespace.
Objects can carry custom metadata, set by the application that writes them.
File storage has paths and folders, while object storage has buckets and keys.

What is object storage used for?

For large amounts of data that is written once and read many times: backups, archives, media libraries, data lakes, logs, and data sets for analytics and AI. Object storage scales to billions of objects and usually costs less per terabyte than file storage, which is why most cloud storage is object storage.

What does S3 expose as metadata, and what does it leave out?

S3 exposes the key, size, last modified time, storage class, entity tag, and any user metadata set at upload. It does not record a last access time in the way a file system does, so access patterns have to come from access logging or from an inventory report rather than the object itself.

Is object storage metadata indexed like any other storage?

Yes. It ends up in the same catalog as file system metadata, and it is searched and reported on the same way. Only the route is different: instead of walking a folder tree, the scanner asks the storage for a list of objects through its API, and treats the parts of each key separated by slashes like folders. Because objects carry fewer fields than files, such as no owner and no last access date, tags and business attributes often matter more for object data.

What is the difference between file storage and object storage?

File storage organizes data in a directory tree and is reached over NFS or SMB. Object storage holds flat objects with a key, reached over an HTTP API such as S3. Object storage lets you attach custom metadata to each object, which file systems generally do not, so some description can travel with the object itself. For a file on a NAS share, that description has to live somewhere else, which is what a catalog is for.

What is an S3 bucket?

A bucket is a container for objects in Amazon S3, and in other storage that uses the same S3 interface. Each bucket has a name, a region, and access rules, and it holds any number of objects, each with a key that works like a file path. Many organizations have hundreds of buckets, which is one reason object storage belongs in any data inventory.