Introduction

Amazon has recently launched a new feature in Amazon S3 called Amazon S3 Metadata. It is considered to be easiest, fastest, powerful and efficient solution for quickly discovering and understanding your S3 data.

By automatically generating and updating metadata in near real-time, it enables seamless data curation, identification, and utilization for business analytics, real-time inference applications, and more.

Integration with AWS Glue Data Catalog, currently in preview, enables seamless streaming, querying, and visualization of data—including S3 Metadata tables—using a range of AWS analytics services such as Amazon Kinesis Data Firehose, Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon QuickSight.

Furthermore, S3 Metadata integrates with Amazon Bedrock, allowing for the annotation of AI-generated content, such as videos, with metadata that includes details such as the AI origin, creation timestamp, and the specific model used in the generation process.

This supports both system-defined object metadata, such as size and object source, as well as customizable metadata, allowing you to label objects with relevant tags like product SKUs, transaction IDs, content ratings, and other key information to enhance data organization and usability.

Is it Available for all regions?

no, for now this feature is available in only two AWS Regions:

1. US East: N. Virginia, Ohio

2. US West: Oregon

Where is this S3 Metadata Preview?

You can find the option to preview the S3 Metadata, is by clicking the Metadata tab of the bucket in the regions where this new feature is available.

Enabling Metadata

• Create a table bucket, navigate to Metadata-Preview tab and click on Create metadata configuration.

  

• You need to first create table bucket. Click on it, this will take you to new page to create the table bucket:

  

• Specify the Name for Table Bucket. Make sure to enable the Integration for AWS analysis services for your use cases , as it will be disabled by default. Click on create table.

  The table bucket name must qualifies below criteria for the naming convention:

  • It should be unique for the account in the current Region.

  • Should be between 3-36 characters.

  • No Uppercase characters are allowed

  • Only should be a either in lowercase characters, numbers and hyphen is allowed

  • It should start & end with either a letter or number.

• Move to below tab and click on browse S3. You will get your table listed there that you have created just now

  

  

• Select it and click on create Metadata configuration.

  

  

• You can view your metadata table data by using services such as Amazon Athena and Amazon Redshift if your destination table bucket is integrated with AWS Glue Data Catalog

    Format: arn:aws:s3tables:region:accountnumber:bucket/<table-bucket>
  

Using CLI

• You can also create table bucket using below command:

aws s3tables create-table-bucket --name <tablebucketname> --region <region_name>

• Viewing details about an Amazon S3 table bucket

aws s3tables get-table-bucket --table-bucket-arn arn:aws:s3tables:us-east-1:<AWSAccountnumber>:bucket/<tablebucketname>

Error in Metadata Preview

What if you are getting error in the metadata page as below??

Analysis: The error is caused by insufficient permissions for the user.

Resolution: You need to assign required permission for the user. Follow below steps:

1. Navigate to the user in IAM console.

2. Click on the Permissions tab, and then click Add permissions.

3. Select the Create inline policy option and navigate to the JSON tab in the policy editor.

4. Copy and paste the below policy document into the policy editor.

   This policy document is a starting point, and if you want to add more permissions. Edit and play around the "Action" and "Resource" fields to get the desired specific actions and resources required for your use case:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketMetadataTableConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::<your-bucket-name>"
            ]
        }
    ]
}

6. Review and click on Next.

7. Give name to this inline policy, review the permissions defined in the policy, and then choose Create Policy to create.

By following these steps, you can grant the necessary permissions to the specific user to perform s3:GetBucketMetadataTableConfiguration on the resource arn of S3 bucket.

Now you have the required permission, feel free to move to above section and proceed for the configuration.

Is enabling S3 Metadata free?

No, it will be charged according to S3 pricing

Deleting a table bucket

aws s3tables delete-table-bucket \
    --region us-east-2 \
    --table-bucket-arn arn:aws:s3tables:us-east-1:111122223333:bucket/amzn-s3-demo-bucket1

Thanks for reading till the end, I hope it was helpful.

If you like my work, Let's connect and collaborate😃. I am available on the below platforms and very much active there:

Linkedinℹ️
GitHub😻
Blogs👩🏾‍💻
Topmate🤝

Let me know if this helps and feel free to add your suggestions and resources to help the community🤝