I am trying to configure AWS Firehose to write to a specific s3 directory. For this, in the prefix section of Firehose console, I gave "Sample/". Attaching the screenshot.

Even after this, in the S3, its creating the folder structure S3bucket/Sample/YYYY/MM/dd/file.json.

Is there a way to omit the expressions (eg.YYYY/MM/dd) and let the firehose use only the prefix, like this => S3bucket/Sample/file.json?

I am trying to configure AWS Firehose to write to a specific s3 directory. For this, in the prefix section of Firehose console, I gave "Sample/". Attaching the screenshot.

Even after this, in the S3, its creating the folder structure S3bucket/Sample/YYYY/MM/dd/file.json.

Is there a way to omit the expressions (eg.YYYY/MM/dd) and let the firehose use only the prefix, like this => S3bucket/Sample/file.json?

• amazon-web-services
• amazon-s3
• amazon-kinesis
• amazon-kinesis-firehose

Share Improve this question Follow asked Nov 20, 2024 at 15:31 hari krishnanhari krishnan 1333 bronze badges

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 0

One solution is to add lambda function that triggers on S3 events and changes the final file name. In your case removing the date and other stuff and final name like, file.json

You can avoid the default time partitions using the dynamic partitioning. That will override the default partitions.

Only caveat is that the dynamic partitioning should be enabled while creating the firehose stream.

You can refer the documentation on how to enable it. https://docs.aws.amazon/firehose/latest/dev/dynamic-partitioning-partitioning-keys.html