How to use Amazon S3 to receive data from Data Stream

Hands-on to configure Amazon S3 to receive data from Data Stream.

View as Markdown

While configuring streams, you need to set up a specific endpoint to stream your Azion data.

Continue reading for a step by step on how to connect an Amazon S3 endpoint to receive data from Data Stream.

Amazon S3 requirements

The S3 credentials used with the endpoint must include the s3:ListBucket permission (to list and view objects within a bucket) and the s3:PutObject permission (to add or overwrite objects within a bucket), ensuring the necessary access for reading and writing operations in the information storage, including logs. Find out more about S3 credentials.

To get started, you must complete a few configurations in Amazon S3:

Create an Amazon AWS account.
Register an Identity and Access Management (IAM) credential.
Create an Amazon S3 bucket.
Optionally, you can also upload an object.
Optionally, you can set up and manage Server Side Encryption (SSE) with Amazon S3-Managed Keys (SSE-S3).

You’ll also need to save and create the following information:

Region
Access Key
Secret Key

Region

This is the region where your Amazon S3 is running.

On the top bar of your dashboard, right next to the help button with the ’?’ icon, you find a dropdown menu, the Regions selector, where you can see the name of the region and its code.
Find the code and keep it to use it later. It looks similar to this: us-east-1

You can find more details on the AWS Regions and Zones documentation.

Access Key and Secret Key

Access the Identify and Access Management (IAM) panel.
Click the Create new access key button.
A pop-up will appear on the screen with your key information.
Click on Display Access Key to display your key information as indicated below:
- Access Key ID, with a format similar to AKIA5LVCRTER524RAIA3
- Secret Access Key, with a format similar to N9UTjunb299M5ryfKdG2cKh2+Pe2mBiGZXaBpVff
Save the two keys information in a safe place to use them later on Azion Data Stream.

You can also download the EMP file with the information for the keys.

For more details on setting up Amazon S3, see the AWS documentation page.

Configuring the endpoint in Data Stream

Next, follow these steps to configure the new endpoint you created in Amazon S3 in your Azion Data Stream.

You can find detailed steps for the entire configuration on the How to use Data Stream guide.

All key fields will be masked and an icon that allows revealing the field content will be only displayed to users with the Edit Data Stream permission. Users without the Edit Data Stream permission can only view the settings of the existing stream but not edit or create new ones.

In the Destination configurations:

On the Connector dropdown menu, select Simple Storage Service (S3).
On URL, add the URL of your S3 host.
On Bucket Name, add the name of your S3 bucket on AWS.
On Region, add the region configured on AWS.
On Access Key, add the access key you created on AWS.
On Secret Key, add the secret key you created on AWS.
On Object Key Prefix (optional), add the prefix for the name of your uploaded object. Example: user/logs.
On Content Type, add the format of the object that’ll be created in your S3 bucket. It can be plain/text or application/gzip.
Make sure the Active switch is on.
Click the Save button.

On the Endpoint Type dropdown menu, select Simple Storage Service (S3).
On Host URL, add the URL of your S3 host.
On Bucket Name, add the name of your S3 bucket on AWS.
On Region, add the region configured on AWS.
On Access Key, add the access key you created on AWS.
On Secret Key, add the secret key you created on AWS.
On Object Key Prefix (optional), add the prefix for the name of your uploaded object. Example: user/logs.
On Content Type, add the format of the object that’ll be created in your S3 bucket. It can be plain/text or application/gzip.
Make sure the Active switch is on.
Click the Save button.

The Object Key Prefix field refers to the prefix for the name of your uploaded object. The file name is composed of the Prefix, the date and time the packets are sent in the YYYY/MM/DD/hh/mm/ format, and the UUID. Example: user/logs/2024/10/12/06/24/37d66e78-c308-4006-9d4d-1c013ed89276.

After saving the configurations, your data will be streamed to the newly configured endpoint.

You can keep track of the calls made by Data Stream to Amazon S3 on Real-Time Events. To do so, select Data Source > Data Stream and choose the filters options as you wish.

Watch a video tutorial on how to use Amazon S3 with Data Stream on Azion’s YouTube channel:

Trademarks

Amazon Simple Storage Service is a registered trademark of Amazon Web Services in the United States and other countries.