Send Data Stream Data to Object Storage

Hands-on to configure an Azion Object Storage connector to receive data from Data Stream.

While configuring streams, you need to set up a specific endpoint to stream your Azion data.

Continue reading for a step-by-step guide on configuring an Azion Object Storage connector to receive data from Data Stream.

Requirements

The S3 credentials used with the endpoint must include the ListBucket permission (to list and view objects within a bucket) and the PutObject permission (to add or overwrite objects within a bucket), ensuring the necessary access for reading and writing operations in the information storage, including logs. Find out more about S3 credentials.

Create an S3 credential via API and configure access to the bucket through s3cmd.
- Save the Access Key and Secret Key linked to your credential. You’ll need them later.
Create an Object Storage bucket, or use an existing one.

Configuring the connector in Data Stream

Next, follow these steps to configure the new connector in your Azion Data Stream.

You can find detailed steps for the entire configuration in the How to use Data Stream guide.

All key fields will be masked and an icon that allows revealing the field content will be only displayed to users with the Edit Data Stream permission. Users without the Edit Data Stream permission can only view the settings of the existing stream but not edit or create new ones.

On the Connector dropdown menu, select Simple Storage Service (S3).
On URL, add the URL of the Azion Object Storage S3 Host. Example: s3.us-east-005.azionstorage.net.
On Bucket Name, add the name of your S3 bucket on Object Storage. Example: mys3bucket.
On Region, add the region configured on Object Storage. Example: us-east-005.
On Access Key, add the access key you created on Object Storage. Example: ORIA5ZEH9MW4NL5OITY4.
On Secret Key, add the secret key you created on Object Storage. Example: +PLjkUWJyOLth3anuWXcLLVrMLeiiiThIokaPEiw.
On Object Key Prefix (optional), add the prefix for the name of your uploaded object. Example: user/logs.
On Content Type, add the format of the object that’ll be created in your S3 bucket. It can be plain/text or application/gzip.
Make sure the Active switch is on.
Click the Save button.

After saving the configurations, your data will be streamed to the newly configured endpoint.

The Object Key Prefix field refers to the prefix for the name of your uploaded object. The file name is composed of the Prefix, the date and time the packets are sent in the YYYY/MM/DD/hh/mm/ format, and the UUID. Example: user/logs/2024/10/12/06/24/37d66e78-c308-4006-9d4d-1c013ed89276.

You can keep track of the calls made by Data Stream to Object Storage in Real-Time Events. To do so, select Data Source > Data Stream and choose the filter options as you wish.