# Data Stream

import Tag from '~/components/webkit/Tag.vue'
import DocButton from '~/components/webkit/DocButton.vue';

**Data Stream** is an [Observe](/en/documentation/products/observe/) product that allows you to feed your stream processing, SIEM, and big data platforms with the event logs from your applications on Azion in real time. To keep an enhanced performance, it uses ASCII encoding to avoid parser issues and problems in data interpretation.

Data Stream is responsible for collecting, grouping, and transmitting raw logs to your chosen destination, providing flexibility for data integration. You can choose from which of the available Azion product and domains you want to collect your logs from and connect them to the endpoint of your data analysis platforms. You can also decide which data you want to use on your analysis by choosing among the available variables.

By accessing Data Stream through [Azion Console](https://console.azion.com/data-stream) or [Azion API](https://api.azion.com/#04257075-3691-4b5f-b22b-22a5b6653f3f), you can perform a variety of actions to manage your data flows, including:

- Have an organized set of logs.
- [Create a new data stream](/en/documentation/products/guides/use-data-stream/).
- Connect your Data Streams to [endpoints](/en/documentation/products/guides/#observe).
- [Edit data stream settings](/en/documentation/products/guides/use-data-stream/) directly from the list by selecting the stream you want to update.
- [View detailed information](/en/documentation/products/observe/troubleshoot/data-stream-understand-metrics/) about each stream to ensure everything is running smoothly.
- [Delete or stop data streams](/en/documentation/products/guides/observe/delete-data-stream/) that are no longer needed to optimize your system.
- Understand the behavior of your users.
- Analyze the performance of your content and applications.
- Identify data on security threats.
- Make informed decisions.
- Improve your applications and your business through reliable observability practices.

After configuring your Data Stream, you can check your logs successfully being sent through [Real-Time Events](/en/documentation/products/observe/real-time-events/).

:::tip
Visit the [Guides page](/en/documentation/products/guides/#observe) and the [Observe section](/en/documentation/products/observe/overview/) to explore more options for managing your data streams.
:::

---

## Implementation

| Task | Guide |
| ---- | ----- |
| Accessing Data Stream | [Data Stream first steps](/en/documentation/products/observe/data-stream/first-steps/) |
| Configure Data Stream | [How to configure Data Stream main settings](/en/documentation/products/guides/use-data-stream/) |
| Stop or delete a data stream| [How to stop or delete a data stream](/en/documentation/products/guides/observe/delete-data-stream/) |
| Associate domains | [How to associate domains on Data Stream](/en/documentation/products/guides/data-stream-associate-domains/) |
| Create customize template | [How to create a custom template on Data Stream](/en/documentation/products/guides/data-stream-custom-template/) |

---

## How to manage Data Stream

Azion Data Stream allows you create, edit, view, and delete data stream configurations.

You can [create a new data stream](/en/documentation/products/guides/use-data-stream/) through [Azion Console](https://console.azion.com/data-stream):

1. Go to **Product menu** > **Data Stream**.
2. Click the **+ Stream** button.
3. Complete the configuration.
4. Click the **Save** button.

To [edit](/en/documentation/products/guides/use-data-stream/) any stream, go to the **Data Stream** page and select the stream you want to work with. Edit the configurations as desired and save the changes.

If you want to [stop a data stream](/en/documentation/products/guides/observe/delete-data-stream/) in your Azion account, follow these steps: 

1. [Access Azion Console](https://console.azion.com/) > **Data Stream**.
2. In the list, select the stream you want to stop.
  - On the right side of the row, you can see a label indicating the stream's status: **Active** or **Inactive**.
3. On the configuration page, navigate to the Status section.
4. Disable the switch to stop the data streaming.
5. Click the **Save** button.

To activate the streaming again, repeat this process, enabling the switch in the **Status** section.

To [delete](/en/documentation/products/guides/observe/delete-data-stream/) a stream: go to the **Data Stream** page, navigate to the right side of the row, and click the **trash icon** for the stream you want to delete. 

---

## Logs

By default, **Data Stream** sends your event logs when the code block with variables reaches *2,000 records*, *every 60 seconds*, or when the packet size reaches the value determined in the *payload Max Size field*, whichever occurs first. However, in case you're using the AWS Kinesis Firehose endpoint, Data Stream will send your event logs when the block reaches *500 records* or every *60 seconds*.

For example, if a package reaches 2,000 records in 13 seconds, it'll be sent.

Once you activate a stream, there's a propagation time until the logs become available for consultation in tools or products such as **Real-Time Events**.

To find detailed reference about each section of configuring a Data Stream, keep reading the following subsections.

---

## Data sources

A **Data Source** represents the application at Azion that generates the event logs you want to use. By selecting one, you decide where your data will be collected from and the remaining product settings are configured according to your choice.

Selecting a Data Source in the dropdown list is mandatory. You can choose between:

- [Activity History](#activity-history)
- [Applications](#applications)
- [Functions](#functions)
- [WAF Events](#waf-events)

Each data source has a preset of *variables*, combined in a template, representing the specific information you can receive from your event logs. See each data source's prerequisites and variables and what data they provide next.

### Activity History

:::note
You can't associate domains if you use the **Activity History** data source.
:::

The **Activity History** data source displays the data for [logs activity](/en/documentation/products/accounts/activity-history/) regarding your account on Azion Console. The following variables are available for this option:

| Variable | Description | Example |
| --- | --- | --- |
| $account_id | Azion account identifier. | "8437" |
| $created_at | Date and time when the resource was created. Enables filtering and ordering by creation time. | "2026-03-30T14:25:00Z" |
| $author_email | Email address of the Azion Console user who performed the action. | "myemail@gmail.com" |
| $author_name | Name of the Azion Console user who performed the action. | "User name" |
| $client | Unique Azion customer identifier. | "4529r" |
| $comment | Optional field available for users to add comments when performing changes. | "Changed Rule" |
| $parent_resource_id | Unique identifier of the parent resource, if any. | "1754317987" |
| $parent_resource_name | Identifier of the parent resource type, if any. | "APIv4" |
| $parent_resource_type | Identifier of the parent resource name, if any. | "Application" |
| $referer_header | `Referer` header of the page from where the API was called. Returns when the API call is made from a UI. | "https://console.azion.com/applications/edit/1754317987/rules-engine" |
| $remote_port | Source port that generated the request. | "80" |
| $request_data | Data received in the payload of the request generated by the user. | `{"id": 460948, "name": "Debug rule", "active": true, "last_editor": "myemail@gmail.com", "last_modified": "2026-03-26T11:27:10.228491Z"}` |
| $resource_id | Unique identifier of the resource that was created or modified. | "460948" |
| $resource_name | Identifier of the resource name that was created or modified. | "Debug rule" |
| $resource_type | Identifier of the resource type that was created or modified. | "Application Response Rule" |
| $title | Title of the activity, composed of: model name, name, and type of activity. | "Application Response Rule Debug rule was edited" |
| $type | Type of action performed on Azion Console. | "edited" |
| $user_agent | `User-Agent` header sent in the request. | "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" |
| $user_id | Unique identifier of the user who performed the action. | "2696" |
| $user_ip | IP address of the user/source that generated the request. | "179.191.167.238" |
| $time | Request date and time. | "2026-03-26T11:27:11Z" |

---

### Applications

The **Applications** data source provides the data from requests made to your Applications at Azion. The following variables are available for this option.

Learn more about [Applications](/en/documentation/products/build/applications/).

| Variable | Description |
| --- | --- |
| $asn | Autonomous System Number (ASN) Allocation, which are IP address networks managed by one or more network operators that have a clear and unique routing policy. Example: AS52580 |
| $bytes_sent | Number of bytes sent to a client. Example: 191 |
| $client | Unique Azion customer identifier. Example: 4529r |
| $configuration | Unique Azion configuration identifier set on virtual host configuration file. Example: 1595368520 |
| $country | Client's country detected via IP address geolocation. Example: United States |
| $host | Host information sent on the request line. Stores: host name from the request line, *or* host name from the “Host” request header field, *or* the server name matching a request. |
| $http_referrer | Address of the page the user made the request from. Value of the Referer header. Example: https://example.com |
| $http_user_agent | End user's application, operating system, vendor, and/or version. Value of the User-Agent header. Example: Mozilla/5.0 (Windows NT 10.0; Win64; x64) |
| $proxy_status | HTTP error status code or origin when no response is obtained from the upstream. Example: 520. **In case of cache, the response is `-`**. |
| $remote_addr | IP address of the origin that generated the request. |
| $remote_port | Remote port of the origin that generated the request. |
| $request_id | Unique request identifier. Example: 5f222ae5938482c32a822dbf15e19f0f |
| $request_length | Request length, including request line, headers, and body. |
| $request_method | HTTP request method. Example: GET or POST. |
| $request_time | Request processing time, in seconds, since the first bytes were read from the client. Example: `0.234` |
| $request_uri | URI of the request made by the end user, without the host and protocol information and with arguments. Example: /v1?v=bo%20dim |
| $requestPath | Request URI without Query String, Host, and Protocol information. Example: if `request_uri`: /jira/plans/48/scenarios/27?vop=320#plan/backlog, then `requestPath`: /jira/plans/48/scenarios/27 |
| $requestQuery | URI parameters of the request. Example: `requestQuery: vid=320#plan/backlog` |
| $scheme | Request scheme. Example: HTTP or HTTPS. |
| $sent_http_content_type | Content-Type header sent in the origin’s response. Example: text/html; charset=UTF-8. |
| $sent_http_x_original_image_size | “X-Original-Image-Size” header sent in the origin’s response. Used by IMS to inform original image size. Example: 987390 |
| $server_addr | IP address of the server that received the request. |
| $server_port | Remote port of the server that received the request. Example: 443 |
| $server_protocol | Request protocol. Example: `HTTP/1.1`, `HTTP/2.0`, `HTTP/3.0` |
| $session_id | Identification of the session. |
| $ssl_cipher | Cipher string used to establish TLS connection. Example: TLS_AES_256_GCM_SHA384 |
| $ssl_protocol | Protocol for an established TLS connection. Example: TLS v1.2 |
| $ssl_server_name | Server name informed by the client that is trying to connect. Example: www.example.com |
| $ssl_session_reused | Returns `r` if the TLS session was reused; otherwise, returns `.`. |
| $state | Client's state detected via IP address geolocation. Example: CA |
| $status | HTTP status code of the request. Example: 200 |
| $stream | ID set through virtual host configuration based on location directive. Set on virtual host configuration file. |
| $tcpinfo_rtt | Round-Trip Time (RTT) in microseconds measured by the edge for the user. Available on systems that support the TCP_INFO socket option. Example: `72052` |
| $time | Request date and time. Example: Oct. 31st, 2022 - 19:30:41 |
| $traceback | Provides the names of the Rules Engine from your Application and your Firewall that are run by the request. |
| $upstream_addr | Client IP address and port. Can also store multiple servers or server groups. Example: 192.168.1.1:80. When the response is `127.0.0.1:1666`, the upstream is Azion [Cells Runtime](/en/documentation/runtime/overview/). |
| $upstream_bytes_received | Number of bytes received by the origin's edge if the content isn't cached. Example: 8304 |
| $upstream_bytes_sent | Number of bytes sent to the origin. Example: 2733 |
| $upstream_cache_status | Status of the local cache. Example: MISS, BYPASS, EXPIRED, STALE, UPDATING, REVALIDATED, or HIT |
| $upstream_connect_time | Time it takes for the edge to establish a connection with the origin, in seconds. In the case of TLS, it includes time spent on handshake. Example: `0.123`. **Returns `0` for KeepAlive and `-` for cache** |
| $upstream_header_time | Time it takes for the edge to receive the response header from the origin, in seconds. Example: `0.345`. **In case of cache, the response is `-`** |
| $upstream_local_addr | Local IP address used by the edge to establish the connection with the origin server. It represents the outgoing (source) IP used when communicating with the upstream. Example: `10.0.12.34`. |
| $upstream_response_time | Time it takes for the edge to receive a default response from the origin, in seconds, including headers and body. Example: `0.876`. **In case of cache, the response is `-`** |
| $upstream_status | HTTP status code of the origin. If a server cannot be selected, the variable keeps the 502 (Bad Gateway) status code. Example: 200. **In case of cache, the response is `-`**. |
| $waf_attack_action | Reports WAF’s action regarding the action. Can be: $BLOCK, $PASS, $LEARNING_BLOCK, or $LEARNING_PASS. |
| $waf_attack_family | Informs the classification of the WAF infraction detected in the request. Example: SQL, XSS, TRAVERSAL, among others. |
| $waf_block | Informs whether the WAF blocked the action or not. `0` when action *wasn't* blocked and `1` when action *was* blocked. When in *Learning Mode*, it won't be blocked regardless of the return. |
| $waf_headers | When the request headers sent by the user are analyzed by the WAF module and tagged as **blocked** with `$waf_block = 1`, it contains a `base64` encoded string. Otherwise, it contains a dash character `-`. It applies to both *WAF Learning* or *Blocking* modes. |
| $waf_learning | Informs if WAF is in Learning mode. Can be `0` or `1`. |
| $waf_match | List of infractions found in the end user's request. It's formed by key-value elements: the *key* refers to the type of violation detected; the *value* shows the string that generated the infraction. |
| $waf_score | Reports the score that will be increased in case of a match with the rules set for the WAF. |
| $waf_total_blocked | Informs the total number of blocked requests. |
| $waf_total_processed | Informs the total number of processed requests. |

:::note
You can add the `$traceback` variable when using a custom template for the Applications data source if you have the *Debug rules* option activated in your application. See more on [Debugging rules on Application](/en/documentation/products/build/applications/rules-engine/#debugging-rules).
:::

The variables: `$upstream_bytes_received`, `$upstream_cache_status`, `$upstream_connect_time`, `$upstream_header_time`, `$upstream_response_time`, and `$upstream_status` can have *more than one comma-separated element*. When a connection is triggered, either by internal redirection or choice of source with [Load Balancer](/en/documentation/products/secure/connectors/load-balancer/), for example, each value contained in the field represents the respective initiated connection. The field can be separated by:

- A comma, representing multiple IPs.
- A colon, representing internal redirection.

If several servers were contacted during the request processing, their addresses are separated by *commas*. For example: `192.168.1.1:80, 192.168.1.2:80`.

If an internal redirect from one server group to another happens, initiated by *X-Accel-Redirect* or *Error Responses*, then the server addresses from different groups are separated by *colons*. For example: `192.168.1.1:80, 192.168.1.2:80, unix:/tmp/sock : 192.168.10.1:80, 192.168.10.2:80`.

If a server can't be selected, the variable keeps the name of the server group.

Considering multiple values as transitions in the connection, the last value tends to be the most important. If you use the *Error Responses* feature on your Applications, you'll see two values on upstream fields that represent the status of the origin and the result of the request that was made to get the content to be delivered instead. In normal cases, you may get `502 : 200`.

*502* is the HTTP error code for the response of the first try to get content from the origin server. Because it returned an *502* error, considering you have configured an Error Responses for status 502, another request will be made in order to get the URI defined. Then, the page will be delivered and the HTTP status will be added to the *upstream* fields, respecting its position for all of them. In this example, it results in the `502 : 200` composition.

---

### Functions

Requires: <Tag severity="info">Functions</Tag>

The **Functions** data source provides the data from requests made to your [Functions](/en/documentation/products/build/applications/functions/) at Azion.

The following variables are available for this option:

| Variable | Description |
| --- | --- |
| $client | Unique Azion customer identifier. Example: 4529r |
| $edge_function_id | Identification of your Function. Example: 1321 |
| $global_id | Settings identification. |
| $log_level | Level of the log generator: ERROR, WARN, INFO, DEBUG, or TRACE. |
| $log_message | Editable message used in the log function. Available for users to identify and report a given behavior. |
| $message_source | The source of the message. When messages are generated by the Console API: `CONSOLE`; when it's related to an error message: `RUNTIME`. |
| $request_id | Unique request identifier. Example: 5f222ae5938482c32a822dbf15e19f0f |
| $time | Request date and time. Example: Oct. 31st, 2022 - 19:30:41 |

---

### WAF Events

Requires: <Tag severity="info">Web Application Firewall</Tag>

The **WAF Events** data source provides the data from requests analyzed by [Web Application Firewall (WAF)](/en/documentation/products/secure/firewall/web-application-firewall/) to allow you to map the score assigned to the request, the WAF rules that matched, and the reason for the block.

The following variables are available for this option:

| Variable | Description |
| --- | --- |
| $blocked | Informs whether the WAF blocked the action or not. `0` when action *wasn't* blocked and `1` when action *was* blocked. When in *Learning Mode*, it won't be blocked regardless of the return. |
| $client | Unique Azion customer identifier. Example: 4529r |
| $configuration | Unique Azion configuration identifier set on virtual host configuration file. Example: 1595368520 |
| $country | Client's country detected via IP address geolocation. Example: United States |
| $headers | When the request headers sent by the user are analyzed by the WAF module and tagged as **blocked** with `$waf_block = 1`, it contains a `base64` encoded string. Otherwise, it contains a dash character `-`. It applies to both *WAF Learning* or *Blocking* modes. |
| $host | Host information sent on the request line. Stores: host name from the request line, *or* host name from the “Host” request header field, *or* the server name matching a request. |
| $remote_addr | IP address of the origin that generated the request. |
| $requestPath | Request URI without Query String, Host, and Protocol information. Example: if `request_uri`: /jira/plans/48/scenarios/27?vop=320#plan/backlog, then `requestPath`: /jira/plans/48/scenarios/27 |
| $requestQuery | URI parameters of the request. Example: `requestQuery: vid=320#plan/backlog` |
| $server_protocol | Request protocol. Example: `HTTP/1.1`, `HTTP/2.0`, `HTTP/3.0` |
| $time | Request date and time. Example: Oct. 31st, 2022 - 19:30:41 |
| $truncated_body | This variable has been deprecated. It won't have a value assigned to it, only `-` instead of a value. |
| $version | The Azion Log version used. Example: v5. |
| $waf_args | The request arguments. |
| $waf_attack_action | Reports WAF's action regarding the action: $BLOCK, $PASS, $LEARNING_BLOCK, or $LEARNING_PASS. |
| $waf_attack_family | Informs the classification of the WAF infraction detected in the request. Examples: SQL, XSS, TRAVERSAL, among others. |
| $waf_learning | Informs if WAF is in Learning mode. Can be `0` or `1`. |
| $waf_match | List of infractions found in the end user's request. It's formed by key-value elements: the *key* refers to the type of violation detected; the *value* shows the string that generated the infraction. |
| $waf_score | Reports the score that will be increased in case of a match with the rules set for the WAF. |
| $waf_server | Hostname used in the WAF request. Example: api-login.azion.com.br |
| $waf_uri | URI used in the WAF request. Example: /access/v2/after-login |

---

## Templates

A **Data Stream** template provides the preset of variables available for each data source in a format suitable to transfer your event logs. After selecting your data source, you can:

- Select the corresponding template, provided by Azion.
- Customize your own template, choosing which variables you want to use.

You can find four templates provided by Azion and a **Custom Template**, which provides you the option to decide which variables to use. Templates are available in the **Template** dropdown menu and the variables for the templates are shown in the **Data Set** code field in JSON format.

See which template corresponds to which data source:

| Data Source | Template |
| --- | --- |
| Activity History | Activity History Collector |
| Applications | Applications + WAF Event Collector |
| Functions | Functions Event Collector |
| WAF Events | WAF Event Collector |
| All | Custom Template |

By selecting one of the templates provided by Azion, you can't modify the variables shown in the **Data Set** code field. If you select **Custom Template**, you're able to customize which variables you want to use according to your needs.

<DocButton href="/en/documentation/products/guides/data-stream-custom-template/" label="How to create a custom template on Data Stream" kind="secondary" size="medium" />

---

## Domains

You can associate your [existing domains](/en/documentation/products/build/applications/domains/) registered on Azion to your stream. If you haven't registered any domains to your account yet, see the [Creating a new domain associated with your Application](/en/documentation/products/guides/configure-a-domain/) documentation.

When you associate a domain, the events related with that or those specific domains are collected and sent to the endpoint you configure through a Data Stream. You can associate one or more domains and you have the option to **Filter Domains** or select **All Domains**.

When you select **All Domains**, the platform automatically selects all current *and future* domains you have on your Azion Console account.

<DocButton kind="secondary" href="/en/documentation/products/guides/data-stream-associate-domains/" label="How to associate domains on Data Stream" size="medium" />

In case you select the **All Domains** option, you can also set the percentage of data you want to receive randomly from your stream through the *Sampling* option. In addition to filtering by sampling, it can also reduce costs of data collection and analysis.

:::note
Currently, the *Sampling* option isn't available for all Azion Console accounts. If you'd like to use it on your account, [contact the sales team](https://www.azion.com/en/contact/).
:::

The **Sampling (%)** field should contain the percentage of data you want to receive. This percentage will return the total data related to all your domains.

When the Sampling option is enabled, you're allowed to add only *one* stream in your account. Once this stream is disabled, the **Add Streaming** option will be enabled again on the Data Stream screen on Azion Console.

---

## Endpoints

The *endpoint* is the destination where you want to send the data collected by Azion to, which is usually a stream processing platform or analysis tool you use. The endpoint type represents the method you want to configure as a destination to your data.

Azion supports the following endpoints:

- [Implementation](#implementation)
- [How to manage Data Stream](#how-to-manage-data-stream)
- [Logs](#logs)
- [Data sources](#data-sources)
  - [Activity History](#activity-history)
  - [Applications](#applications)
  - [Functions](#functions)
  - [WAF Events](#waf-events)
- [Templates](#templates)
- [Domains](#domains)
- [Endpoints](#endpoints)
  - [Apache Kafka](#apache-kafka)
  - [AWS Kinesis Data Firehose](#aws-kinesis-data-firehose)
  - [Azure Blob Storage](#azure-blob-storage)
  - [Azure Monitor](#azure-monitor)
  - [Datadog](#datadog)
  - [Elasticsearch](#elasticsearch)
  - [Google BigQuery](#google-bigquery)
  - [IBM QRadar](#ibm-qradar)
  - [S3 - Simple Storage Service](#s3---simple-storage-service)
  - [Splunk](#splunk)
  - [Standard HTTP/HTTPS POST](#standard-httphttps-post)
- [Payload](#payload)
  - [Customizable payload with Activity History](#customizable-payload-with-activity-history)
- [Error treatment](#error-treatment)
  - [HTTP 504 Status Behavior](#http-504-status-behavior)
  - [HTTP 503 Status Behavior](#http-503-status-behavior)
- [Limits](#limits)

:::tip
Can't find the endpoint connector you're looking for? Take part in a quick [Data Stream connector survey](https://forms.gle/jhUTEsoqbdn174j59) and share the ones you use and would like to find here.
:::

To configure an endpoint, you must select an **Endpoint Type** from the dropdown list in Azion Console. Then, you must fill the presented fields according to your choice of endpoint type.

See more about each available endpoint and their fields next. Fields marked with an asterisk `*` on Console are mandatory.

---

### Apache Kafka

To configure the [Apache Kafka](https://kafka.apache.org/quickstart) endpoint, you need access to their platform to get the required information:

- **Bootstrap Servers**: the servers—hosts and ports—in the Kafka cluster. You can add one or more Kafka servers by separating them with a comma `,` and *no space*. They must be informed in this format: `myownhost.com:2021,imaginaryhost.com:4525,anotherhost:4030`.

  There’s no need to include all the servers in your cluster in this field, only the few servers that will be used for the initial connection.

:::tip
  It's recommended that you use more than one server to increase redundancy and availability.
:::

- **Kafka Topic**: the topic name from your Kafka cluster to which Data Stream should send messages to. This field only accepts *one* topic. Example: `azure.analytics.fct.pageviews.0`

- **Transport Layer Security (TLS)**: option to send encrypted data using *Transport Layer Security (TLS)*. It's a cryptographic protocol designed to provide secure communication on a computer network and is commonly used as an HTTPS security layer. To use it, select **Yes**.

If you want to use this protocol, make sure the endpoint that will receive the data is properly protected with a digital certificate issued by a globally recognized *Certificate Authority (CA)*, such as IndenTrust, DigiCert, Sectigo, GoDaddy, GlobalSign, or Let’s Encrypt.

The TLS variable `use_tls` receives either `true` or `false` to enable/disable its use.

---

### AWS Kinesis Data Firehose

:::note
When using the AWS Kinesis Data Firehose endpoint, your events are grouped in blocks of up to *500* records, instead of the default 2000, or *every 60 seconds*, whichever occurs first.
:::

To configure the [AWS Kinesis Data Firehose](https://aws.amazon.com/kinesis/data-firehose/) endpoint, you need access to their platform to get the required information:

- **Stream Name**: the delivery stream name that the user defined when they created the Kinesis Data Firehose in AWS's platform. Example: `MyKDFConnector`
- **Region**: the region where your Amazon Kinesis instance is running. Example: `us-east-1`
- **Access Key**: the public key to access the Data Firehose, which is given by AWS. Example: `your-access-key-id`
- **Secret Key**: the secret key to access the Data Firehose, which is given by AWS. Example: `your-secret-access-key`

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-amazon-kinesis/" label="How to use Amazon Kinesis Data Firehose to receive data" size="medium" />


---

### Azure Blob Storage 

To configure the [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/) endpoint, you need access to their platform to get the required information:

- **Storage Account**: the storage account name you defined in Blob Storage. Example: `mystorageaccount`
- **Container Name**: the storage container name you defined in Blob Storage. Example: `mycontainer`
- **Blob SAS Token**: the token generated by Blob Storage. It should have create, read, write, and list accesses granted. Example: `your-sas-token-here`

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-azure-blob/" label="How to use Azure Blob Storage to receive data" size="medium" />

---

### Azure Monitor

To configure the [Azure Monitor](https://azure.microsoft.com/en-us/products/monitor) endpoint, you need access to their platform to get the required information:

- **Log Type**: the record type of the data that's being submitted. It can contain only *letters*, *numbers*, *the underscore (_) character*, and it can't exceed *100 characters*. Example: `AzureMonitorTest`
- **Shared Key**: the Shared Key of the Workspace in Azure Monitor. Example: `your-shared-key-here`
- **Time Generated Field**: used for the TimeGenerated field, which specifies how long it'll take for the log to be available after being collected. If it isn't specified, it uses the ingestion time. Example: `myCustomTimeField`
- **Workspace ID**: the ID of your Workspace in Azure Monitor. Example: `your-workspace-id-here`

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-azure-monitor/" label="How to use Azure Monitor to receive data" size="medium" />

---

### Datadog

To configure the [Datadog](https://www.datadoghq.com/) endpoint, you need access to their platform to get the required information:

- **Datadog URL**: the URL or URI of your Datadog endpoint. Example: `https://inputs.splunk-client.splunkcloud.com:1337/services/collector`
- **API Key**: the API key generated through the Datadog dashboard. Example: `your-datadog-api-key-here`

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-datadog/" label="How to use Datadog to receive data" size="medium" />

---

### Elasticsearch

To configure the [Elasticsearch](https://www.elastic.co/pt/elasticsearch/) endpoint, you need access to their platform to get the required information:

- **Elasticsearch URL**: the URL address + the Elasticsearch index that will receive the collected data. Example: `https://elasticsearch-domain.com/myindex`
- **Encoded API Key**: the `Base64` key corresponding to the `"encoded"` value generated while creating the API Key in ElasticSearch. Example: `your-elasticsearch-encoded-api-key-here`

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-elasticsearch/" label="How to use Elasticsearch to receive data" size="medium" />

---

### Google BigQuery

To configure the [Google BigQuery](https://cloud.google.com/bigquery) endpoint, you need access to their platform to get the required information:

- **Project ID**: your project ID on Google Cloud. Example: `mycustomGBQproject01`
- **Dataset ID**: the name you have to your dataset on Google BigQuery. It's a unique identifier per project and case sensitive. Example: `myGBQdataset`
- **Table ID**: your name that you choose for the table on Google BigQuery. Example: `mypagaviewtable01`
- **Service Account Key**: the JSON file provided by Google Cloud. It has the following format:

```json
{
    "type": "service_account",
    "project_id": "mycustomGBQproject01",
    "private_key_id": "key-id",
    "private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
    "client_email": "service-account-email",
    "client_id": "client-id",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://accounts.google.com/o/oauth2/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}
```

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-google-bigquery/" label="How to use Google BigQuery to receive data" size="medium" />

---

### IBM QRadar

To configure the [IBM Qradar](https://www.ibm.com/qradar) endpoint, you need access to their platform to get the required information:

- **URL**: The URL that will receive the collected data.

---

### S3 - Simple Storage Service

:::note
The S3 credentials used with the endpoint must include the `List Bucket` permission (to list and view objects within a bucket) and the `Put Object` permission (to add or overwrite objects within a bucket), ensuring the necessary access for reading and writing operations in the information storage, including logs. Find out more about [S3 credentials](/en/documentation/products/store/storage/s3-protocol-for-object-storage/).
:::

You can use any type of S3 (Simple Storage Service) provider of your choice, including [Azion Object Storage](/en/documentation/products/store/object-storage/). To configure the S3 endpoint, you need access to the chosen platform to get the required information:

- **Host URL**: the URL of the Host S3. Example: `https://myownhost.s3.us-east-1.myprovider.com`
    - When using the Amazon S3 Storage provider, you can use the default AWS endpoint: [https://s3.amazonaws.com](https://s3.amazonaws.com/).
- **Bucket Name**: the name of the Bucket that the object will be sent to. You define the bucket name. Example: `mys3bucket`
    - The bucket must be created before enabling Data Stream to send the objects.
- **Region**: the region in which your bucket is hosted. Example: `us-east-1`
- **Access Key**: the public key to access your bucket given by your provider. Example: `your-access-key-id`
- **Secret Key**: the secret key to access your bucket given by your provider. Example: `your-secret-access-key`
- **Object Key Prefix**: the prefix for the name of your uploaded object. The file name is composed of the Prefix, the date and time the packets are sent in the YYYY/MM/DD/hh/mm/ format, and the UUID. Example: if you use `user/logs` as the prefix, the sent objects will be saved following this structure `user/logs/2024/10/12/06/24/37d66e78-c308-4006-9d4d-1c013ed89276`.
- **Content Type**: the format in which the object will be created in your bucket. You can chose between *plain/text* or *application/gzip*.

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-amazon-s3/" label="How to use Amazon S3 to receive data" size="medium" />
<DocButton kind="secondary" href="/en/documentation/products/guides/connector-azion-object-storage/" label="How to use Azion Object Storage to receive data" size="medium" />


---

### Splunk

To configure the [Splunk](https://www.splunk.com/) endpoint, you need access to their platform to get the required information:

- **Splunk URL**: the URL that will receive the collected data. If you have an alternative index to point, you can add it at the end of the URL. Example: `https://inputs.splunkcloud.com:8080/services/collector?index=myindex`
- **API Key**: the HTTP Event Collector Token provided during your Splunk installation. Example: `your-splunk-api-key-here`

<DocButton kind="secondary" href="/en/documentation/products/guides/endpoint-splunk/" label="How to use Splunk to receive data" size="medium" />

---

### Standard HTTP/HTTPS POST

:::tip
You can customize [a payload](#payload) when using this endpoint.
:::

To configure the Standard HTTP/HTTPS POST endpoint, you need to get the required information:

- **Endpoint URL**: the URL that will receive the collected data. Example: `https://app.domain.com/`
- **Custom Headers** (*optional*): the *names* and *values* for each header to send to the endpoint. You can enter one or more custom headers for your HTTP/HTTPS request. Example: `header-name:value`

---

## Payload

Requires: <Tag severity="info">Standard HTTP/HTTPS POST endpoint</Tag>

When using the Standard HTTP/HTTPS POST, the payload for your endpoint isn't predefined. You have the option to customize the essential information that will be sent in your data as you best see fit.

To customize the payload sent by the connector in **Data Stream**, you have the following fields:

- **Max Size** (*optional*): defines the size of data packets that will be sent in *bytes*. It accepts values starting from *1000000*.
- **Log Line Separator** (*optional*): defines what information will be used at the end of each log line. Used to break information into different lines.
- **Payload Format** (*optional*): defines which information will be sent in your data, for each Data Stream request.

By default, Data Stream recommends a NDJSON format with the use of `\n` as a log line separator and `$dataset` as a payload format, which uses the information from the **Data Set** code box, describing the variables chosen as a template.

You can choose other options for both fields. Depending on your logs, you can use `,` as a log line separator, for example.

A NDJSON format isn't wrapped with typical JSON arrays `[]`, and each data is presented in a different line, without a comma separating them. It can be useful for structured data with processing of one record at a time. A JSON format, on the other hand, is well known and has tabular data, using arrays `[]` and being separated by commas. With it, the payload can be treated as a single record.

For example, if the **Log Line Separator** receives `,` and the **Payload Format** receives `[$dataset]`, and the template has the following variables in the **Data Set** code box:

$request_method\
$host\
$status

You get a JSON response similar to this:

```json
    [
    	{
			"request_method": "GET",
         	"host": "www.onedomain.com",
          "status": "200"
        },
        {
        	"request_method": "POST",
          	"host": "www.anotherdomain.com.br",
          	"status": "200"
        }
	]
```

But if the **Log Line Separator** receives `\n` and the **Payload Format** receives `$dataset`, you get a NDJSON response similar to this:

```json
{"request_method": "GET", "host": "www.onedomain.com", "status": "200"}
{"request_method": "POST", "host": "www.anotherdomain.com.br", "status": "200"}
```

### Customizable payload with Activity History

You can customize the payload with specific information if you're using the [Activity History](#activity-history) data source and the [Standard HTTP/HTTPS POST](#standard-http-https-post) endpoint.

The following information must be used in the payload fields to configure it:

- **Log Line Separator**: \n
- **Payload Format**: 'v1\t$time_iso8601\t$clientid\t$title\t$comment\t$type\t$author_name\t$author_email'

---

## Error treatment

**Data Stream** servers work in two steps: they monitor the endpoints once a minute (1x/min) and state whether the endpoint is *available* or *unavailable*. The cost of sending messages with an error is very high, so prior monitoring is required.

Data Stream attempts to send messages to endpoints that are *available*. If the endpoint is *unavailable*, messages aren't sent, as the information is discarded. During the next minute, Data Stream sends the data again if the endpoint is considered *available*.

The endpoint must be approved by all Azion servers to be considered as *available*. If one of the servers indicates that the endpoint is *unavailable*, messages aren't sent.

### HTTP 504 Status Behavior

Occurs when the endpoint responds to the test successfully, but doesn't receive messages within the timeout: *20 seconds* for HTTP POST type endpoints.

### HTTP 503 Status Behavior

Occurs when the endpoint is declared *unavailable* by endpoint monitoring. Therefore, Data Stream doesn't attempt to send the messages.

Since the system is distributed, it isn't possible to know the specific server that sends messages to each endpoint.

---

## Limits

:::tip
**Increase limits** <br></br>
You can request to increase the limits based on your plan. Contact the [technical support team](/en/documentation/services/support/) to request it.
:::

These are the **default limits**:

| Scope | Limit |
| ------- | ----- |
| Data Stream | 60 seconds or packages of 2,000 requests, whichever happens first.<br /><br />For the **AWS Kinesis Data Firehose** endpoint: 60 seconds or packages of 500 requests |
| Sampling | 1 active stream |
| Time for data transfer to client's infrastructure | Up to 3 minutes |

<br /><br />