Data Stream

Data Stream is an Observe product that allows you to feed your stream processing, SIEM, and big data platforms with the event logs from your applications on Azion in real time. To keep an enhanced performance, it uses ASCII encoding to avoid parser issues and problems in data interpretation.

Data Stream is responsible for collecting, grouping, and transmitting raw logs to your chosen destination, providing flexibility for data integration. You can choose from which of the available Azion product and domains you want to collect your logs from and connect them to the endpoint of your data analysis platforms. You can also decide which data you want to use on your analysis by choosing among the available variables.

By accessing Data Stream through Azion Console or Azion API, you can perform a variety of actions to manage your data flows, including:

• Have an organized set of logs.
• Create a new data stream.
• Connect your Data Streams to endpoints.
• Edit data stream settings directly from the list by selecting the stream you want to update.
• View detailed information about each stream to ensure everything is running smoothly.
• Delete or stop data streams that are no longer needed to optimize your system.
• Understand the behavior of your users.
• Analyze the performance of your content and applications.
• Identify data on security threats.
• Make informed decisions.
• Improve your applications and your business through reliable observability practices.

After configuring your Data Stream, you can check your logs successfully being sent through Real-Time Events.

Implementation

How to manage Data Stream

Azion Data Stream allows you create, edit, view, and delete data stream configurations.

You can create a new data stream through Azion Console:

1. Go to Product menu > Data Stream.
2. Click the + Stream button.
3. Complete the configuration.
4. Click the Save button.

To edit any stream, go to the Data Stream page and select the stream you want to work with. Edit the configurations as desired and save the changes.

If you want to stop a data stream in your Azion account, follow these steps:

1. Access Azion Console > Data Stream.
2. In the list, select the stream you want to stop.

• On the right side of the row, you can see a label indicating the stream’s status: Active or Inactive.

3. On the configuration page, navigate to the Status section.
4. Disable the switch to stop the data streaming.
5. Click the Save button.

To activate the streaming again, repeat this process, enabling the switch in the Status section.

To delete a stream: go to the Data Stream page, navigate to the right side of the row, and click the trash icon for the stream you want to delete.

Logs

By default, Data Stream sends your event logs when the code block with variables reaches 2,000 records, every 60 seconds, or when the packet size reaches the value determined in the payload Max Size field, whichever occurs first. However, in case you’re using the AWS Kinesis Firehose endpoint, Data Stream will send your event logs when the block reaches 500 records or every 60 seconds.

For example, if a package reaches 2,000 records in 13 seconds, it’ll be sent.

Once you activate a stream, there’s a propagation time until the logs become available for consultation in tools or products such as Real-Time Events.

To find detailed reference about each section of configuring a Data Stream, keep reading the following subsections.

Data sources

A Data Source represents the application at Azion that generates the event logs you want to use. By selecting one, you decide where your data will be collected from and the remaining product settings are configured according to your choice.

Selecting a Data Source in the dropdown list is mandatory. You can choose between:

• Activity History
• Applications
• Functions
• WAF Events

Each data source has a preset of variables, combined in a template, representing the specific information you can receive from your event logs. See each data source’s prerequisites and variables and what data they provide next.

Activity History

The Activity History data source displays the data for logs activity regarding your account on Azion Console. The following variables are available for this option:

Applications

The Applications data source provides the data from requests made to your Applications at Azion. The following variables are available for this option.

Learn more about Applications.

The variables: $upstream_bytes_received, $upstream_cache_status, $upstream_connect_time, $upstream_header_time, $upstream_response_time, and $upstream_status can have more than one comma-separated element. When a connection is triggered, either by internal redirection or choice of source with Load Balancer, for example, each value contained in the field represents the respective initiated connection. The field can be separated by:

• A comma, representing multiple IPs.
• A colon, representing internal redirection.

If several servers were contacted during the request processing, their addresses are separated by commas. For example: 192.168.1.1:80, 192.168.1.2:80.

If an internal redirect from one server group to another happens, initiated by X-Accel-Redirect or Error Responses, then the server addresses from different groups are separated by colons. For example: 192.168.1.1:80, 192.168.1.2:80, unix:/tmp/sock : 192.168.10.1:80, 192.168.10.2:80.

If a server can’t be selected, the variable keeps the name of the server group.

Considering multiple values as transitions in the connection, the last value tends to be the most important. If you use the Error Responses feature on your Applications, you’ll see two values on upstream fields that represent the status of the origin and the result of the request that was made to get the content to be delivered instead. In normal cases, you may get 502 : 200.

502 is the HTTP error code for the response of the first try to get content from the origin server. Because it returned an 502 error, considering you have configured an Error Responses for status 502, another request will be made in order to get the URI defined. Then, the page will be delivered and the HTTP status will be added to the upstream fields, respecting its position for all of them. In this example, it results in the 502 : 200 composition.

Functions

Requires: Functions

The Functions data source provides the data from requests made to your Functions at Azion.

The following variables are available for this option:

WAF Events

Requires: Web Application Firewall

The WAF Events data source provides the data from requests analyzed by Web Application Firewall (WAF) to allow you to map the score assigned to the request, the WAF rules that matched, and the reason for the block.

The following variables are available for this option:

Templates

A Data Stream template provides the preset of variables available for each data source in a format suitable to transfer your event logs. After selecting your data source, you can:

• Select the corresponding template, provided by Azion.
• Customize your own template, choosing which variables you want to use.

You can find four templates provided by Azion and a Custom Template, which provides you the option to decide which variables to use. Templates are available in the Template dropdown menu and the variables for the templates are shown in the Data Set code field in JSON format.

See which template corresponds to which data source:

By selecting one of the templates provided by Azion, you can’t modify the variables shown in the Data Set code field. If you select Custom Template, you’re able to customize which variables you want to use according to your needs.

Domains

You can associate your existing domains registered on Azion to your stream. If you haven’t registered any domains to your account yet, see the Creating a new domain associated with your Application documentation.

When you associate a domain, the events related with that or those specific domains are collected and sent to the endpoint you configure through a Data Stream. You can associate one or more domains and you have the option to Filter Domains or select All Domains.

When you select All Domains, the platform automatically selects all current and future domains you have on your Azion Console account.

In case you select the All Domains option, you can also set the percentage of data you want to receive randomly from your stream through the Sampling option. In addition to filtering by sampling, it can also reduce costs of data collection and analysis.

The Sampling (%) field should contain the percentage of data you want to receive. This percentage will return the total data related to all your domains.

When the Sampling option is enabled, you’re allowed to add only one stream in your account. Once this stream is disabled, the Add Streaming option will be enabled again on the Data Stream screen on Azion Console.

Endpoints

The endpoint is the destination where you want to send the data collected by Azion to, which is usually a stream processing platform or analysis tool you use. The endpoint type represents the method you want to configure as a destination to your data.

Azion supports the following endpoints:

• Implementation
• How to manage Data Stream
• Logs
• Data sources
  • Activity History
  • Applications
  • Functions
  • WAF Events
• Templates
• Domains
• Endpoints
  • Apache Kafka
  • AWS Kinesis Data Firehose
  • Azure Blob Storage
  • Azure Monitor
  • Datadog
  • Elasticsearch
  • Google BigQuery
  • IBM QRadar
  • S3 - Simple Storage Service
  • Splunk
  • Standard HTTP/HTTPS POST
• Payload
  • Customizable payload with Activity History
• Error treatment
  • HTTP 504 Status Behavior
  • HTTP 503 Status Behavior
• Limits

To configure an endpoint, you must select an Endpoint Type from the dropdown list in Azion Console. Then, you must fill the presented fields according to your choice of endpoint type.

See more about each available endpoint and their fields next. Fields marked with an asterisk * on Console are mandatory.

Apache Kafka

To configure the Apache Kafka endpoint, you need access to their platform to get the required information:

• Bootstrap Servers: the servers—hosts and ports—in the Kafka cluster. You can add one or more Kafka servers by separating them with a comma , and no space. They must be informed in this format: myownhost.com:2021,imaginaryhost.com:4525,anotherhost:4030.

  There’s no need to include all the servers in your cluster in this field, only the few servers that will be used for the initial connection.

• Kafka Topic: the topic name from your Kafka cluster to which Data Stream should send messages to. This field only accepts one topic. Example: azure.analytics.fct.pageviews.0

• Transport Layer Security (TLS): option to send encrypted data using Transport Layer Security (TLS). It’s a cryptographic protocol designed to provide secure communication on a computer network and is commonly used as an HTTPS security layer. To use it, select Yes.

If you want to use this protocol, make sure the endpoint that will receive the data is properly protected with a digital certificate issued by a globally recognized Certificate Authority (CA), such as IndenTrust, DigiCert, Sectigo, GoDaddy, GlobalSign, or Let’s Encrypt.

The TLS variable use_tls receives either true or false to enable/disable its use.

AWS Kinesis Data Firehose

To configure the AWS Kinesis Data Firehose endpoint, you need access to their platform to get the required information:

• Stream Name: the delivery stream name that the user defined when they created the Kinesis Data Firehose in AWS’s platform. Example: MyKDFConnector
• Region: the region where your Amazon Kinesis instance is running. Example: us-east-1
• Access Key: the public key to access the Data Firehose, which is given by AWS. Example: your-access-key-id
• Secret Key: the secret key to access the Data Firehose, which is given by AWS. Example: your-secret-access-key

Azure Blob Storage

To configure the Azure Blob Storage endpoint, you need access to their platform to get the required information:

• Storage Account: the storage account name you defined in Blob Storage. Example: mystorageaccount
• Container Name: the storage container name you defined in Blob Storage. Example: mycontainer
• Blob SAS Token: the token generated by Blob Storage. It should have create, read, write, and list accesses granted. Example: your-sas-token-here

Azure Monitor

To configure the Azure Monitor endpoint, you need access to their platform to get the required information:

• Log Type: the record type of the data that’s being submitted. It can contain only letters, numbers, the underscore (_) character, and it can’t exceed 100 characters. Example: AzureMonitorTest
• Shared Key: the Shared Key of the Workspace in Azure Monitor. Example: your-shared-key-here
• Time Generated Field: used for the TimeGenerated field, which specifies how long it’ll take for the log to be available after being collected. If it isn’t specified, it uses the ingestion time. Example: myCustomTimeField
• Workspace ID: the ID of your Workspace in Azure Monitor. Example: your-workspace-id-here

Datadog

To configure the Datadog endpoint, you need access to their platform to get the required information:

• Datadog URL: the URL or URI of your Datadog endpoint. Example: https://inputs.splunk-client.splunkcloud.com:1337/services/collector
• API Key: the API key generated through the Datadog dashboard. Example: your-datadog-api-key-here

Elasticsearch

To configure the Elasticsearch endpoint, you need access to their platform to get the required information:

• Elasticsearch URL: the URL address + the Elasticsearch index that will receive the collected data. Example: https://elasticsearch-domain.com/myindex
• Encoded API Key: the Base64 key corresponding to the "encoded" value generated while creating the API Key in ElasticSearch. Example: your-elasticsearch-encoded-api-key-here

Google BigQuery

To configure the Google BigQuery endpoint, you need access to their platform to get the required information:

• Project ID: your project ID on Google Cloud. Example: mycustomGBQproject01
• Dataset ID: the name you have to your dataset on Google BigQuery. It’s a unique identifier per project and case sensitive. Example: myGBQdataset
• Table ID: your name that you choose for the table on Google BigQuery. Example: mypagaviewtable01
• Service Account Key: the JSON file provided by Google Cloud. It has the following format:

IBM QRadar

To configure the IBM Qradar endpoint, you need access to their platform to get the required information:

• URL: The URL that will receive the collected data.

S3 - Simple Storage Service

You can use any type of S3 (Simple Storage Service) provider of your choice, including Azion Object Storage. To configure the S3 endpoint, you need access to the chosen platform to get the required information:

• Host URL: the URL of the Host S3. Example: https://myownhost.s3.us-east-1.myprovider.com
  • When using the Amazon S3 Storage provider, you can use the default AWS endpoint: https://s3.amazonaws.com.
• Bucket Name: the name of the Bucket that the object will be sent to. You define the bucket name. Example: mys3bucket
  • The bucket must be created before enabling Data Stream to send the objects.
• Region: the region in which your bucket is hosted. Example: us-east-1
• Access Key: the public key to access your bucket given by your provider. Example: your-access-key-id
• Secret Key: the secret key to access your bucket given by your provider. Example: your-secret-access-key
• Object Key Prefix: the prefix for the name of your uploaded object. The file name is composed of the Prefix, the date and time the packets are sent in the YYYY/MM/DD/hh/mm/ format, and the UUID. Example: if you use user/logs as the prefix, the sent objects will be saved following this structure user/logs/2024/10/12/06/24/37d66e78-c308-4006-9d4d-1c013ed89276.
• Content Type: the format in which the object will be created in your bucket. You can chose between plain/text or application/gzip.

Splunk

To configure the Splunk endpoint, you need access to their platform to get the required information:

• Splunk URL: the URL that will receive the collected data. If you have an alternative index to point, you can add it at the end of the URL. Example: https://inputs.splunkcloud.com:8080/services/collector?index=myindex
• API Key: the HTTP Event Collector Token provided during your Splunk installation. Example: your-splunk-api-key-here

Standard HTTP/HTTPS POST

To configure the Standard HTTP/HTTPS POST endpoint, you need to get the required information:

• Endpoint URL: the URL that will receive the collected data. Example: https://app.domain.com/
• Custom Headers (optional): the names and values for each header to send to the endpoint. You can enter one or more custom headers for your HTTP/HTTPS request. Example: header-name:value

Payload

Requires: Standard HTTP/HTTPS POST endpoint

When using the Standard HTTP/HTTPS POST, the payload for your endpoint isn’t predefined. You have the option to customize the essential information that will be sent in your data as you best see fit.

To customize the payload sent by the connector in Data Stream, you have the following fields:

• Max Size (optional): defines the size of data packets that will be sent in bytes. It accepts values starting from 1000000.
• Log Line Separator (optional): defines what information will be used at the end of each log line. Used to break information into different lines.
• Payload Format (optional): defines which information will be sent in your data, for each Data Stream request.

By default, Data Stream recommends a NDJSON format with the use of \n as a log line separator and $dataset as a payload format, which uses the information from the Data Set code box, describing the variables chosen as a template.

You can choose other options for both fields. Depending on your logs, you can use , as a log line separator, for example.

A NDJSON format isn’t wrapped with typical JSON arrays [], and each data is presented in a different line, without a comma separating them. It can be useful for structured data with processing of one record at a time. A JSON format, on the other hand, is well known and has tabular data, using arrays [] and being separated by commas. With it, the payload can be treated as a single record.

For example, if the Log Line Separator receives , and the Payload Format receives [$dataset], and the template has the following variables in the Data Set code box:

$request_method
$host
$status

You get a JSON response similar to this:

But if the Log Line Separator receives \n and the Payload Format receives $dataset, you get a NDJSON response similar to this:

Customizable payload with Activity History

You can customize the payload with specific information if you’re using the Activity History data source and the Standard HTTP/HTTPS POST endpoint.

The following information must be used in the payload fields to configure it:

• Log Line Separator: \n
• Payload Format: ‘v1\t$time_iso8601\t$clientid\t$title\t$comment\t$type\t$author_name\t$author_email’

Error treatment

Data Stream servers work in two steps: they monitor the endpoints once a minute (1x/min) and state whether the endpoint is available or unavailable. The cost of sending messages with an error is very high, so prior monitoring is required.

Data Stream attempts to send messages to endpoints that are available. If the endpoint is unavailable, messages aren’t sent, as the information is discarded. During the next minute, Data Stream sends the data again if the endpoint is considered available.

The endpoint must be approved by all Azion servers to be considered as available. If one of the servers indicates that the endpoint is unavailable, messages aren’t sent.

HTTP 504 Status Behavior

Occurs when the endpoint responds to the test successfully, but doesn’t receive messages within the timeout: 20 seconds for HTTP POST type endpoints.

HTTP 503 Status Behavior

Occurs when the endpoint is declared unavailable by endpoint monitoring. Therefore, Data Stream doesn’t attempt to send the messages.

Since the system is distributed, it isn’t possible to know the specific server that sends messages to each endpoint.

Limits

These are the default limits: