# Send Data Stream Data to Google BigQuery

import Tabs from '~/components/tabs/Tabs'
import Code from '~/components/Code/Code.astro'

While configuring [streams](/en/documentation/products/observe/data-stream/), you need to set up a specific endpoint to stream your Azion data.

Continue reading for a step by step on how to connect a [Google BigQuery](https://cloud.google.com/bigquery/) endpoint to receive data from Data Stream.

---

## Google BigQuery requirements

To get started with **Google BigQuery**, you must follow a few steps:

1. Create [a Google Cloud Platform](https://cloud.google.com/) account.
2. Create [a project](https://developers.google.com/workspace/guides/create-project) on the Google Cloud Platform. The **Project ID** will be given while you create it, make sure to save it.
3. Create a [service account](https://cloud.google.com/iam/docs/service-accounts-create) on the Google Platform.
4. The service account must have the **BigQuery Admin** permissions. Make sure you select that option in the **Role** dropdown list.

> For more details about the standard permissions assigned to the role of data editor of BigQuery, access [BigQuery Roles](https://cloud.google.com/iam/docs/understanding-roles#bigquery-roles).

You'll also need to create and configure the following information:

- Enable the BigQuery API
- A dataset
- A service account key

### Enabling BigQuery API

Next, you must access the [API Manager](https://console.cloud.google.com/apis/library) and enable the **BigQuery API**.

The BigQuery API supports an endpoint to stream rows into a table. However, this endpoint isn't supported in the Free Tier version. To use it, it's necessary to enable the full version of the platform with the billing configuration.

Find more details on this step in the [documentation of Billing management on projects](https://cloud.google.com/billing/docs/how-to/modify-project). You can consult the fees for this API in the **Streaming Inserts** section's price table.

### Creating a dataset

After enabling the API, you'll need to create a dataset. To do so, you must first have created a project in the [Google Cloud Console](https://console.cloud.google.com/bigquery). By default, BigQuery is already enabled in new projects.

After creating the project, follow these steps:

1. On your Google Cloud console, open the BigQuery page.
2. Select the project you want to create a dataset on.
3. Click the **Actions** option, with the vertical ellipsis > **Create dataset**.
4. Fill in the [required information](https://cloud.google.com/bigquery/docs/datasets#create-dataset). After choosing a **Dataset ID**, make sure to save it.
5. Click **Create dataset**.

After creating a dataset, you must create a table:

1. [Create a table](https://cloud.google.com/bigquery/docs/tables) and associate it to the dataset you've just created.
2. Make sure you save the **Table** name you choose. That's the Table ID.
3. On **Schema**, add the structure of the data that will be inserted.
4. Click **Create table**.

Once you create the table, it's possible to ingest data through the BigQuery API.

### Creating a service account key

Next, you must create a private key to continue your configuration.

1. After creating the service account, access your service account.
2. On the left menu, click **IAM & Admin** > **Service Accounts**.
3. Select the service account you've created from the list.
4. Click on the **KEYS** tab on the upper menu > **ADD KEY** > **Create new Key**.
5. On **Key Type**, choose the **JSON** option > **CREATE**.
6. After the confirmation, a **JSON file** will be automatically downloaded with the credentials.

The file's content should look similar to this:

```json
{
  "type": "service_account",
  "project_id": "azion-data",
  "private_key_id": "-----BEGIN RSA PRIVATE KEY-----\n<your_private_key>\n-----END RSA PRIVATE KEY-----",
  "private_key": "-----BEGIN PRIVATE KEY-----\\nxxx\\n-----END PRIVATE KEY-----\\n",
  "client_email": "myemail@azion.com",
   ...
}
```

---

## Configuring the endpoint in Data Stream

Next, follow these steps to configure the new endpoint you created in **Google BigQuery** in your Azion **Data Stream**.

You can find detailed steps for the entire configuration on the [How to use Data Stream guide](/en/documentation/products/guides/use-data-stream/).

:::note
All `key` fields will be masked and an icon that allows revealing the field content will be only displayed to users with the `Edit Data Stream` permission. Users without the `Edit Data Stream` permission can only view the settings of the existing stream but not edit or create new ones.
:::

<Tabs client:visible>
    <Fragment slot="tab.console">Console</Fragment>
    <Fragment slot="tab.rtm">Real-Time Manager</Fragment>

<Fragment slot="panel.console">

In the **Destination** configurations:

1. On the **Connector** dropdown menu, select **Google BigQuery**.
2. On **Project ID**, add the ID of your project in Google Cloud.
3. On **Dataset ID**, add the ID of your dataset created on Google BigQuery.
4. On **Table ID**, add the ID of your table that will receive the streamed data.
5. On **Service Account Key**, paste the content of the downloaded `JSON` file that contains your private access key.

The private access key should look similar to this:

<Code lang="json" code={`
{
	"type": "service_account",
	"project_id": "[name]",
	"private_key_id": "key-id",
	"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
	"client_email": "email@myemailaccount.com",
	"client_id": "client-id",
	"auth_uri": "https://accounts.google.com/o/oauth2/auth",
	"token_uri": "https://oauth2.googleapis.com/token",
	"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
	"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/mytest.com"
}
`} />

6. Make sure the **Active** switch is on.
7. Click the **Save** button.

</Fragment>

<Fragment slot="panel.rtm">

In the **Destination** configurations:

1. On the **Endpoint Type** dropdown menu, select **Google BigQuery**.
2. On **Project ID**, add the ID of your project in Google Cloud.
3. On **Dataset ID**, add the ID of your dataset created on Google BigQuery.
4. On **Table ID**, add the ID of your table that will receive the streamed data.
5. On **Service Account Key**, paste the content of the downloaded `JSON` file that contains your private access key.

The private access key should look similar to this:

<Code lang="json" code={`
{
	"type": "service_account",
	"project_id": "[name]",
	"private_key_id": "key-id",
	"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
	"client_email": "email@myemailaccount.com",
	"client_id": "client-id",
	"auth_uri": "https://accounts.google.com/o/oauth2/auth",
	"token_uri": "https://oauth2.googleapis.com/token",
	"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
	"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/mytest.com"
}
`} />

6. Make sure the **Active** switch is on.
7. Click the **Save** button.

</Fragment>
</Tabs>

All authentication with Google Oauth2 and generation of JWTokens will be performed by the Data Stream backend systems.

After saving the configurations, your data will be streamed to the newly configured endpoint.

You can keep track of the calls made by Data Stream to Google BigQuery on [Real-Time Events](/en/documentation/products/observe/real-time-events/). To do so, select **Data Source** > **Data Stream** and choose the filters options as you wish.



---

### Trademarks

[Google BigQuery](https://cloud.google.com/bigquery) and [Google Cloud Platform](https://cloud.google.com/) are registered trademarks of Google LLC in the US and/or other countries.