Send Data Stream Data to Google BigQuery

Hands-on to use Google BigQuery endpoint connector to receive data from Data Stream.

View as Markdown

While configuring streams, you need to set up a specific endpoint to stream your Azion data.

Continue reading for a step by step on how to connect a Google BigQuery endpoint to receive data from Data Stream.

Google BigQuery requirements

To get started with Google BigQuery, you must follow a few steps:

  1. Create a Google Cloud Platform account.
  2. Create a project on the Google Cloud Platform. The Project ID will be given while you create it, make sure to save it.
  3. Create a service account on the Google Platform.
  4. The service account must have the BigQuery Admin permissions. Make sure you select that option in the Role dropdown list.

For more details about the standard permissions assigned to the role of data editor of BigQuery, access BigQuery Roles.

You’ll also need to create and configure the following information:

  • Enable the BigQuery API
  • A dataset
  • A service account key

Enabling BigQuery API

Next, you must access the API Manager and enable the BigQuery API.

The BigQuery API supports an endpoint to stream rows into a table. However, this endpoint isn’t supported in the Free Tier version. To use it, it’s necessary to enable the full version of the platform with the billing configuration.

Find more details on this step in the documentation of Billing management on projects. You can consult the fees for this API in the Streaming Inserts section’s price table.

Creating a dataset

After enabling the API, you’ll need to create a dataset. To do so, you must first have created a project in the Google Cloud Console. By default, BigQuery is already enabled in new projects.

After creating the project, follow these steps:

  1. On your Google Cloud console, open the BigQuery page.
  2. Select the project you want to create a dataset on.
  3. Click the Actions option, with the vertical ellipsis > Create dataset.
  4. Fill in the required information. After choosing a Dataset ID, make sure to save it.
  5. Click Create dataset.

After creating a dataset, you must create a table:

  1. Create a table and associate it to the dataset you’ve just created.
  2. Make sure you save the Table name you choose. That’s the Table ID.
  3. On Schema, add the structure of the data that will be inserted.
  4. Click Create table.

Once you create the table, it’s possible to ingest data through the BigQuery API.

Creating a service account key

Next, you must create a private key to continue your configuration.

  1. After creating the service account, access your service account.
  2. On the left menu, click IAM & Admin > Service Accounts.
  3. Select the service account you’ve created from the list.
  4. Click on the KEYS tab on the upper menu > ADD KEY > Create new Key.
  5. On Key Type, choose the JSON option > CREATE.
  6. After the confirmation, a JSON file will be automatically downloaded with the credentials.

The file’s content should look similar to this:

{
"type": "service_account",
"project_id": "azion-data",
"private_key_id": "-----BEGIN RSA PRIVATE KEY-----\n<your_private_key>\n-----END RSA PRIVATE KEY-----",
"private_key": "-----BEGIN PRIVATE KEY-----\\nxxx\\n-----END PRIVATE KEY-----\\n",
"client_email": "myemail@azion.com",
...
}

Configuring the endpoint in Data Stream

Next, follow these steps to configure the new endpoint you created in Google BigQuery in your Azion Data Stream.

You can find detailed steps for the entire configuration on the How to use Data Stream guide.

All key fields will be masked and an icon that allows revealing the field content will be only displayed to users with the Edit Data Stream permission. Users without the Edit Data Stream permission can only view the settings of the existing stream but not edit or create new ones.

In the Destination configurations:

  1. On the Connector dropdown menu, select Google BigQuery.
  2. On Project ID, add the ID of your project in Google Cloud.
  3. On Dataset ID, add the ID of your dataset created on Google BigQuery.
  4. On Table ID, add the ID of your table that will receive the streamed data.
  5. On Service Account Key, paste the content of the downloaded JSON file that contains your private access key.

The private access key should look similar to this:

{
"type": "service_account",
"project_id": "[name]",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY-----
private-key
-----END PRIVATE KEY-----
",
"client_email": "email@myemailaccount.com",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/mytest.com"
}
  1. Make sure the Active switch is on.
  2. Click the Save button.

All authentication with Google Oauth2 and generation of JWTokens will be performed by the Data Stream backend systems.

After saving the configurations, your data will be streamed to the newly configured endpoint.

You can keep track of the calls made by Data Stream to Google BigQuery on Real-Time Events. To do so, select Data Source > Data Stream and choose the filters options as you wish.

Trademarks

Google BigQuery and Google Cloud Platform are registered trademarks of Google LLC in the US and/or other countries.