Skip to main content

BigQuery Integration

Export Sealmetrics analytics data to Google BigQuery for advanced analysis, custom reporting, and data warehousing.


Overview

The BigQuery integration allows you to:

  • Automatically sync analytics data to your BigQuery dataset
  • Run custom SQL queries on your traffic and conversion data
  • Integrate with BI tools like Looker, Data Studio, or Tableau
  • Perform advanced analysis not available in the dashboard

Base path: /integrations/bigquery

note

BigQuery integration is available on all plans (Growth, Scale, and Enterprise).


Setup Flow

  1. Create a Google Cloud service account with BigQuery permissions
  2. Configure the integration by uploading the service account JSON file
  3. The dataset and tables are created automatically during setup
  4. Enable automatic sync or trigger manual/backfill syncs

Configure Integration

Create Integration

POST /integrations/bigquery?account_id={account_id}

This endpoint uses query parameters for configuration and a multipart/form-data body to upload the Google Cloud service account JSON file. The dataset and tables are created in BigQuery as part of this call.

Query Parameters:

ParameterTypeRequiredDescription
account_idstringYesAccount ID
gcp_project_idstringYesGoogle Cloud project ID (6-30 chars, lowercase letters, digits and hyphens, starts with a letter)
dataset_idstringNoBigQuery dataset name (alphanumeric and underscores). Default: sealmetrics
locationenumNoDataset location: EU (default), US, europe-west1, us-central1, us-east1
sync_frequencyenumNohourly, daily (default), manual
data_typesstring[]NoData types to sync. Default: traffic, conversions, microconversions, pages, landing_pages. Also valid: traffic_hourly, accounts
backfill_daysintegerNoDays of historical data to sync initially (0-365). Default: 30
initial_date_fromdateNoCustom initial sync start date (overrides backfill_days; must be set with initial_date_to)
initial_date_todateNoCustom initial sync end date (must be set with initial_date_from)

Form Data (multipart):

FieldTypeRequiredDescription
service_account_filefileYesGoogle Cloud service account JSON file (must end in .json and have "type": "service_account")
note

A custom date range (initial_date_from/initial_date_to) cannot exceed 365 days, and initial_date_to cannot be in the future.

Example request:

curl -X POST "https://api.sealmetrics.com/api/v1/integrations/bigquery?account_id=my-account&gcp_project_id=my-analytics-project&dataset_id=sealmetrics&location=EU&sync_frequency=daily&data_types=traffic&data_types=conversions&backfill_days=30" \
-H "X-API-Key: sm_your_api_key" \
-F "service_account_file=@service-account.json"

Response (201 Created):

{
"integration": {
"id": 1,
"account_id": "my-account",
"gcp_project_id": "my-analytics-project",
"dataset_id": "sealmetrics",
"location": "EU",
"sync_frequency": "daily",
"data_types": ["traffic", "conversions", "microconversions", "pages", "landing_pages"],
"backfill_days": 30,
"initial_date_from": null,
"initial_date_to": null,
"is_active": true,
"setup_completed": true,
"setup_completed_at": "2025-01-10T14:30:05Z",
"tables_created": ["fact_traffic_daily", "fact_conversions", "fact_microconversions", "fact_pages", "fact_landing_pages", "sync_metadata", "dim_countries"],
"last_sync_at": null,
"last_sync_status": null,
"last_error": null,
"created_at": "2025-01-10T14:30:00Z",
"updated_at": "2025-01-10T14:30:05Z",
"created_by": 42
},
"dataset_ref": "my-analytics-project.sealmetrics",
"tables_created": ["fact_traffic_daily", "fact_conversions", "fact_microconversions", "fact_pages", "fact_landing_pages", "sync_metadata", "dim_countries"],
"message": "BigQuery integration configured. Dataset: my-analytics-project.sealmetrics"
}
note

Data is not automatically synced after setup. Use the sync endpoint or wait for a scheduled sync.

Get Integration

GET /integrations/bigquery?account_id={account_id}

Returns null if no integration is configured.

Response:

{
"id": 1,
"account_id": "my-account",
"gcp_project_id": "my-analytics-project",
"dataset_id": "sealmetrics",
"location": "EU",
"sync_frequency": "daily",
"data_types": ["traffic", "conversions", "microconversions", "pages", "landing_pages"],
"backfill_days": 30,
"initial_date_from": null,
"initial_date_to": null,
"is_active": true,
"setup_completed": true,
"setup_completed_at": "2025-01-01T10:00:05Z",
"tables_created": ["fact_traffic_daily", "fact_conversions", "fact_microconversions", "fact_pages", "fact_landing_pages", "sync_metadata", "dim_countries"],
"last_sync_at": "2025-01-10T03:00:00Z",
"last_sync_status": "success",
"last_error": null,
"created_at": "2025-01-01T10:00:00Z",
"updated_at": "2025-01-10T03:00:00Z",
"created_by": 42
}

Update Integration

PATCH /integrations/bigquery?account_id={account_id}

Request Body (JSON):

{
"sync_frequency": "hourly",
"data_types": ["traffic", "conversions", "microconversions", "pages"],
"is_active": true,
"backfill_days": 60
}
FieldTypeDescription
sync_frequencyenumhourly, daily, manual
data_typesstring[]Which data to sync (at least one)
is_activebooleanEnable/disable sync
backfill_daysintegerHistorical data range (0-365)

All fields are optional. Only provided fields are updated. If data_types changes, any new tables are created in BigQuery automatically.

Delete Integration

DELETE /integrations/bigquery?account_id={account_id}

Removes the integration configuration. Data already synced to BigQuery is not deleted — you retain full ownership of your BigQuery data.

Response: 204 No Content


Initial Setup

The dataset and tables are created automatically when you create the integration. You can also (re)create the BigQuery dataset and tables explicitly using the stored credentials — useful if the integration was configured but the resources need to be recreated.

POST /integrations/bigquery/setup?account_id={account_id}

This endpoint:

  1. Creates the dataset if it doesn't exist
  2. Creates all tables based on the configured data_types
  3. Always creates the sync_metadata table (for transparency) and the dim_countries lookup table

Response:

{
"status": "success",
"dataset_ref": "my-analytics-project.sealmetrics",
"tables_created": ["fact_traffic_daily", "fact_conversions", "fact_microconversions", "fact_pages", "fact_landing_pages", "sync_metadata", "dim_countries"],
"location": "EU",
"message": "BigQuery dataset and tables created successfully"
}

Sync Operations

Trigger Manual Sync

POST /integrations/bigquery/sync?account_id={account_id}

Trigger an immediate incremental sync of recent data. The date range is passed as query parameters (there is no request body).

Query ParameterTypeDescription
date_fromdateStart date (default: last sync minus 2 days, or backfill_days ago)
date_todateEnd date (default: today)

Response:

{
"status": "success",
"sync_id": "550e8400-e29b-41d4-a716-446655440000",
"sync_type": "incremental",
"date_from": "2025-01-01",
"date_to": "2025-01-10",
"tables": {
"fact_traffic_daily": {"rows": 12000, "bytes": 3200000},
"fact_conversions": {"rows": 3420, "bytes": 1300000}
},
"total_rows": 15420,
"total_bytes": 4500000,
"duration_seconds": 45.2,
"message": "Synced 15,420 rows to BigQuery"
}

Backfill Historical Data

POST /integrations/bigquery/backfill?account_id={account_id}

Sync historical data that may be missing. Backfills are processed in chunks for better progress visibility and resilience; failed chunks are tracked and can be retried.

Request Body (JSON):

{
"date_from": "2024-01-01",
"date_to": "2024-12-31",
"chunk_days": 7
}
FieldTypeRequiredDescription
date_fromdateYesStart date for backfill (cannot be in the future)
date_todateYesEnd date for backfill (cannot be in the future)
chunk_daysintegerNoDays per chunk (1-30, default 7). Smaller chunks = more progress updates
caution

A backfill range cannot exceed 365 days. For larger historical imports, run multiple backfills. Backfills consume BigQuery quota and may take several minutes.

Response:

{
"sync_id": "550e8400-e29b-41d4-a716-446655440001",
"sync_type": "backfill",
"date_from": "2024-01-01",
"date_to": "2024-12-31",
"status": "success",
"tables": {
"fact_traffic_daily": {"rows": 980000, "bytes": 210000000},
"fact_conversions": {"rows": 270000, "bytes": 90000000}
},
"total_rows": 1250000,
"total_bytes": 300000000,
"total_chunks": 52,
"failed_chunks": [],
"duration_seconds": 540.7,
"message": "Backfill completed successfully. 1,250,000 rows synced."
}

Retry Failed Sync

POST /integrations/bigquery/retry/{log_id}?account_id={account_id}

Retry a failed sync operation, reusing the date range from the original sync. Only failed or partial syncs can be retried.

Response:

{
"status": "success",
"original_log_id": 123,
"new_sync_id": "550e8400-e29b-41d4-a716-446655440002",
"sync_type": "incremental",
"total_rows": 15420,
"message": "Retry successful. 15,420 rows synced."
}

Sync Logs

List Sync Logs

GET /integrations/bigquery/logs?account_id={account_id}

Query Parameters:

ParameterTypeDefaultDescription
limitinteger20Max results (1-100)
offsetinteger0Skip N results

Response:

{
"logs": [
{
"id": 123,
"sync_id": "550e8400-e29b-41d4-a716-446655440000",
"sync_type": "incremental",
"sync_started_at": "2025-01-10T03:00:00Z",
"sync_completed_at": "2025-01-10T03:00:45Z",
"date_from": "2025-01-09",
"date_to": "2025-01-09",
"tables_synced": {
"fact_traffic_daily": {"rows": 12000, "bytes": 3200000},
"fact_conversions": {"rows": 3420, "bytes": 1300000}
},
"total_rows_synced": 15420,
"total_bytes_synced": 4500000,
"status": "success",
"error_message": null,
"duration_seconds": 45.2
}
],
"total": 30
}

Sync Status Values

StatusDescription
runningSync in progress
successSync completed successfully
partialSome tables failed
failedSync failed completely
cancelledSync was cancelled

Get Sync Log Detail

GET /integrations/bigquery/logs/{log_id}?account_id={account_id}

Returns detailed information for a single sync operation, including per-table results and any error message.


Schema Information

SealMetrics is a cookieless, aggregated analytics platform. The BigQuery export is a star schema of pre-aggregated daily (and optionally hourly) facts — there is no session-, visitor-, or event-level data. Every fact table is aggregated by date and a set of dimensions (UTM, geo, device, channel).

The tables created depend on the integration's data_types:

Tabledata_typeDescription
fact_traffic_dailytrafficDaily traffic with all dimensions (UTM, geo, device, channel)
fact_traffic_hourlytraffic_hourlyHourly traffic for intraday analysis (opt-in, 90-day TTL)
fact_pagespagesPage performance metrics
fact_landing_pageslanding_pagesLanding page performance with attribution
fact_conversionsconversionsConversion events with full attribution (aggregated by day + dimensions)
fact_microconversionsmicroconversionsMicroconversion events (form fills, clicks, etc.)
dim_accountsaccountsAccount metadata for context and JOINs (synced from PostgreSQL)
dim_countriesISO 3166-1 country code lookup table (always created, static data)
sync_metadataSync operation metadata for transparency and auditing (always created)

All fact tables are partitioned by date and clustered (primarily by account_id). Every synced row carries sync_id, synced_at, and account_id provenance columns.

Get Table Schema

GET /integrations/bigquery/schema

Returns the schema for all BigQuery tables that SealMetrics creates. This endpoint does not require an account_id — it describes the static star-schema definition.

Response:

{
"tables": {
"fact_traffic_daily": {
"description": "Daily traffic with all dimensions (UTM, geo, device, channel)",
"data_type": "traffic",
"schema": [
{"name": "sync_id", "type": "STRING", "mode": "REQUIRED"},
{"name": "synced_at", "type": "TIMESTAMP", "mode": "REQUIRED"},
{"name": "account_id", "type": "STRING", "mode": "REQUIRED"},
{"name": "date", "type": "DATE", "mode": "REQUIRED"},
{"name": "utm_source", "type": "STRING", "mode": "NULLABLE"},
{"name": "utm_medium", "type": "STRING", "mode": "NULLABLE"},
{"name": "utm_campaign", "type": "STRING", "mode": "NULLABLE"},
{"name": "utm_term", "type": "STRING", "mode": "NULLABLE"},
{"name": "utm_content", "type": "STRING", "mode": "NULLABLE"},
{"name": "channel_group", "type": "STRING", "mode": "NULLABLE"},
{"name": "country", "type": "STRING", "mode": "NULLABLE"},
{"name": "device_type", "type": "STRING", "mode": "NULLABLE"},
{"name": "browser", "type": "STRING", "mode": "NULLABLE"},
{"name": "os", "type": "STRING", "mode": "NULLABLE"},
{"name": "day_of_week", "type": "INT64", "mode": "NULLABLE"},
{"name": "entrances", "type": "INT64", "mode": "NULLABLE"},
{"name": "engaged_entrances", "type": "INT64", "mode": "NULLABLE"},
{"name": "page_views", "type": "INT64", "mode": "NULLABLE"},
{"name": "microconversions", "type": "INT64", "mode": "NULLABLE"},
{"name": "conversions", "type": "INT64", "mode": "NULLABLE"},
{"name": "revenue", "type": "NUMERIC", "mode": "NULLABLE"}
],
"partition_field": "date",
"clustering_fields": ["account_id", "utm_source", "country"],
"partition_expiration_days": null
}
}
}

Each table entry includes description, data_type, the full schema (field name, type, mode), partition_field, clustering_fields, and partition_expiration_days (e.g. 90 for fact_traffic_hourly).

Table Fields

fact_traffic_daily / fact_traffic_hourly

Aggregated traffic. fact_traffic_hourly adds an hour (0-23) column and omits utm_campaign/utm_term/utm_content; it has a 90-day partition expiration.

FieldTypeDescription
sync_idSTRINGSync operation identifier
synced_atTIMESTAMPWhen the row was synced
account_idSTRINGAccount identifier
dateDATEAggregation day (partition field)
hourINT64Hour 0-23 (fact_traffic_hourly only)
utm_source / utm_medium / utm_campaign / utm_term / utm_contentSTRINGUTM dimensions
channel_groupSTRINGChannel grouping
countrySTRINGCountry code (ISO)
device_type / browser / osSTRINGDevice/tech dimensions
day_of_weekINT64ISO day of week (1=Mon, 7=Sun)
entrancesINT64Entrances
engaged_entrancesINT64Engaged entrances
page_viewsINT64Page views
microconversionsINT64Microconversions
conversionsINT64Conversions
revenueNUMERICRevenue

fact_pages

FieldTypeDescription
sync_idSTRINGSync operation identifier
synced_atTIMESTAMPWhen the row was synced
account_idSTRINGAccount identifier
dateDATEAggregation day (partition field)
page_pathSTRINGPage path
content_groupingSTRINGContent grouping
countrySTRINGCountry code (ISO)
channel_groupSTRINGChannel grouping
entrancesINT64Entrances
engaged_entrancesINT64Engaged entrances
page_viewsINT64Page views

fact_landing_pages

FieldTypeDescription
sync_idSTRINGSync operation identifier
synced_atTIMESTAMPWhen the row was synced
account_idSTRINGAccount identifier
dateDATEAggregation day (partition field)
landing_pageSTRINGLanding page path
content_groupingSTRINGContent grouping
utm_source / utm_mediumSTRINGUTM dimensions
channel_groupSTRINGChannel grouping
countrySTRINGCountry code (ISO)
entrancesINT64Entrances
engaged_entrancesINT64Engaged entrances
microconversionsINT64Microconversions
conversionsINT64Conversions
revenueNUMERICRevenue

fact_conversions

Conversion events aggregated by day and attribution dimensions.

FieldTypeDescription
sync_idSTRINGSync operation identifier
synced_atTIMESTAMPWhen the row was synced
account_idSTRINGAccount identifier
dateDATEAggregation day (partition field)
conversion_typeSTRINGConversion type
utm_source / utm_medium / utm_campaign / utm_term / utm_contentSTRINGUTM dimensions
channel_groupSTRINGChannel grouping
countrySTRINGCountry code (ISO)
device_type / browser / osSTRINGDevice/tech dimensions
landing_pageSTRINGLanding page path
click_idSTRINGClick identifier (clid)
countINT64Number of conversions
amountNUMERICPer-conversion value
revenueNUMERICTotal revenue
propertiesJSONConversion properties

fact_microconversions

FieldTypeDescription
sync_idSTRINGSync operation identifier
synced_atTIMESTAMPWhen the row was synced
account_idSTRINGAccount identifier
dateDATEAggregation day (partition field)
conversion_typeSTRINGMicroconversion type
utm_source / utm_medium / utm_campaignSTRINGUTM dimensions
channel_groupSTRINGChannel grouping
countrySTRINGCountry code (ISO)
device_typeSTRINGDevice type
countINT64Number of microconversions
propertiesJSONEvent properties

dim_accounts

Account metadata (synced from PostgreSQL when the accounts data type is enabled).

FieldTypeDescription
sync_idSTRINGSync operation identifier
synced_atTIMESTAMPWhen the row was synced
account_idSTRINGAccount identifier
account_nameSTRINGAccount name
timezoneSTRINGAccount timezone
currencySTRINGAccount currency
plan_tierSTRINGPlan tier
organization_nameSTRINGOrganization name
created_atTIMESTAMPAccount creation time
is_activeBOOLWhether the account is active

dim_countries

Static ISO 3166-1 lookup, always created.

FieldTypeDescription
country_codeSTRINGISO 3166-1 country code
country_nameSTRINGCountry name
continentSTRINGContinent
regionSTRINGRegion

sync_metadata

Per-sync audit record, always created.

FieldTypeDescription
sync_idSTRINGSync operation identifier
synced_atTIMESTAMPWhen the sync ran
sync_typeSTRINGincremental or backfill
date_fromDATEStart of synced range
date_toDATEEnd of synced range
tables_syncedSTRING (REPEATED)Tables included in the sync
total_rowsINT64Total rows synced
duration_secondsFLOAT64Sync duration
sealmetrics_versionSTRINGSealMetrics export version

Required Permissions

The service account needs these BigQuery permissions:

PermissionPurpose
bigquery.datasets.createCreate dataset (if needed)
bigquery.datasets.getRead dataset metadata
bigquery.tables.createCreate tables
bigquery.tables.getRead table metadata
bigquery.tables.updateDataInsert data
bigquery.jobs.createRun load jobs

Recommended Role: roles/bigquery.dataEditor on the dataset.


Code Examples

Python - Setup and Sync

import requests

API_KEY = "sm_your_api_key"
BASE_URL = "https://api.sealmetrics.com/api/v1"
ACCOUNT_ID = "my-account"

def setup_bigquery(gcp_project_id, dataset_id, credentials_path):
"""Configure BigQuery integration (multipart file upload)."""

# Configuration goes in query params; the service account JSON is uploaded
# as a multipart file. The dataset and tables are created automatically.
with open(credentials_path, "rb") as f:
response = requests.post(
f"{BASE_URL}/integrations/bigquery",
headers={"X-API-Key": API_KEY},
params={
"account_id": ACCOUNT_ID,
"gcp_project_id": gcp_project_id,
"dataset_id": dataset_id,
"location": "EU",
"sync_frequency": "daily",
"data_types": ["traffic", "conversions", "microconversions",
"pages", "landing_pages"],
"backfill_days": 30,
},
files={"service_account_file": ("service-account.json", f, "application/json")},
)
response.raise_for_status()
return response.json()


def trigger_backfill(date_from, date_to):
"""Backfill historical data (max 365 days)."""
response = requests.post(
f"{BASE_URL}/integrations/bigquery/backfill",
headers={"X-API-Key": API_KEY},
params={"account_id": ACCOUNT_ID},
json={
"date_from": date_from,
"date_to": date_to,
"chunk_days": 7,
}
)
response.raise_for_status()
return response.json()


def check_sync_status(sync_id):
"""Check status of a sync operation."""
response = requests.get(
f"{BASE_URL}/integrations/bigquery/logs",
headers={"X-API-Key": API_KEY},
params={"account_id": ACCOUNT_ID}
)

for log in response.json()["logs"]:
if str(log["sync_id"]) == str(sync_id):
return log["status"]

return "not_found"

JavaScript - Monitor Sync

async function monitorSync(syncId) {
const checkStatus = async () => {
const response = await fetch(
`${BASE_URL}/integrations/bigquery/logs?account_id=${ACCOUNT_ID}`,
{
headers: { 'X-API-Key': API_KEY }
}
);

const { logs } = await response.json();
const log = logs.find(l => l.sync_id === syncId);

return log?.status || 'not_found';
};

let status = await checkStatus();

while (status === 'running') {
await new Promise(r => setTimeout(r, 5000)); // Wait 5s
status = await checkStatus();
console.log(`Sync status: ${status}`);
}

return status;
}

Error Codes

HTTP CodeErrorDescription
400invalid_credentialsService account JSON is malformed
401authentication_failedCredentials rejected by Google
403insufficient_permissionsMissing required BigQuery permissions
404integration_not_foundNo BigQuery integration configured
409sync_in_progressAnother sync is already running
429quota_exceededBigQuery API quota exceeded

Best Practices

1. Use Dedicated Service Account

Create a service account specifically for Sealmetrics:

gcloud iam service-accounts create sealmetrics-sync \
--display-name="Sealmetrics BigQuery Sync"

2. Limit Dataset Permissions

Grant permissions only on the specific dataset:

bq add-iam-policy-binding \
--member="serviceAccount:sealmetrics-sync@project.iam.gserviceaccount.com" \
--role="roles/bigquery.dataEditor" \
project:sealmetrics

3. Choose the Right Sync Frequency

Set sync_frequency to daily (default) for most use cases, hourly for near-real-time intraday analysis, or manual to trigger syncs only on demand via the sync endpoint.

4. Keep Data in the EU for GDPR

Use the default location of EU (or europe-west1) to keep your aggregated analytics data within the EU.

5. Monitor Sync Logs

Regularly check sync logs for failures and watch for the failed or partial status.