Google BigQuery Reference
Authentication options and data type handling for BigQuery connections.
Server
servers:
- server: production
type: bigquery
project: datameshexample-product
dataset: datacontract_cli_test_dataset
Authentication
Authentication uses a Service Account Key or Application Default Credentials (ADC) — including Workload Identity Federation (WIF), the GCE metadata server, and gcloud auth application-default login. The account needs the BigQuery Job User, BigQuery Data Viewer and BigQuery Read Session User roles.
| Variable | Example | Description |
|---|---|---|
DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH | ~/service-access-key.json | Service Account key JSON file used by test. If unset, ADC/WIF is used. |
DATACONTRACT_BIGQUERY_BILLING_PROJECT | my-compute-project | Optional. Project to bill query jobs to. Requires bigquery.jobUser, bigquery.readSessionUser on the billing project and bigquery.dataViewer on the data project. |
DATACONTRACT_BIGQUERY_IMPERSONATION_ACCOUNT | runner@my-project.iam.gserviceaccount.com | Optional. Service account to impersonate, using the key file above (or ADC) as the source principal. Requires roles/iam.serviceAccountTokenCreator on the target. |
project and dataset come from the contract's servers block, and can be overridden with DATACONTRACT_BIGQUERY_PROJECT and DATACONTRACT_BIGQUERY_DATASET. datacontract import bigquery always uses ADC — set GOOGLE_APPLICATION_CREDENTIALS to point it at a key file.
Query location and encryption key
By default, query jobs run in the location BigQuery chooses (US when none can be inferred) and carry no encryption key. Projects that enforce the org policy constraints/gcp.restrictNonCmekServices reject such jobs:
Your administrator requires that you specify an encryption key for queries in project `my-project`.
Set the query location and a Cloud KMS key with these variables, or as customProperties on the server so each environment can declare its own:
| Variable | Server customProperties | Example | Description |
|---|---|---|---|
DATACONTRACT_BIGQUERY_LOCATION | location | europe-west3 | Optional. Location in which query jobs run. Must match the location of the datasets. |
DATACONTRACT_BIGQUERY_KMS_KEY_NAME | kmsKeyName | projects/my-kms-project/locations/europe-west3/keyRings/my-ring/cryptoKeys/my-key | Optional. Cloud KMS key that encrypts the query results. The caller's service account needs roles/cloudkms.cryptoKeyEncrypterDecrypter on the key. |
The variable wins over the custom property.
A location that doesn't match the datasets' fails with 403 Access Denied: … User does not have permission to query table …, or perhaps it does not exist.
servers:
- server: bigquery
type: bigquery
project: my-project
dataset: my_dataset
customProperties:
- property: location
value: europe-west3
- property: kmsKeyName
value: projects/my-kms-project/locations/europe-west3/keyRings/my-ring/cryptoKeys/my-key
Data types
Importing
datacontract import bigquery maps BigQuery types as follows; the original BigQuery type is kept as physicalType. An unknown type fails the import with an error.
| BigQuery type | logicalType |
|---|---|
STRING | string (maxLength if declared) |
BYTES | array |
INTEGER, INT64 | integer |
FLOAT, FLOAT64 | number |
NUMERIC, BIGNUMERIC | number (precision/scale as custom properties) |
BOOLEAN, BOOL | boolean |
TIMESTAMP, DATETIME | timestamp |
DATE | date |
TIME | time |
GEOGRAPHY, JSON | object |
INTERVAL | string |
RECORD, STRUCT | object with nested properties |
RANGE | array with typed items |
any type with mode REPEATED | array with typed items |
Fields with mode REQUIRED become required: true. Tables, external tables, and snapshots become schema physicalType: table; views and materialized views become view.
Testing
BigQuery supports native type introspection: the declared physicalType is checked against the actual column type from the BigQuery API. Timezone variants of timestamps are interchangeable; parameters (precision, length) are only enforced when the contract declares them. If the physicalType cannot be interpreted as a BigQuery type, the check falls back to the logical type category comparison.
Logical type mapping
When no physicalType is declared, the CLI derives the native type from the logicalType — for example in datacontract export sql and the dbt exports. This table is generated from the converter in the CLI's code:
logicalType | BigQuery type |
|---|---|
string | STRING |
integer | INTEGER |
number | NUMERIC |
boolean | BOOL |
date | DATE |
timestamp | TIMESTAMP |
time | TIME |
object | STRUCT<> |
array | ARRAY<STRING> |
map | JSON |
vector | ARRAY<FLOAT64> |