Import: BigQuery

Creates a data contract from BigQuery, either directly from the BigQuery API or from an exported table definition (JSON file).

From the BigQuery API

datacontract import bigquery \
  --project my-project \
  --dataset my_dataset \
  --table orders \
  --output datacontract.yaml

Repeat --table for multiple tables, or omit it to import every table in the dataset. Authentication uses Application Default Credentials (gcloud auth application-default login, or GOOGLE_APPLICATION_CREDENTIALS).

The generated contract includes a ready-to-test servers block, so you can run datacontract test datacontract.yaml immediately afterwards — see the BigQuery connection guide for the full 5-minute walkthrough and troubleshooting.

From an exported JSON file

{/* AUTOGENERATED EXAMPLE: do not edit by hand; regenerate with the update script. */}

Given this orders.json:

{
  "kind": "bigquery#table",
  "tableReference": {
    "projectId": "my-gcp-project",
    "datasetId": "orders",
    "tableId": "orders"
  },
  "description": "One row per customer order.",
  "schema": {
    "fields": [
      { "name": "order_id", "type": "STRING", "mode": "REQUIRED", "description": "Unique identifier of the order." },
      { "name": "order_timestamp", "type": "TIMESTAMP", "mode": "REQUIRED" },
      { "name": "customer_id", "type": "STRING", "mode": "REQUIRED" },
      { "name": "order_total", "type": "NUMERIC", "mode": "REQUIRED" },
      { "name": "status", "type": "STRING", "mode": "REQUIRED" }
    ]
  }
}

Run:

datacontract import bigquery --source orders.json

to produce the data contract:

version: 1.0.0
kind: DataContract
apiVersion: v3.1.0
id: my-data-contract
name: My Data Contract
status: draft
servers:
- server: bigquery
  type: bigquery
  dataset: orders
  project: my-gcp-project
schema:
- name: orders
  physicalType: table
  description: One row per customer order.
  logicalType: object
  physicalName: orders
  properties:
  - name: order_id
    physicalType: STRING
    description: Unique identifier of the order.
    logicalType: string
    required: true
  - name: order_timestamp
    physicalType: TIMESTAMP
    logicalType: timestamp
    required: true
  - name: customer_id
    physicalType: STRING
    logicalType: string
    required: true
  - name: order_total
    physicalType: NUMERIC
    logicalType: number
# …

{/* END AUTOGENERATED EXAMPLE */}

All options: datacontract import bigquery.

This site is open source. Improve this page.