Skip to main content

Dremio

Test tables and views in Dremio, in any source, space or folder, however deeply nested.

1. Install​

uv tool install --python python3.11 --upgrade 'datacontract-cli[dremio]'

See Installation for pip, pipx, and Docker.

2. Authenticate​

Create a .env file in your working directory (or export the variables) with a Dremio user:

# .env
DATACONTRACT_DREMIO_USERNAME=analyst
DATACONTRACT_DREMIO_PASSWORD=mysecretpassword

Or with a personal access token instead:

# .env
DATACONTRACT_DREMIO_TOKEN=your-personal-access-token

The user needs to be able to query the datasets the contract describes. TLS and the other options are in the Dremio Reference.

3. Create a contract from your tables​

Import the tables and views of a source, space or folder, with the full type of every column. This also generates a ready-to-test servers block:

datacontract import dremio \
--source dremio.acme.com \
--schema lake.sales.marts \
--table orders \
--output datacontract.yaml

Repeat --table for multiple datasets, or omit it to import every table and view in the path. Add --port if the REST API doesn't listen on the default 9047.

The contract it writes points the servers block at the web console of your Dremio, and the schema at the path of the source, space or folder that holds the tables. Dremio keeps no NOT NULL or unique constraint, so required and unique below are added by hand:

apiVersion: v3.1.0
kind: DataContract
id: orders
name: Orders
version: 1.0.0
status: active
servers:
- server: production
type: dremio
host: dremio.acme.com
port: 9047
schema: lake.sales.marts
schema:
- name: orders
physicalName: orders
physicalType: table
properties:
- name: order_id
logicalType: string
physicalType: VARCHAR
required: true
unique: true
- name: order_total
logicalType: number
physicalType: DECIMAL(10,2)

The schema is a path: its names are separated by periods, and a name that has a period of its own goes in double quotes, the way Dremio SQL writes it ($scratch."bucket.v1".raw). Tables are then named by their name alone.

4. Test the actual data​

datacontract test datacontract.yaml
Testing datacontract.yaml
Server: production (type=dremio, host=dremio.acme.com, port=9047, schema=lake.sales.marts)
╭────────┬──────────────────────────────────────────────────────────────┬─────────────┬─────────╮
│ Result │ Check │ Field │ Details │
├────────┼──────────────────────────────────────────────────────────────┼─────────────┼─────────┤
│ passed │ Check that field 'order_id' is present │ order_id │ │
│ passed │ Check that field order_id has physical type VARCHAR │ order_id │ │
│ passed │ Check that field order_id has no missing values │ order_id │ │
│ passed │ Check that unique field order_id has no duplicate values │ order_id │ │
│ passed │ Check that field 'order_total' is present │ order_total │ │
│ passed │ Check that field order_total has physical type DECIMAL(10,2) │ order_total │ │
╰────────┴──────────────────────────────────────────────────────────────┴─────────────┴─────────╯
🟢 Data contract is valid. Ran 6 checks. Took 1.8 seconds.

Each check is a Dremio job, so a test run takes as long as your engine needs to answer a few dozen aggregations.

5. Let it catch a violation​

The contract becomes valuable when it detects drift. Tighten an expectation — for example, add a quality rule to a schema in datacontract.yaml:

schema:
- name: orders
# ...
quality:
- type: sql
description: No order has a negative total
query: SELECT COUNT(*) FROM orders WHERE order_total < 0
mustBe: 0

Run datacontract test datacontract.yaml again: every violation is listed as an error, and the command exits with code 1 — ready for CI/CD and scheduled runs so you catch drift before your consumers do.

Reference​

All authentication options and the data type handling: Dremio Reference.

Troubleshooting​

  • Dremio login failed (HTTP 401): the user name and password, or the token, are not accepted.
  • Object 'orders' not found: the schema in the servers block must be the path of the folder that holds the table. A table in a folder of a file system source (S3, $scratch) only appears once Dremio has read its metadata.
  • A connection that times out: port is the port of the web console and REST API (9047 by default), not the Arrow Flight (32010) or JDBC port.