Define your Quality Rules

Beyond schema checks (presence, type, nullability), a data contract can declare quality rules. When you run datacontract test, executable rules run against the data source and are reported alongside the schema checks.

Quality rules use the ODCS quality attribute, which can be attached either to a schema (table/object level) or to an individual property (column/field level):

schema:
  - name: orders
    quality:                       # schema-level rule
      - type: library
        metric: rowCount
        mustBeGreaterThan: 0
    properties:
      - name: order_total
        quality:                   # property-level rule
          - type: library
            metric: nullValues
            mustBe: 0

Quality rule types

ODCS defines four types of quality rule. Each has its own page:

SQLRun a custom SQL query and compare the result to a threshold. LibraryPredefined, portable checks such as rowCount, nullValues, and duplicateValues. TextA human-readable expectation, for documentation only. CustomEngine-specific checks (e.g. DQX, SodaCL, Great Expectations).

Running only quality checks

Use --checks to restrict a run to quality rules:

datacontract test --checks quality datacontract.yaml

Where quality rules are used

Executable rules (sql, library) run during test. All rule types are also translated when you export to a quality engine:

• dbt — rules become native dbt tests, with dbt_utils tests and singular SQL tests for rules that cannot be expressed natively.
• SodaCL — rules become Soda checks.
• Great Expectations — rules become expectations in a suite.
• DQX — rules become Databricks DQX checks.

Inspecting failing rows

When a quality (or schema) check fails, add --include-failed-samples to test to collect a small sample of the offending rows (identifier + offending columns; sensitive columns are omitted). This is off by default.

datacontract test --include-failed-samples datacontract.yaml

Learn more

• The full quality syntax is part of the Open Data Contract Standard.

This site is open source. Improve this page.