Skip to main content

Custom Quality Rules

A type: custom rule carries a check written in the native syntax of a specific quality engine. You name the engine with engine and provide the engine-specific definition under implementation. Custom rules let you use capabilities of an engine that the portable library and SQL rules don't cover.

Custom rules are consumed by the matching engine — typically via the corresponding exporter — rather than being compiled to portable SQL.

note

datacontract test does not run custom quality rules except for those with engine: datacontract-cli. Consider type: sql as an executed alternative.

Custom quality checks​

With engine: datacontract-cli, a custom rule uses a custom quality check: a parameterised SQL check defined once in a folder and run by datacontract test.

quality:
  - type: custom
    engine: datacontract-cli
    implementation:
      check: between
      arguments:
        min: 0
        max: 10000

Databricks DQX​

schema:
  - name: events
    properties:
      - name: event_type
        logicalType: string
        quality:
          - type: custom
            engine: dqx
            implementation:
              criticality: error
              check:
                function: is_in_list
                arguments:
                  allowed:
                    - click
                    - view
                    - purchase

Export these to a DQX check file with datacontract export dqx.

SodaCL​

schema:
  - name: orders
    quality:
      - type: custom
        engine: soda
        implementation: |
          checks for orders:
            - row_count > 10

Export to a SodaCL file with datacontract export sodacl.

warning

Raw SodaCL custom checks (type: custom, engine: soda) are no longer executed by datacontract test since soda-core was removed; such a rule is reported as a warning. Migrate it to a SQL rule (type: sql) to keep it executable, or keep it as a custom rule purely for the SodaCL export.

Other engines​

The same pattern applies to other engines (for example Great Expectations). Provide engine and an implementation understood by that engine, and use the matching exporter to generate its native artifact.