Skip to main content

Amazon S3

Go from files in a bucket to a tested data contract in about five minutes. Works with S3 and any S3-compatible endpoint (e.g. MinIO), for data in CSV, JSON, Delta, or Parquet format.

For tables in an Amazon S3 table bucket, use the dedicated Amazon S3 Tables guide with type: iceberg instead.

1. Install​

uv tool install --python python3.11 --upgrade 'datacontract-cli[s3]'

See Installation for pip, pipx, and Docker.

2. Authenticate​

The easiest way is to sign in to AWS once — the CLI picks the session up, so no key is stored anywhere:

aws sso login   # or any other way of getting an AWS session

Prefer static keys? Set them directly and they are used instead:

# .env
DATACONTRACT_S3_REGION=eu-central-1
DATACONTRACT_S3_ACCESS_KEY_ID=AKIAXV5Q5QABCDEFGH
DATACONTRACT_S3_SECRET_ACCESS_KEY=93S7LRrJcqLaaaa/XXXXXXXXXXXXX

Public buckets need neither: with nothing configured the objects are read unsigned.

3. Create a contract from your files​

Import the schema straight from the bucket. This also generates a ready-to-test servers block:

datacontract import s3 \
  --source 's3://my-bucket/orders/*.json' \
  --output datacontract.yaml

The format is taken from the file suffix; pass --format for Delta tables, which have none. Add --endpoint-url for an S3-compatible store such as MinIO.

The import reads only the field names and types. Properties like required cannot be inferred from the files and need to be adjusted manually.

4. Test the actual data​

datacontract test datacontract.yaml
Testing datacontract.yaml
Server: production (type=s3, format=json, location=s3://my-bucket/orders/*.json)
╭────────┬─────────────────────────────────────────────────┬──────────┬─────────╮
│ Result │ Check                                           │ Field    │ Details │
├────────┼─────────────────────────────────────────────────┼──────────┼─────────┤
│  ...   │                                                 │          │         │
│ passed │ Check that field 'order_id' is present          │ order_id │         │
│ passed │ Check that field order_id has no missing values │ order_id │         │
│  ...   │                                                 │          │         │
╰────────┴─────────────────────────────────────────────────┴──────────┴─────────╯
🟢 Data contract is valid. Ran 11 checks. Took 3.6 seconds.

5. Let it catch a violation​

The contract becomes valuable when it detects drift. Tighten an expectation — for example, add a quality rule:

schema:
  - name: orders
    # ...
    quality:
      - type: sql
        description: No order has a negative total
        query: SELECT COUNT(*) FROM orders WHERE order_total < 0
        mustBe: 0

Run datacontract test datacontract.yaml again: every violation is listed as an error, and the command exits with code 1 — ready for CI/CD and scheduled runs so you catch drift before your consumers do.

Reference​

All authentication options and the data type handling per file format: S3 Reference.

CSV files are read as UTF-8 unless the server declares an encoding (ODCS v3.2.0), such as ISO-8859-1 or UTF-16. JSON files are always read as UTF-8.

Troubleshooting​

  • 403 Forbidden / Access Denied — the key pair lacks s3:GetObject/s3:ListBucket on the location, or DATACONTRACT_S3_REGION doesn't match the bucket's region.
  • No files found that match the pattern — check the location glob; it matches object keys, not directories.
  • MinIO / S3-compatible storage fails to connect — set endpointUrl in the servers block; the CLI then switches to path-style addressing.