System Columns¶
Metaxy reserves a set of system-managed columns that it attaches to user-defined feature metadata tables. These columns are part of the storage interface and are used by the metadata store. Learn more about the storage layout design here.
Canonical column names¶
| Canonical name | Explanation | Level | Type |
|---|---|---|---|
metaxy_provenance_by_field |
Derived from upstream data versions and code version per field | sample | struct |
metaxy_provenance |
Hash of metaxy_provenance_by_field |
sample | string |
metaxy_data_version_by_field |
Defaults to metaxy_provenance_by_field, can be user-defined |
sample | struct |
metaxy_data_version |
Hash of metaxy_data_version_by_field |
sample | string |
metaxy_feature_version |
Derived from versions of relevant upstream fields | feature | string |
metaxy_project_version |
Derived from all Metaxy features which belong to the same Project | project | string |
metaxy_definition_version |
Hash of the feature spec and Pydantic model schema | feature | string |
metaxy_created_at |
Timestamp when the metadata row was created | sample | string |
metaxy_updated_at |
Timestamp when the metadata row was last written to the store | sample | string |
metaxy_deleted_at |
Timestamp when the metadata row was soft-deleted (null if active) | sample | string |
metaxy_materialization_id |
External orchestration run ID (e.g., Dagster, Airflow) for tracking | run | string |
All system column names start with the metaxy_ prefix.
Example Table¶
| id | metaxy_feature_version | metaxy_data_version | metaxy_data_version_by_field | metaxy_provenance | metaxy_provenance_by_field | metaxy_created_at | metaxy_updated_at | metaxy_deleted_at |
|---|---|---|---|---|---|---|---|---|
| video_001 | a1b2c3d4 | e7f8a9b0 | {"audio": "a7f3c2d8", "frames": "b9e1f4a2"} |
e7f8a9b0 | {"audio": "a7f3c2d8", "frames": "b9e1f4a2"} |
2024-01-15T10:30:00Z | 2024-01-15T10:30:00Z | null |
| video_002 | a1b2c3d4 | c1e4b9d8 | {"audio": "d4b8e9c1", "frames": "f2a6d7b3"} |
c1e4b9d8 | {"audio": "d4b8e9c1", "frames": "f2a6d7b3"} |
2024-01-15T10:31:00Z | 2024-01-15T10:31:00Z | null |
| video_003 | a1b2c3d4 | k1j2ah7v | {"audio": "custom01", "frames": "custom02"} |
a8e2f4c9 | {"audio": "c9f2a8e4", "frames": "e7d3b1c5"} |
2024-01-15T10:32:00Z | 2024-01-16T14:20:00Z | null |
| video_001 | f5d6e7c8 | b2c3d4e5 | {"audio": "b1e4f9a7", "frames": "a8c2e6d9"} |
b2c3d4e5 | {"audio": "b1e4f9a7", "frames": "a8c2e6d9"} |
2024-01-18T09:00:00Z | 2024-01-18T09:00:00Z | null |
It can also contain custom user-defined columns (1).
- and in fact,
idis such a column, because ID columns are customizable