Configuration¶

Configuring Metaxy¶

Metaxy can be configured using TOML configuration files, environment variables, or programmatically.

Either TOML-based or environment-based configuration is required to use the Metaxy CLI.

Example

metaxy.toml

project = "my_package"

[stores.dev]
type = "metaxy.ext.polars.DeltaMetadataStore"
[stores.dev.config]
root_path = "${HOME}/.metaxy/metadata"

Configuration Priority¶

When the same setting is defined in multiple places, Metaxy uses the following priority order (highest to lowest):

Explicit arguments - Values passed directly to MetaxyConfig()
Environment variables - Values from METAXY_* environment variables
Configuration files - Values from metaxy.toml or pyproject.toml

Config Discovery¶

Configuration files are discovered automatically by searching in the current or parent directories. metaxy.toml takes precedence over pyproject.toml.

Templating Environment Variables¶

Metaxy supports templating environment variables in configuration files using the ${VARIABLE_NAME} syntax.

Example

metaxy.toml

[stores.branch.config]
root_path = "s3://my-bucket/${BRANCH_NAME}"

Configuration Inheritance¶

A configuration file can inherit settings from another file using the extend field. This is especially useful in monorepos: a shared metaxy.toml can define stores, plugins, and other common settings.

metaxy.toml

project = "my_project"
extend = "../base.toml"

The extend path can be either relative to the file that contains it or an absolute path.

When a child config extends a parent, scalar fields are replaced, dictionary fields are shallow-merged, and array fields are extended.

Configuration Options¶

Main Metaxy configuration.

Loads from (in order of precedence):

Init arguments
Environment variables (METAXY_*)
Config file (metaxy.toml or [tool.metaxy] in pyproject.toml )

Environment variables can be templated with ${MY_VAR:-default} syntax.

Accessing current configuration

config = MetaxyConfig.load()

Getting a configured metadata store

store = config.get_store("prod")

Show JSON schema:

{
  "$defs": {
    "FeatureKey": {
      "description": "Feature key as a sequence of string parts.\n\nHashable for use as dict keys in registries.\nParts cannot contain forward slashes (/) or double underscores (__).\n\nExample:\n\n    ```py\n    FeatureKey(\"a/b/c\")  # String format\n    # FeatureKey(parts=['a', 'b', 'c'])\n\n    FeatureKey([\"a\", \"b\", \"c\"])  # List format\n    # FeatureKey(parts=['a', 'b', 'c'])\n\n    FeatureKey(FeatureKey([\"a\", \"b\", \"c\"]))  # FeatureKey copy\n    # FeatureKey(parts=['a', 'b', 'c'])\n    ```",
      "items": {
        "type": "string"
      },
      "title": "FeatureKey",
      "type": "array"
    },
    "FeatureSelection": {
      "additionalProperties": false,
      "description": "Selects a set of features from a metadata store.\n\nFields can be combined \u2014 both `projects` and `keys` may be set simultaneously\nto select all features from those projects *plus* the individually listed keys.\n\nSet `all=True` to select every feature in the store.\n\nSupports set operators `|`, `&`, and `-` which merge the underlying fields.\n\nExamples:\n    >>> mx.FeatureSelection(projects=[\"my-project\"])\n    FeatureSelection(projects=['my-project'], keys=None, all=None)\n\n    >>> mx.FeatureSelection(keys=[\"raw/video\", mx.FeatureKey(\"ml/embeddings\")])\n    FeatureSelection(projects=None, keys=[raw/video, ml/embeddings], all=None)\n\n    >>> mx.FeatureSelection(all=True)\n    FeatureSelection(projects=None, keys=None, all=True)\n\n    >>> mx.FeatureSelection(projects=[\"a\"]) | mx.FeatureSelection(projects=[\"b\"])\n    FeatureSelection(projects=['a', 'b'], keys=None, all=None)\n\n    >>> mx.FeatureSelection(keys=[\"a/b\", \"c/d\"]) & mx.FeatureSelection(keys=[\"c/d\"])\n    FeatureSelection(projects=None, keys=[c/d], all=None)\n\n    >>> mx.FeatureSelection(projects=[\"a\", \"b\", \"c\"]) - mx.FeatureSelection(projects=[\"b\"])\n    FeatureSelection(projects=['a', 'c'], keys=None, all=None)",
      "properties": {
        "projects": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Projects"
        },
        "keys": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/FeatureKey"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Keys"
        },
        "all": {
          "anyOf": [
            {
              "type": "boolean"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "All"
        }
      },
      "title": "FeatureSelection",
      "type": "object"
    },
    "PluginConfig": {
      "additionalProperties": true,
      "description": "Configuration for Metaxy plugins",
      "properties": {
        "enable": {
          "default": false,
          "description": "Whether to enable the plugin.",
          "title": "Enable",
          "type": "boolean"
        }
      },
      "title": "PluginConfig",
      "type": "object"
    },
    "StoreConfig": {
      "additionalProperties": false,
      "description": "Configuration options for metadata stores.",
      "properties": {
        "type": {
          "description": "Full import path to metadata store class (e.g., `\"metaxy.ext.duckdb.DuckDBMetadataStore\"`)",
          "title": "Type",
          "type": "string"
        },
        "config": {
          "additionalProperties": true,
          "description": "Store-specific configuration parameters (constructor kwargs). Includes `fallback_stores`, database connection parameters, etc.",
          "title": "Config",
          "type": "object"
        }
      },
      "required": [
        "type"
      ],
      "title": "StoreConfig",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "description": "Main Metaxy configuration.\n\nLoads from (in order of precedence):\n\n1. Init arguments\n\n2. Environment variables (METAXY_*)\n\n3. Config file (`metaxy.toml` or `[tool.metaxy]` in `pyproject.toml` )\n\nEnvironment variables can be templated with `${MY_VAR:-default}` syntax.\n\nExample: Accessing current configuration\n    <!-- skip next -->\n    ```py\n    config = MetaxyConfig.load()\n    ```\n\nExample: Getting a configured metadata store\n    ```py\n    store = config.get_store(\"prod\")\n    ```",
  "properties": {
    "store": {
      "default": "dev",
      "description": "Default metadata store to use",
      "title": "Store",
      "type": "string"
    },
    "stores": {
      "additionalProperties": {
        "$ref": "#/$defs/StoreConfig"
      },
      "description": "Named store configurations",
      "title": "Stores",
      "type": "object"
    },
    "extend": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "A relative or absolute path to a Metaxy configuration file to inherit settings from.",
      "title": "Extend"
    },
    "entrypoints": {
      "description": "List of Python module paths to load for feature discovery",
      "items": {
        "type": "string"
      },
      "title": "Entrypoints",
      "type": "array"
    },
    "theme": {
      "default": "default",
      "description": "Graph rendering theme for CLI visualization",
      "title": "Theme",
      "type": "string"
    },
    "ext": {
      "additionalProperties": {
        "$ref": "#/$defs/PluginConfig"
      },
      "description": "Configuration for Metaxy integrations with third-party tools",
      "title": "Ext",
      "type": "object"
    },
    "hash_truncation_length": {
      "default": 8,
      "description": "Truncate hash values to this length.",
      "minimum": 8,
      "title": "Hash Truncation Length",
      "type": "integer"
    },
    "enable_map_datatype": {
      "default": false,
      "description": "Preserve [`Map` datatype](/guide/concepts/metadata-stores.md/#map-datatype) across Metaxy operations. Requires `polars-map` to be installed. Experimental.",
      "title": "Enable Map Datatype",
      "type": "boolean"
    },
    "auto_create_tables": {
      "default": false,
      "description": "Auto-create tables when opening stores. It is not advised to enable this setting in production.",
      "title": "Auto Create Tables",
      "type": "boolean"
    },
    "project": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "[Project](/guide/concepts/projects.md) name. Used to scope operations to enable multiple independent projects in a shared metadata store. Does not modify feature keys or table names. Project names must be valid alphanumeric strings with dashes, underscores, and cannot contain forward slashes (`/`) or double underscores (`__`)",
      "title": "Project"
    },
    "locked": {
      "anyOf": [
        {
          "type": "boolean"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Whether to raise an error if an external feature doesn't have a matching feature version when [syncing external features][metaxy.sync_external_features] from the metadata store.",
      "title": "Locked"
    },
    "sync": {
      "default": true,
      "description": "Whether to automatically [sync external feature definitions][metaxy.sync_external_features] from the metadata during some operations. It's recommended to keep this enabled as it ensures versioning correctness for external feature definitions with a negligible performance impact.",
      "title": "Sync",
      "type": "boolean"
    },
    "extra_features": {
      "description": "Extra features to load from the metadata store when calling [`sync_external_features`][metaxy.sync_external_features]. Each entry is a [`FeatureSelection`][metaxy.FeatureSelection]. All entries are combined together. Learn more [here](/guide/concepts/definitions/external-features.md/#loading-extra-features).",
      "items": {
        "$ref": "#/$defs/FeatureSelection"
      },
      "title": "Extra Features",
      "type": "array"
    },
    "metaxy_lock_path": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Relative or absolute path to the lock file, resolved from the config file's location. Defaults to `metaxy.lock` next to the config file.",
      "title": "Metaxy Lock Path"
    }
  },
  "title": "MetaxyConfig",
  "type": "object"
}

Config:

env_prefix: METAXY_
env_nested_delimiter: __
frozen: True

store `pydantic-field` ¶

store: str = 'dev'

Default metadata store to use

metaxy.tomlpyproject.tomlEnvironment Variable

store = "dev"

[tool.metaxy]
store = "dev"

export METAXY_STORE=dev

stores `pydantic-field` ¶

stores: dict[str, StoreConfig]

Named store configurations

extend `pydantic-field` ¶

extend: str | None = None

A relative or absolute path to a Metaxy configuration file to inherit settings from.

metaxy.tomlpyproject.tomlEnvironment Variable

extend = "..."

[tool.metaxy]
extend = "..."

export METAXY_EXTEND=...

entrypoints `pydantic-field` ¶

entrypoints: list[str]

List of Python module paths to load for feature discovery

metaxy.tomlpyproject.tomlEnvironment Variable

entrypoints = []

[tool.metaxy]
entrypoints = []

export METAXY_ENTRYPOINTS=[]

hash_truncation_length `pydantic-field` ¶

hash_truncation_length: int = 8

Truncate hash values to this length.

metaxy.tomlpyproject.tomlEnvironment Variable

hash_truncation_length = 8

[tool.metaxy]
hash_truncation_length = 8

export METAXY_HASH_TRUNCATION_LENGTH=8

enable_map_datatype `pydantic-field` ¶

enable_map_datatype: bool = False

Preserve Map datatype across Metaxy operations. Requires polars-map to be installed. Experimental.

metaxy.tomlpyproject.tomlEnvironment Variable

enable_map_datatype = false

[tool.metaxy]
enable_map_datatype = false

export METAXY_ENABLE_MAP_DATATYPE=false

auto_create_tables `pydantic-field` ¶

auto_create_tables: bool = False

Auto-create tables when opening stores. It is not advised to enable this setting in production.

metaxy.tomlpyproject.tomlEnvironment Variable

auto_create_tables = false

[tool.metaxy]
auto_create_tables = false

export METAXY_AUTO_CREATE_TABLES=false

project `pydantic-field` ¶

project: str | None = None

Project name. Used to scope operations to enable multiple independent projects in a shared metadata store. Does not modify feature keys or table names. Project names must be valid alphanumeric strings with dashes, underscores, and cannot contain forward slashes (/) or double underscores (__)

metaxy.tomlpyproject.tomlEnvironment Variable

project = "..."

[tool.metaxy]
project = "..."

export METAXY_PROJECT=...

locked `pydantic-field` ¶

locked: bool | None = None

Whether to raise an error if an external feature doesn't have a matching feature version when syncing external features from the metadata store.

metaxy.tomlpyproject.tomlEnvironment Variable

locked = false

[tool.metaxy]
locked = false

export METAXY_LOCKED=...

sync `pydantic-field` ¶

sync: bool = True

Whether to automatically sync external feature definitions from the metadata during some operations. It's recommended to keep this enabled as it ensures versioning correctness for external feature definitions with a negligible performance impact.

metaxy.tomlpyproject.tomlEnvironment Variable

sync = true

[tool.metaxy]
sync = true

export METAXY_SYNC=true

extra_features `pydantic-field` ¶

extra_features: list[FeatureSelection]

Extra features to load from the metadata store when calling sync_external_features. Each entry is a FeatureSelection. All entries are combined together. Learn more here.

metaxy_lock_path `pydantic-field` ¶

metaxy_lock_path: str | None = None

Relative or absolute path to the lock file, resolved from the config file's location. Defaults to metaxy.lock next to the config file.

metaxy.tomlpyproject.tomlEnvironment Variable

metaxy_lock_path = "..."

[tool.metaxy]
metaxy_lock_path = "..."

export METAXY_METAXY_LOCK_PATH=...

The stores field configures metadata store backends. This is a mapping of store names to their configurations. The default store is named "dev".

Configuration options for metadata stores.

Show JSON schema:

{
  "additionalProperties": false,
  "description": "Configuration options for metadata stores.",
  "properties": {
    "type": {
      "description": "Full import path to metadata store class (e.g., `\"metaxy.ext.duckdb.DuckDBMetadataStore\"`)",
      "title": "Type",
      "type": "string"
    },
    "config": {
      "additionalProperties": true,
      "description": "Store-specific configuration parameters (constructor kwargs). Includes `fallback_stores`, database connection parameters, etc.",
      "title": "Config",
      "type": "object"
    }
  },
  "required": [
    "type"
  ],
  "title": "StoreConfig",
  "type": "object"
}

Config:

extra: forbid
frozen: True

type `pydantic-field` ¶

type: str

Full import path to metadata store class (e.g., "metaxy.ext.duckdb.DuckDBMetadataStore")

metaxy.tomlpyproject.tomlEnvironment Variable

[stores.dev]
type = "..."

[tool.metaxy.stores.dev]
type = "..."

export METAXY_STORES__DEV__TYPE=...

config `pydantic-field` ¶

config: dict[str, Any]

Store-specific configuration parameters (constructor kwargs). Includes fallback_stores, database connection parameters, etc.

metaxy.tomlpyproject.tomlEnvironment Variable

[stores.dev]
config = {}

[tool.metaxy.stores.dev]
config = {}

export METAXY_STORES__DEV__CONFIG={}

Configuration¶

Configuring Metaxy¶

Configuration Priority¶

Config Discovery¶

Templating Environment Variables¶

Configuration Inheritance¶

Configuration Options¶

store pydantic-field ¶

stores pydantic-field ¶

extend pydantic-field ¶

entrypoints pydantic-field ¶

hash_truncation_length pydantic-field ¶

enable_map_datatype pydantic-field ¶

auto_create_tables pydantic-field ¶

project pydantic-field ¶

locked pydantic-field ¶

sync pydantic-field ¶

extra_features pydantic-field ¶

metaxy_lock_path pydantic-field ¶

type pydantic-field ¶

config pydantic-field ¶

store `pydantic-field` ¶

stores `pydantic-field` ¶

extend `pydantic-field` ¶

entrypoints `pydantic-field` ¶

hash_truncation_length `pydantic-field` ¶

enable_map_datatype `pydantic-field` ¶

auto_create_tables `pydantic-field` ¶

project `pydantic-field` ¶

locked `pydantic-field` ¶

sync `pydantic-field` ¶

extra_features `pydantic-field` ¶

metaxy_lock_path `pydantic-field` ¶

type `pydantic-field` ¶

config `pydantic-field` ¶