Types¶
Versioning Engine¶
dataclass
¶
LazyIncrement(
*,
new: LazyFrame[Any],
stale: LazyFrame[Any],
orphaned: LazyFrame[Any],
input: LazyFrame[Any] | None = None,
)
Result of an incremental update containing lazy dataframes.
Attributes:
-
new(LazyFrame[Any]) –New samples from upstream not present in current metadata
-
stale(LazyFrame[Any]) –Samples with provenance different to what was processed before
-
orphaned(LazyFrame[Any]) –Samples that have been processed before but are no longer present in upstream
-
input(LazyFrame[Any] | None) –Joined upstream metadata with
FeatureDeprules applied.
Functions¶
Collect all lazy frames to eager DataFrames.
Tip
If all lazy frames are Polars frames, leverages
polars.collect_all
to optimize the collection process and take advantage of common subplan elimination.
Parameters:
-
**kwargs(Any, default:{}) –backend-specific keyword arguments to pass to the collect method of the lazy frames.
Returns:
-
Increment(Increment) –The collected increment.
Source code in src/metaxy/versioning/types.py
def collect(self, **kwargs: Any) -> Increment:
"""Collect all lazy frames to eager DataFrames.
!!! tip
If all lazy frames are Polars frames, leverages
[`polars.collect_all`](https://docs.pola.rs/api/python/stable/reference/api/polars.collect_all.html)
to optimize the collection process and take advantage of common subplan elimination.
Args:
**kwargs: backend-specific keyword arguments to pass to the collect method of the lazy frames.
Returns:
Increment: The collected increment.
"""
if (
self.new.implementation
== self.stale.implementation
== self.orphaned.implementation
== nw.Implementation.POLARS
):
polars_eager_increment = PolarsLazyIncrement(
new=self.new.to_native(),
stale=self.stale.to_native(),
orphaned=self.orphaned.to_native(),
).collect(**kwargs)
return Increment(
new=nw.from_native(polars_eager_increment.new),
stale=nw.from_native(polars_eager_increment.stale),
orphaned=nw.from_native(polars_eager_increment.orphaned),
)
else:
return Increment(
new=self.new.collect(**kwargs),
stale=self.stale.collect(**kwargs),
orphaned=self.orphaned.collect(**kwargs),
)
to_polars() -> PolarsLazyIncrement
Convert to Polars.
Tip
If the Narwhals lazy frames are already backed by Polars, this is a no-op.
Warning
If the Narwhals lazy frames are not backed by Polars, this will trigger a full materialization for them.
Source code in src/metaxy/versioning/types.py
def to_polars(self) -> PolarsLazyIncrement:
"""Convert to Polars.
!!! tip
If the Narwhals lazy frames are already backed by Polars, this is a no-op.
!!! warning
If the Narwhals lazy frames are **not** backed by Polars, this will
trigger a full materialization for them.
"""
return PolarsLazyIncrement(
new=lazy_frame_to_polars(self.new),
stale=lazy_frame_to_polars(self.stale),
orphaned=lazy_frame_to_polars(self.orphaned),
input=lazy_frame_to_polars(self.input) if self.input is not None else None,
)
Bases: NamedTuple
Result of an incremental update containing eager dataframes.
Attributes:
-
new(DataFrame[Any]) –New samples from upstream not present in current metadata
-
stale(DataFrame[Any]) –Samples with provenance different to what was processed before
-
orphaned(DataFrame[Any]) –Samples that have been processed before but are no longer present in upstream
Functions¶
to_polars() -> PolarsIncrement
dataclass
¶
PolarsLazyIncrement(
*,
new: LazyFrame,
stale: LazyFrame,
orphaned: LazyFrame,
input: LazyFrame | None = None,
)
Like LazyIncrement, but converted to Polars lazy frames.
Attributes:
-
new(LazyFrame) –New samples from upstream not present in current metadata
-
stale(LazyFrame) –Samples with provenance different to what was processed before
-
orphaned(LazyFrame) –Samples that have been processed before but are no longer present in upstream
-
input(LazyFrame | None) –Joined upstream metadata with
FeatureDeprules applied.
Attributes¶
class-attribute
instance-attribute
¶
Functions¶
collect(**kwargs: Any) -> PolarsIncrement
Collect into a PolarsIncrement.
Tip
Leverages polars.collect_all
to optimize the collection process and take advantage of common subplan elimination.
Parameters:
-
**kwargs(Any, default:{}) –backend-specific keyword arguments to pass to the collect method of the lazy frames.
Returns:
-
PolarsIncrement(PolarsIncrement) –The collected increment.
Source code in src/metaxy/versioning/types.py
def collect(self, **kwargs: Any) -> PolarsIncrement:
"""Collect into a [`PolarsIncrement`][metaxy.versioning.types.PolarsIncrement].
!!! tip
Leverages [`polars.collect_all`](https://docs.pola.rs/api/python/stable/reference/api/polars.collect_all.html)
to optimize the collection process and take advantage of common subplan elimination.
Args:
**kwargs: backend-specific keyword arguments to pass to the collect method of the lazy frames.
Returns:
PolarsIncrement: The collected increment.
"""
added, changed, removed = pl.collect_all([self.new, self.stale, self.orphaned], **kwargs)
return PolarsIncrement(added, changed, removed) # ty: ignore[invalid-argument-type]
Keys¶
Types for working with feature and field keys.
Canonical Keys¶
Bases: _Key
Feature key as a sequence of string parts.
Hashable for use as dict keys in registries. Parts cannot contain forward slashes (/) or double underscores (__).
Example:
```py
FeatureKey("a/b/c") # String format
# FeatureKey(parts=['a', 'b', 'c'])
FeatureKey(["a", "b", "c"]) # List format
# FeatureKey(parts=['a', 'b', 'c'])
FeatureKey(FeatureKey(["a", "b", "c"])) # FeatureKey copy
# FeatureKey(parts=['a', 'b', 'c'])
```
Source code in src/metaxy/models/types.py
Bases: _Key
Field key as a sequence of string parts.
Hashable for use as dict keys in registries. Parts cannot contain forward slashes (/) or double underscores (__).
Example:
```py
FieldKey("a/b/c") # String format
# FieldKey(parts=['a', 'b', 'c'])
FieldKey(["a", "b", "c"]) # List format
# FieldKey(parts=['a', 'b', 'c'])
FieldKey(FieldKey(["a", "b", "c"])) # FieldKey copy
# FieldKey(parts=['a', 'b', 'c'])
```
Source code in src/metaxy/models/types.py
Type Annotations¶
These are typically used to annotate function parameters. Most APIs in Metaxy accepts them and perform type coercion into canonical types.
module-attribute
¶
CoercibleToFeatureKey: TypeAlias = "str | Sequence[str] | FeatureKey | type[BaseFeature] | FeatureDefinition | FeatureSpec"
Type alias for values that can be coerced to a FeatureKey.
Accepted formats:
str: Slash-separated string like"raw/video"or"ml/embeddings/v2"Sequence[str]: sequences of parts like["user", "profile"]FeatureKey: Pass through unchangedtype[BaseFeature]: AnyBaseFeaturesubclass - extracts its key via.spec().keyFeatureDefinition: Extracts its key via.keyFeatureSpec: Extracts its key via.key
module-attribute
¶
Pydantic Type Annotations¶
These types are used for type coercion into canonical types with Pydantic.
module-attribute
¶
ValidatedFeatureKeySequence: TypeAlias = Sequence[
ValidatedFeatureKey
]
module-attribute
¶
ValidatedFieldKeySequence: TypeAlias = Sequence[
ValidatedFieldKey
]
Adapters¶
These can perform type coercsion into canonical types in non-pydantic code.
module-attribute
¶
ValidatedFeatureKeyAdapter: TypeAdapter[
ValidatedFeatureKey
] = TypeAdapter(ValidatedFeatureKey)
module-attribute
¶
ValidatedFeatureKeySequenceAdapter: TypeAdapter[
ValidatedFeatureKeySequence
] = TypeAdapter(ValidatedFeatureKeySequence)
module-attribute
¶
ValidatedFieldKeyAdapter: TypeAdapter[ValidatedFieldKey] = (
TypeAdapter(ValidatedFieldKey)
)
module-attribute
¶
ValidatedFieldKeySequenceAdapter: TypeAdapter[
ValidatedFieldKeySequence
] = TypeAdapter(ValidatedFieldKeySequence)
Other Types¶
Bases: NamedTuple
Result of recording a feature graph snapshot.
Attributes:
-
project_version(str) –The deterministic hash of the graph's project version
-
already_pushed(bool) –True if this project_version was already pushed previously
-
updated_features(list[str]) –List of feature keys with updated information (changed definition_version)