Skip to content

Creating & Managing Features

Defining Feature pipelines

What programming languages do you support for defining features?

For each feature in Tecton, you will create a python-based feature definition file that includes all of the metadata and logic you want Tecton to manage for you.

Tecton's transformation logic is managed in Spark, using PySpark or SQL transformations. If your model requires request-time transformations, those are managed in python.

See Feature Views for more details.

What data types are supported for feature values?

Tecton supports the following feature data types:

  • Int64: LongType in Tecton on Spark (Databricks or EMR). NUMBER in Tecton on Snowflake.
  • Float64: DoubleType in Tecton on Spark (Databricks or EMR). FLOAT in Tecton on Snowflake.
  • String: StringType in Tecton on Spark (Databricks or EMR). STRING in Tecton on Snowflake.
  • Bool: BooleanType in Tecton on Spark (Databricks or EMR). BOOLEAN in Tecton on Snowflake. Bool is not supported as an element of an Array.

In Tecton on Spark (Databricks or EMR), the following data types are also supported:

  • Array with Int64, Float32, Float64, and String elements.
  • Struct which can contained named fields of any other feature types.
    • Structs are only supported as On-Demand Feature View outputs.
    • See this documentation for usage examples.

Feature materialization and lineage

What happens when the definition of a feature changes?

If a feature's definition changes, Tecton automatically detects all the dependencies on that feature, surfaces that for you, and asks you to confirm if you want to go forward with the changes. If you would like to roll back the changes or see the feature lineage, these definitions are backed by git. So you can always track the state of the world of your feature store, at all times.

How far back does Tecton support time-travel?

You set your features' backfill start date in Tecton. Time-travel can be performed as far back as feature data exists.

What support do you provide for time travel?

Tecton performs time travel on a row level basis - our granularity of time travel can be quite specific. If you have event driven ML models where you're regularly making predictions and you need to go back to every single specific point in time, and get the feature values as of that point in time, Tecton will handle that full travel query as opposed to just being able to get all feature values at a single point in time.

Does Tecton provide the functionality to replay and fix a backfill if the underlying data source is updated?

Yes, it is possible to kick off an "overwrite backfill" for a particular time range. Tecton will replay all transformations. This functionality is currently in private preview. To run an overwrite backfill, contact support@tecton.ai.

When scheduling materializations, does Tecton only materialize new data? Or does Tecton re-materialize all data?

Generally speaking, Tecton only reads and computes new data. There may be instances in which more historical data is required (eg, computing a one month average at materialization time requires knowing the full window of information).

What does Tecton do for data lineage? Does it support the entire data flow?

For data lineage, we consider both how features are created and how they are consumed. For feature creation, we show you the entire data flow - from your raw data sources, to the different transformations being ran, to where the data is being stored. For feature consumption, we have concept of a FeatureService which maps to the features for a model that is running. For any feature, you can see which services are using it and, likewise for any service, what are all the features that are inside of it - there is bidirectional tracking.

Does Tecton have an Airflow or Prefect integration?

Tecton does not currently have a first-class Airflow or Prefect integration. However, since Tecton's SDK is a plain Python package, it can be run from any environment with access to Python.

Currently, reading features from Tecton requires access to a Spark cluster. Ensure your Airflow environment has access to a Spark cluster when using Airflow with Tecton.

Sharing Features

Can users inspect features?

Tecton provides a variety of ways for your data scientists to inspect features. They can review the actual code of the feature, see summary statistics for all features, and query the feature's data using the Tecton SDK.

Can users register and discover features in Tecton?

Yes, with Tecton, you register the entire transformation logic, plus metadata around families, owners, custom tags, and more. The Tecton Web UI then allows users to access, browse and discover different families of features if you break them down by different use cases or filter down to different metadata tags that you can add on to these features.

How can users ensure there are no duplicate features ingested?

The Tecton Feature Store manages feature dependencies via the names of the objects that are configured for Tecton (eg, data sources, feature views, and services). It is possible to have users submit similar features with different names; we would recommend users first look to reuse features that exist in the feature store.

Handling Nulls

Does Tecton support null feature values?

Yes. Tecton supports nulls for feature values and for request data fields. Null values may be returned when data is missing (e.g. for a brand new user), when a materialized feature view column computes null (e.g. SELECT NULLIF(a, b)), or when an On-Demand Feature View returns None.

Nulls may also be members of arrays, e.g. ["foo", null], or members of structs.

Numeric null inputs in Spark Pandas On-Demand Feature Views

Note

If you expect to use numeric nulls, Python mode (mode="python") is strongly recommended.

Spark offline Pandas-mode (mode="pandas") On-Demand Feature Views inputs have special handling for numeric (i.e. Integer or Float) null values; numeric null inputs are cast to NaN.

By contrast, when running online (i.e. serving a production HTTP request), in Python mode, or on Tecton on Snowflake, numeric nulls are provided as None like all other data types.

See the following example given the request input to this feature view are provided as {input_int: null, input_float: null}:

from tecton import RequestSource, on_demand_feature_view
from tecton.types import Int64, Float64, Field

request = RequestSource([Field('input_int', Int64), Field('input_float', Float64)])

@on_demand_feature_view(
    sources=[request],
    mode="pandas",
    schema=[Field('output_int', Int64), Field('output_float', Float64)],
)
def numeric_null_example(request_df):
    import pandas

    print(request_df["input_int"][0]) # `nan` in Spark offline. `None` in all other cases.
    print(request_df["input_float"][0]) # `nan` in Spark offline. `None` in all other cases.

    # `None` values in the output features are correctly handled as `null` in all cases.
    return pandas.DataFrame.from_records([{
        "output_int": None,
        "output_float": None
    }])