Image by editor
# Introduction
feature engineering Data science and machine learning are an essential process in any AI system, along with workflows. It involves the creation of meaningful explanatory variables from raw – and often rather dirty – data. The processes behind feature engineering can be extremely simple or highly complex, depending on the volume, structure, and diversity of the dataset as well as the machine learning modeling objectives. While the most popular Python libraries for data manipulation and modeling are Panda And scikit-learnTo enable somewhat basic and moderately scalable feature engineering, there are specialized libraries that go the extra mile in dealing with massive datasets and automating complex transformations, yet they are largely unknown to many.
This article lists 7 under-the-radar Python libraries that push the boundaries of large-scale feature engineering processes.
# 1. Speed up with NVTabular
First, we have NVIDIA-Merlin’s NVTBular: A library designed to apply preprocessing and feature engineering to datasets based on – yes, you guessed it! – Tabular. Its distinguishing feature is its GPU-accelerated approach that is designed to easily manipulate the very large-scale datasets required to train huge deep learning models. The library is specifically designed to help scale pipelines for modern recommender system engines based on deep neural networks (DNNs).
# 2. Automate with FeatureTools
Feature ToolsDesigned by Alteryx, the feature focuses on leveraging automation in engineering processes. This library implements Deep Feature Synthesis (DFS), an algorithm that creates new, “deep” features when relationships are analyzed mathematically. The library can be used on both relational and time series data, making it possible to generate complex feature generation with minimal coding burden on both.
This code excerpt shows an example of how DFS is implemented featuretools The library looks like this on the customers dataset:
customers_df = pd.DataFrame({'customer_id': (101, 102)})
es = es.add_dataframe(
dataframe_name="customers",
dataframe=customers_df,
index="customer_id"
)
es = es.add_relationship(
parent_dataframe_name="customers",
parent_column_name="customer_id",
child_dataframe_name="transactions",
child_column_name="customer_id"
)# 3. Parallel to Dusk
dusk It is growing in popularity as a library to make parallel Python calculations faster and simpler. The core recipe behind Dask is to scale traditional Pandas and Scikit-learn feature transformations through cluster-based computations, facilitating fast and affordable feature engineering pipelines on large datasets that would otherwise exhaust memory.
This article shows a practical Dask walkthrough for performing data preprocessing.
# 4. Adaptation with poles
Competing with Dask in terms of growing popularity, and with Pandas aspiring for a spot on the Python data science podium, we have polar: A Rust-based DataFrame library that uses lazy expression APIs and lazy computations to run efficient, scalable feature engineering and transformations on very large datasets. Considered by many as a high-performance counterpart to Pandas, Polar is very easy to learn and become familiar with if you are fairly familiar with Pandas.
Are you interested in learning more about Polar? This article showcases several practical Polar one-liners for common data science tasks, including feature engineering.
# 5. Stocking up with treats
feast is an open-source library envisioned as a feature store that helps distribute structured data sources for large-scale production-level or production-ready AI applications, especially those based on large language models (LLMs), for both model training and inference tasks. One of its attractive properties is to ensure consistency between both stages: training and estimation in production. Its use as a feature store is also closely linked with feature engineering processes, for example, by using it in combination with other open-source frameworks, denormalized.
# 6. Extracting with tsfresh
Focusing towards larger time series datasets, we have tsfresh Library, with a package that specializes in scalable feature extraction. Ranging from statistical to spectral properties, this library is capable of computing hundreds of meaningful features over large time series, as well as implementing relevance filtering, which, as its name suggests, involves filtering features based on relevance to the machine learning modeling process.
This example code excerpt takes DataFrame This includes a time series dataset that has previously been rolled out in Windows, and applies tsfresh Feature extraction on:
features_rolled = extract_features(
rolled_df,
column_id='id',
column_sort="time",
default_fc_parameters=settings,
n_jobs=0
)# 7. Getting along with the river
Let’s finish dipping our toes in the river current (within reason), with River The library is designed to streamline online machine learning workflows. As part of its functionalities, it has the ability to enable online or streaming feature transformation and feature learning techniques. It can help efficiently deal with issues like unlimited data and concept drift in production. River is designed to robustly handle problems that rarely occur in batch machine learning systems, such as the appearance and disappearance of data features over time.
# wrapping up
This article lists 7 notable Python libraries that can help make feature engineering processes more scalable. Some of them are directly focused on providing specific feature engineering approaches, while others can be used in conjunction with other frameworks, to pursue feature engineering tasks in certain scenarios.
ivan palomares carrascosa Is a leader, author, speaker and consultant in AI, Machine Learning, Deep Learning and LLM. He trains and guides others in using AI in the real world.
