Inside Axion, The Feature Store Architecture Powering Machine Learning Pipelines at Netflix

The feature store is a key component of Netflix’s Machine Learning Platform.

Source: https://www.scrabbl.com/how-netflix-scaled-its-marketing-game-using-machine-learning

I recently started an AI-focused educational newsletter, that already has over 125,000 subscribers. TheSequence is a no-BS (meaning no hype, no news etc) ML-oriented newsletter that takes 5 minutes to read. The goal is to keep you up to date with machine learning projects, research papers and concepts. Please give it a try by subscribing below:

TheSequence

Netflix hosts one of the most sophisticated machine learning(ML) infrastructures in the world. Runnings thousands of ML models both real time and batch requires a sophisticated data and experimentation infrastructure. Over several iterations, Netflix built Axion, a feature store designed to accelerate offline experimentation and model training in ML solutions at Netflix.

Axion is part of a core ML platform at Netflix responsible for serving thousands of models across different applications. Axion interacts with several components of the ML platform fundamentally focused on feature generation and serving.

Axion has several interaction points with other components of Netflix’s ML platform:

· Fact: These are atomic data points about users or videos in Netflix. For instance, a fact could reflect the videos added to the users’ preference list. Facts are typically used by compute applications to generate the corresponding features.

· Compute Application: These are applications that process facts and generate features used by ML models.

· Online Feature Generator: These are components that compute features used in real time ML applications.

· Offline Feature Generator: These components are Spark jobs that run long computation processes in order to calculate features.

· Share Feature Encoders: These are components that are shared between compute…