Pancake Stack: End-to-End, Real-time ML and AI Pipeline


InfoQ has an interesting read on the last-mile of theApache Spark machine learning pipeline, “Chris Fregly on the PANCAKE STACK Workshop and Data Pipelines”.

Its interesting to read that even Netflix and other struggled in this space:

The idea for PipelineIO dates back a few years to my time at Netflix where we were forced to build a custom ML prediction/serving layer. There wasn’t – and still isn’t, in my opinion – a production-ready, fault-tolerant, and low latency open source system to serve Netflix-scale predictions and recommendations in real-time

PipelineIO offers a good architecture overview on its home page:

architecture-overview-645x474

~ by mdavey on September 12, 2016.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: