Covers: theory of ML Pipeline Tools
Estimated time needed to finish: 30 minutes
Questions this item addresses:
  • What is a batch model pipeline looks like?
How to use this item?
  • Section: 5.1 SKLearn Workflow
  • Section: 5.2 Cron
  • Section: 5.3 Workflow Tools
Author(s) / creator(s) / reference(s)
Ben Weber
Denys Linkov.
Maybe add something about it being framework agnostic just so people know
Denys Linkov.
I would add which components or technologies are used, python, airflow ect

General Pipeline Design

Total time needed: ~3 hours
With this recipe, you will learn what is a ML pipeline and how to design a general ML pipeline.
Potential Use Cases
When you have a ML model and you want to deploy it to production, this recipe can help you understand what is the end-to-end ML pipeline looks like.
Who is This For ?
INTERMEDIATEPractitioner in ML fields.
Click on each of the following annotated items to see details.
Resource Asset7/9
ARTICLE 1. What Is a Machine Learning Pipeline?
  • What is a machine learning pipeline?
  • What are the benefits of a machine learning Pipeline?
  • What to consider when building a machine learning pipeline?
15 minutes
ARTICLE 2. What is an ML pipeline and why is it important?
  • Why pipelining is so important?
  • How ML pipelines benefit performance and organization?
  • What are the use cases of an ML pipeline?
  • What are the challenges of an ML pipeline?
15 minutes
ARTICLE 3. Architecting a Machine Learning Pipeline
  • What are the things that a production-ready ML system should try to address?
  • How to architect a ML pipeline?
30 minutes
BOOK_CHAPTER 4. Building Machine Learning Pipelines
  • Why Machine Learning Pipelines?
  • When to Think About Machine Learning Pipelines?
  • Overview of the Steps in a Machine Learning Pipeline?
  • Pipeline Orchestration?
  • Our Example Project?
40 minutes
REPO 5. Building-ML-Pipelines / building-machine-learning-pipelines
  • Example of a ML pipeline?
30 minutes
ARTICLE 6. Workflow Tools for ML Pipelines
  • What is a batch model pipeline looks like?
30 minutes
ARTICLE 7. A Complete Introduction to Apache Airflow
10 minutes
ARTICLE 8. Managed Apache Airflow on AWS — New AWS Service For Data Pipelines
10 minutes
ARTICLE 9. MLOps: Continuous delivery and automation pipelines in machine learning
10 minutes

Concepts Covered

0 comment