Orchestrators / Schedulers¶
Tools to build complex pipelines of batch jobs. They handle dependency resolution, workflow management, visualization.
Links¶
- Luigi vs Airflow vs Pinball
- Airflow Documentation
- Luigi
- Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances
- Snowplow Documentation
Modern Tools¶
Airflow¶
Python-based orchestration framework using DAGs.
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime
with DAG("modern_dag", start_date=datetime(2023, 1, 1), schedule_interval="@daily") as dag:
task1 = BashOperator(task_id="print_date", bash_command="date")
Dagster¶
Data orchestrator for machine learning, analytics, and ETL. Focuses on data assets.
Prefect¶
Modern orchestration tool with native Python semantics and a hybrid execution model.