In the context of Data Engineering, workflow orchestration refers to the process of scheduling and arranging tasks that form your [[Data Pipeline|data pipeline]]. A workflow orchestration tool allows you to schedule, run, and observe the entire process. ## Popular Workflow Orchestration Tools [[Apache Airflow]] [[Dagster]] [[Prefect]] ## Workflow Orchestration Advantages - Create complex custom workflows - Makes it easier to create [[Idempotence|idempotent]] workflows - Alert you if something fails - Allows you to gracefully retry and recover from failures ## Workflow Orchestration Disadvantages - Adds complexity in scheduling - Requires additional infrastructure and maintenance costs %% wiki footer: Please don't edit anything below this line %% ## This note in GitHub <span class="git-footer">[Edit In GitHub](https://github.dev/data-engineering-community/data-engineering-wiki/blob/main/Concepts/Workflow%20Orchestration.md "git-hub-edit-note") | [Copy this note](https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Concepts/Workflow%20Orchestration.md "git-hub-copy-note")</span> <span class="git-footer">Was this page helpful? [👍](https://tally.so/r/mOaxjk?rating=Yes&url=https://dataengineering.wiki/Concepts/Workflow%20Orchestration) or [👎](https://tally.so/r/mOaxjk?rating=No&url=https://dataengineering.wiki/Concepts/Workflow%20Orchestration)</span>