2.2.1 - What is Orchestration?

Architecture

• Extract
  • Pull data from a source
• Transform
  • Data cleaning, transformation, and partitioning
• Load
  • API to Mage, Mage to Postgres, GCS, BigQuery

What is Orchestration?

• A process of dependency management, facilitated through automation
• The data orchestrator manages scheduling, triggering, monitoring, even resource allocation
• Steps = tasks
• Workflows = DAGs (directed acyclic graphs)

A good orchestrator handles

• Workflow management
• Automation
• Error handling
• Recovery
• Monitoring, alerting
• Resource optimization
• Debugging
• Compliance/Auditing