流程引擎>> airflow-tutorial>> 返回
项目作者： dmlogv

项目描述 ：
Learning resources for Airflow Tutorial article.
高级语言： Python
项目地址: git://github.com/dmlogv/airflow-tutorial.git
创建时间： 2020-07-21T12:25:17Z
项目社区：https://github.com/dmlogv/airflow-tutorial
开源协议：
下载


Airflow Tutorial

Learning resources for Airflow Tutorial article.

Contents

  • docker-compose.yml — An example of Airflow cluster with Celery executor. Contains:
    • Apache Airflow
    • PostgreSQL (Airflow metadata)
    • Redis (Task broker)
    • Celery workers
    • Flower (Celery monitoring)
  • docker-compose.db.yml — Additional database servers and sample data fill up:
    • SQL Server x3 (source database servers)
    • Vertica (target database)
    • mssql_init (initialize source data)
  • dags/ — Sample DAGs and common libraries.

Start and stop

To spin up Airflow cluster only (without databases), use:

  1. $ docker-compose up --scale worker=3

To run all described images and create sample databases, execute:

  1. $ docker-compose -f docker-compose.yml -f docker-compose.db.yml up --scale worker=3

To break down containers press Ctrl+C or Command+C and the following command:

  1. $ docker-compose down

or

  1. $ docker-compose -f docker-compose.yml -f docker-compose.db.yml down

Usage

Containers exposes a couple of WebUI’s: