Induction to Apache Airflow
That repository
aims at providing end-to-end examples introducing how to setup, run and
monitor workflows, mainly for data processing pipelines.
In those tutorials, Airflow is used. A full end-to-end example is explained
step by step, and actually used for the Open Travel Data (OPTD)
project.
The full details on how to setup an Airflow server on Proxmox LXC containers
are given in the dedicated airflow/ sub-folder.
Such an Airflow cluster is actually the orchestration engine of the
Open Travel Data (OPTD)
project.
For convenience, most of the Airflow examples are demonstrated both on a local
single-node installation (e.g., on a laptop) and on on the above-mentioned
cluster.
Table of contents generated with markdown-toc
airflow/