Learn to programmatically author^ schedule and monitor workflows with Apache Airflow. Deploy to Kubernetes in AWS.
- Advanced tips for production
- Create your first pipeline
- Create ETL pipeline using Pandas
- Build Docker image for Apache Airflow
- Create helm chart for Apache Airflow
- Deploy Airflow to Kubernetes in AWS
- Basic Airflow components - DAG^ Plugin^ Operator^ Sensor^ Hook^ Xcom^ Variable and Connection
- Advance in branching^ metrics^ performance and log monitoring
- Run development environment with one command through Docker Compose
- Run development environment with one command through Helm and Kubernetes
- The difference between Sequential^ Local^ Celery and Kubernetes Executors
- Understand Apache Airflow s configuration properties
- Investigate Apache Airflow s REST Api
- Explore Apache Airflow s web interface
Apache Airflow is an open-source platform to programmatically author^ schedule and monitor workflows. In this course we are going to start with covering some basic concepts related to Apache Airflow - from the main components - web server and scheduler^ to the internal components like DAG^ Plugin^ Operator^ Sensor^ Hook^ Xcom^ Variable and Connection.
Later in the course I will teach you some more advanced topics like branching^ metrics^ performance and log monitoring^ and Airflow s REST ,API. Additionally I will help you to build your development environment with just one click using Docker and Docker Compose.
Why stop here? After all this^ we will create a Kubernetes cluster in Amazon and we will deploy our application there!
Finally^ I will share with you some useful advanced tips which will be helpful to enhance your simple Airflow project to a production ready system.