Description

Apache Airflow Training in Bangalore Offered by Locus IT with 100% Hands on Practical Classes with Best Industrial Experts and working on Real Time Projects. In this course, we will understand Airflow basic concepts first to write our first DAG and then move forward with more intermediate & advance topics and understand how to troubleshoot, monitor in Airflow.

Audience:

This course is designed for Data Engineers & Data Platform team who want to run their pipelines using Apache Airflow.

 Prerequisites:

• Basic understanding of python.

 Course Objectives:

After this course, you will be able to:

• Understanding DAGs created in Airflow.
• Understand various concepts for better execution.
• Distributing, monitoring workloads on Airflow.
• Understanding troubleshooting in Airflow
• Understand various advance concepts such as templating, dependent DAGS and many more.

Course Outline

1- Apache Airflow Fundamentals

• What is Apache Airflow?
• How Airflow works
• Airflow UI Tour
• Airflow Architecture & Components

2- DAG Fundamentals

• What is a DAG?
• Writing your first DAG
• DAG Skeleton
• DAG Scheduling
• How the Scheduler Works

3– Using Operators

• What is operator?
• Using SQLite Operator
• Using python Operator
• Using Bash Operator
• Running First pipeline with various operators

 4- Configuration

• Setting Configuration Options
• Set up a Database Back-end
• Choosing database back-end
• Setting up a SQLite Database
• Setting up a PostgreSQL Database

5– Connection Management

• Creating a Connection from the CLI
• Exporting Connections from the CLI
• Storing a Connection in Environment Variables
• Connection URI format
• Securing Connections

 6- Mastering DAG

• Manipulating start date & schedule interval
• DAG Back-filling and Catch-up
• Dealing with time zones
• Task dependencies between DagRuns
• Retry & Alerting
• Making DAGs time zone aware

7- Distributing Apache Airflow

• Using Sequential Executors
• Local Executor with Postgress
• Scale out with Celery Executor
• Adding new workers with Celery Executors
• Prioritizing tasks: pools and priority weights
• Using Kubernetes Executors

8- Using Kubernetes with Airflow

• Containers Overview
• Kubernetes Overview
• Scaling with Kubernetes

9- Deploying Airflow on AWS EKS

• Overview of EKS
• Creating an EKS Cluster
• Deploy & Run Airflow with Kubernetes Executors on EKS.
• Using Official Airflow Helm Chart
• Using Managed Airflow Services
• Using 3rd-party images, charts, deployments

10- Monitoring & Troubleshooting

• Troubleshooting from Airflow UI
• Dealing with Failures
• How Logging works in Airflow
• Setting up custom logs
• Storing logs in AWS S3
• Configuring Airflow with Elasticsearch
• Monitoring Airflow with Prometheus & Grafana

11- Advance Concepts

• Branching in DAGS
• Conditional Tasks for your branch
• Defining Trigger Rules
• Using XCOM variables
• Trigger DAG from another DAG
• Templating
• Skipping

12-   Security

• Encrypting sensitive data with Fernet
• Rotating the Fernet Key
• Hiding variables
• Password authentication and filter by owner
• RBAC UI

Please Visit Apache Airflow Official Site: || Locus Academy has more than a decade experience in delivering the training/staffing on Apache Airflow for corporates across the globe. The participants for the training/staffing on Apache Airflow are extremely satisfied and are able to implement the learnings in their on going projects.