Airflow Interview Q&A

Interview Answer: A DAG (Directed Acyclic Graph) is a workflow that defines the sequence of tasks in Apache Airflow. Each task represents a step in the pipeline, and dependencies decide the execution order. Airflow schedules the DAG, executes the tasks, and monitors their status. I use DAGs to automate ETL pipelines.

Interview Answer: XCom (Cross-Communication) is used to share small amounts of data between Airflow tasks. xcom_push() stores a value from one task, and xcom_pull() retrieves that value in another task. I use XCom to pass file names, IDs, or status information between tasks.

# Task 1
ti.xcom_push(key="file_name", value="sales.csv")
# Task 2
file = ti.xcom_pull(key="file_name")

Interview Answer: A DAG can be triggered in multiple ways. It can run on a schedule using cron expressions, be triggered manually from the Airflow UI, through the Airflow CLI, using the REST API, or by another DAG using TriggerDagRunOperator. The method depends on the project requirement.

Interview Answer: If a task fails, Airflow can retry it automatically based on the retry configuration. We define the number of retries and the retry delay in the DAG. If the task still fails after all retries, it is marked as failed and alerts can be sent.

Interview Answer: Apache Airflow is mainly used for scheduling and orchestrating ETL and data pipelines. AWS Step Functions are used to coordinate AWS services like Lambda, Glue, and ECS in serverless workflows. I use Airflow for complex data workflows and Step Functions for AWS event-driven applications.

Interview Answer: Dependencies define the order in which tasks run. In Airflow, I use >> or << operators, or methods like set_upstream() and set_downstream(). This ensures that one task starts only after the previous task completes successfully.

task1 >> task2 >> task3
or
task2.set_upstream(task1)
task2.set_downstream(task3)