Spark Execution Architecture
Understanding the complete Spark execution flow is essential for debugging, optimizing, and answering architecture questions. This diagram shows the journey from user application code to final results.
User / Application
Your PySpark application code (Python scripts, notebooks, or JARs) that defines data processing logic using transformations and actions.
Driver Program
The driver is the central process that creates the SparkSession, plans the execution, and coordinates work across executors. It runs on the client machine or a cluster node.
Logical Plan
Spark parses your code and creates an Abstract Syntax Tree (AST), then converts it into a logical plan representing the requested operations without optimization.
Optimization
The Catalyst Optimizer applies rule-based and cost-based optimizations to transform the logical plan into an optimized physical plan for efficient execution.
DAG (Directed Acyclic Graph)
The optimized execution plan represented as a DAG. Each node represents an operation, and edges represent data flow. Spark uses this to schedule and parallelize work.
Stages
Spark divides the DAG into stages at shuffle boundaries. Each stage contains tasks that can execute in parallel without data movement.
Tasks
Tasks are the smallest units of execution. Each task processes one partition of data. Tasks within a stage are identical but run on different data partitions.
Cluster Manager
Manages cluster resources (YARN, Mesos, Kubernetes, or Standalone). It allocates executor containers and monitors their health.
Executors
JVM processes running on worker nodes that execute tasks, store data in memory/disk, and report status back to the driver.
Task Execution
Executors run tasks in parallel across their available cores. Each task reads its partition, applies transformations, and writes output.
Results
Processed data is collected, written to storage (HDFS, S3, databases), or returned to the driver depending on the action performed.
Driver (Result Collection)
The driver receives results from executors, aggregates them if needed, and makes them available to the user application.
Driver Program
The Driver Program is the heart of a Spark application. It is the process that runs the main() function of your application and is responsible for:
- Reading the Spark application code
- Creating
SparkContext/SparkSession - Communicating with the cluster manager
- Requesting resources (executors, cores, memory)
- Scheduling work (tasks)
- Receiving and aggregating results
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.master("yarn")
.appName("MyApp")
.getOrCreate()
)Logical Plan
When Spark first processes your application code, it creates a Logical Plan that represents the requested operations. This plan goes through two phases:
Spark Code
Your PySpark application code that defines data processing operations using DataFrames, RDDs, or SQL.
Parse
Spark parses the code and creates an Abstract Syntax Tree (AST), then converts it to an unresolved logical plan where column names and types are not yet validated.
Logical Plan
A tree representation of the operations requested by the user. After analysis (resolving columns, tables), it becomes an analyzed logical plan.
Parsed vs Unresolved Logical Plans
| Plan Type | Description |
|---|---|
| Parsed Logical Plan | Created after parsing. Column names and table references are not yet resolved. |
| Unresolved Logical Plan | Spark attempts to resolve references against the catalog. If a column or table doesn't exist, an error is thrown. |
| Analyzed Logical Plan | All references are resolved. Types are checked. This plan is ready for optimization. |
df.explain() to inspect the plan at any stage.
Catalyst Optimizer
The Catalyst Optimizer is Spark's query optimization framework. It automatically rewrites your queries to execute more efficiently without changing the results.
Query
The user's SQL query or DataFrame operation that Spark needs to execute.
Logical Plan
The analyzed logical plan representing the user's requested operations.
Catalyst Optimizer
Applies rule-based and cost-based optimizations to transform the logical plan into an optimized physical plan.
Optimized Plan
The optimized logical plan with reduced computation, fewer shuffles, and better data access patterns.
Physical Execution
The final execution plan with concrete operations that will run on the cluster.
Key Optimizations
Predicate Pushdown
Column Pruning
Constant Folding
WHERE age > 5 + 3 becomes WHERE age > 8.
Join Reordering
DAG — Directed Acyclic Graph
The DAG (Directed Acyclic Graph) is the optimized execution plan that Spark creates after the Catalyst Optimizer finishes. It represents the complete execution flow of your application.
Read Data
Spark reads data from the source (files, tables, streams) and partitions it across executors.
Filter
Applies filter conditions to reduce the dataset. This is a narrow transformation (no shuffle).
Map
Applies a function to each element. Also a narrow transformation that can be pipelined.
Shuffle
Data redistribution across partitions. This is a wide transformation that creates a new stage boundary.
GroupBy
Groups data by key, typically requiring a shuffle to bring same-key records together.
Aggregation
Computes aggregate functions (sum, count, avg) on grouped data.
Stages
Spark divides the DAG into stages at shuffle boundaries. Each stage contains a set of tasks that can execute in parallel without data movement.
Read (Stage 0)
Reading data from the source. Partitions are read in parallel across executors.
Filter (Stage 0)
Applying filter conditions. This is a narrow transformation that stays within the same stage.
Map (Stage 0)
Applying a transformation function. Also stays within Stage 0 as it's a narrow transformation.
GroupBy (Stage 1)
After the shuffle, data is grouped by key. This starts a new stage.
Aggregation (Stage 1)
Computing aggregates on the grouped data within Stage 1.
Tasks and Executors
Understanding how tasks and executors work is key to parallelism in Spark:
- A stage contains multiple tasks
- Tasks can execute in parallel
- Partitions are processed by tasks
- Executor cores execute tasks
Core 1 → Task
Each executor core can run one task at a time. The task processes one partition of data.
Core 2 → Task
Multiple tasks run in parallel across available cores, enabling horizontal scaling.
Core 3 → Task
Each task is independent and processes its own partition without sharing state.
Core 4 → Task
More cores mean more parallelism. The number of tasks in a stage equals the number of partitions.
Adaptive Query Execution (AQE)
Adaptive Query Execution (AQE) is a Spark 3.0+ feature that optimizes queries at runtime based on actual data statistics, rather than just static estimates.
Initial Plan
The query plan created by Catalyst before execution. May contain suboptimal choices based on estimated statistics.
Execution
Execution begins. As data flows through operators, actual statistics are collected.
Runtime Statistics
Real data sizes, partition counts, and skew information are gathered during execution.
Adaptive Optimization
AQE re-optimizes the remaining plan based on actual runtime statistics, making better decisions.
Improved Execution
The remaining stages execute with optimized plans, improving overall performance.