Narrow Transformations
Common narrow transformations include:
# Narrow transformations — each partition maps to one output partition
map() # Apply function to each element
filter() # Keep elements matching condition
flatMap() # Map + flatten
mapPartitions()# Apply function per partition
union() # Combine two RDDsContains a subset of the total dataset. Each partition is processed independently by a single task on an executor.
Another independent slice of data. No dependency on Partition 1 or 3 — true parallelism.
Each partition can even apply a different narrow transformation — Spark handles this transparently.
Each input element produces exactly one output element. The function runs locally on the executor — no data movement across the network.
Evaluates a predicate on each element. Elements that pass are kept; others are discarded. Still one-to-one — no shuffle.
Result of applying map() to Partition 1. Same partition index, same executor — zero network I/O.
Processed independently. If this transformation were pipelined with another narrow op, both would run in the same task.
May have fewer elements than the input (filter removed some), but the partition boundary is preserved.
Key Characteristics
- One-to-one partition relationship
- No data movement between partitions
- Usually faster execution
- Can be pipelined together (fused into a single task)
Wide Transformations
Common wide transformations include:
# Wide transformations — data shuffle required
groupByKey() # Group values by key
reduceByKey() # Aggregate values by key
join() # Join two RDDs/DataFrames
sortByKey() # Sort by key
distinct() # Remove duplicates
repartition() # Change number of partitions
coalesce() # Reduce partitions (no full shuffle)Contains key-value pairs. During shuffle, data is re-partitioned by key — records with the same key must end up on the same output partition.
Data from this partition will be sent to multiple output partitions based on the partitioning key. This cross-partition data movement is the shuffle.
Same as above — records are hashed by key to determine their target partition. Network I/O is proportional to data volume.
The most expensive operation in Spark. Data is serialized, written to disk, transferred across the network, and deserialized. This creates a stage boundary in the DAG.
Contains all records (from all input partitions) whose key hashes to partition A. The number of output partitions is determined by spark.sql.shuffle.partitions (default 200).
After shuffle, each output partition is processed independently again. Wide transformations effectively "reset" the pipeline.
The new partitions may be on completely different executors than the original ones — this is why shuffle is expensive.
Key Characteristics
- Many-to-many partition relationship
- Requires data shuffle across network
- Usually slower execution
- Creates new stage boundaries in the DAG
reduceByKey instead of groupByKey to reduce the amount of data shuffled.Narrow vs Wide — Comparison
| Feature | Narrow | Wide |
|---|---|---|
| Shuffle Required | No | Yes |
| Data Movement | Low | High |
| Performance | Usually faster | Usually slower |
| Relationship | One-to-one | Many-to-many |
| Stage Boundary | No | Yes |
| Examples | map, filter, union |
groupBy, join, reduceByKey |
| Network I/O | Minimal | Significant |
| Pipelining | Yes — fused into one task | No — creates stage boundary |
Common Transformations
Applies a function to each element in the RDD and returns a new RDD.
rdd = sc.parallelize([1, 2, 3, 4, 5])
mapped = rdd.map(lambda x: x * 2)
print(mapped.collect()) # [2, 4, 6, 8, 10]Returns a new RDD containing only the elements that satisfy a given condition.
rdd = sc.parallelize([1, 2, 3, 4, 5])
filtered = rdd.filter(lambda x: x > 2)
print(filtered.collect()) # [3, 4, 5]Applies a function that returns an iterator for each element, then flattens the results into a single RDD.
rdd = sc.parallelize(["hello world", "hi spark"])
flat = rdd.flatMap(lambda x: x.split(" "))
print(flat.collect())
# ["hello", "world", "hi", "spark"]Aggregates values for each key using a reduce function. More efficient than groupByKey() because it combines data locally before shuffle.
rdd = sc.parallelize([("a", 1), ("b", 2), ("a", 3)])
reduced = rdd.reduceByKey(lambda a, b: a + b)
print(reduced.collect())
# [("a", 4), ("b", 2)]Groups all values for each key into an iterable. Generally less efficient than reduceByKey() because it shuffles all data before aggregation.
rdd = sc.parallelize([("a", 1), ("b", 2), ("a", 3)])
grouped = rdd.groupByKey()
for key, values in grouped.collect():
print(f"{key}: {list(values)}")
# a: [1, 3]
# b: [2]Returns a new RDD containing all elements from both input RDDs. Does not deduplicate — use distinct() for that.
rdd1 = sc.parallelize([1, 2, 3])
rdd2 = sc.parallelize([3, 4, 5])
combined = rdd1.union(rdd2)
print(combined.collect())
# [1, 2, 3, 3, 4, 5]Actions
Common Actions
Returns all elements of the RDD as a list to the driver.
rdd = sc.parallelize([1, 2, 3, 4, 5])
print(rdd.collect()) # [1, 2, 3, 4, 5]collect() on large datasets — it brings all data to the driver, which can cause out-of-memory errors.
Returns the number of elements in the RDD.
rdd = sc.parallelize([1, 2, 3, 4, 5])
print(rdd.count()) # 5Displays the first 20 rows of a DataFrame in a tabular format. Useful for quick inspection.
df.show()
# +---+-----+
# | id|name |
# +---+-----+
# | 1|Alice|
# | 2| Bob|
# +---+-----+Returns the first n elements of the RDD. Safer than collect() for large datasets.
rdd = sc.parallelize([1, 2, 3, 4, 5])
print(rdd.take(3)) # [1, 2, 3]Writes the RDD elements to a text file. Each partition creates one output file.
rdd.saveAsTextFile("output/path")
# Creates output/part-00000, output/part-00001, ...Complete DAG Example
Let's trace a complete example showing how Spark builds a DAG from transformations and actions:
# Step 1: Read data (narrow)
rdd = sc.textFile("data.txt")
# Step 2: Transform each line (narrow)
mapped = rdd.map(lambda x: (x.split(",")[0], x))
# Step 3: Filter (narrow)
filtered = mapped.filter(lambda x: x[0] == "key")
# Step 4: Group by key (wide — triggers shuffle!)
grouped = filtered.groupByKey()
# Step 5: Map over groups (narrow)
result = grouped.map(len)
# Step 6: Trigger execution (action)
result.collect()Visual DAG — Stages & Shuffle Boundaries
How Spark Decides Stages
- Spark scans transformations from the action backwards
- Narrow transformations are pipelined into the same stage
- When a wide transformation is encountered, a new stage boundary is created
- Each stage runs tasks in parallel — one per partition
- Stages are connected by shuffle dependencies