🔍 Ctrl+K
🟡 Intermediate

Shuffle

🔀
Definition
Shuffle
Shuffle is the process of redistributing data across partitions. It occurs during wide transformations such as groupByKey(), reduceByKey(), join(), and repartition().
Shuffle Overview
Partition 1
Partition 2
Partition 3
→
SHUFFLE
→
New Partition 1
New Partition 2

When Does Shuffle Happen?

  • groupByKey(), reduceByKey(), join()
  • repartition(), coalesce() (with shuffle)
  • sortByKey(), distinct()
Interview Tip
💡 Shuffle is expensive — it involves disk I/O, data serialization, and network transfer. Minimize wide transformations where possible.
🔴 Advanced

Data Skewness

🚨
Definition
Data skew occurs when data is unevenly distributed across partitions, causing some partitions to have significantly more data than others.
Skewed Partition Distribution
Partition 1
1,000,000 records  ⚠️ OVERLOADED
Partition 2
100,000
Partition 3
100,000
Partition 4
100,000

Problems Caused by Data Skew

  • Uneven workload across executors
  • Longer execution time
  • Out-of-memory errors
  • Slow jobs
Interview Tip
💡 Data skew is one of the most common Spark performance issues. Always check partition sizes in the Spark UI.
🔴 Advanced

Salting Technique

💡
Definition
Salting is a technique to handle data skew by adding a random prefix (salt) to the skewed key, distributing data more evenly across partitions.

Step-by-Step Process

  1. Step 1: Identify the skewed key
  2. Step 2: Create a salt value (random prefix)
  3. Step 3: Redistribute the data
  4. Step 4: Run aggregation
  5. Step 5: Remove/merge the salt information
Salting Flow
Original:
A → 1,000,000 records
↓ SALTING
A_1 → Partition 1
A_2 → Partition 2
A_3 → Partition 3
A_4 → Partition 4
↓
Aggregation
↓
Combine Results
Python 
from pyspark.sql.functions import concat, lit, rand, floor, split

# Step 1: Add salt to skewed key
num_partitions = 4
salted_df = skewed_df.withColumn(
    "salted_key",
    concat(col("key"), lit("_"), (rand() * num_partitions).cast("int"))
)

# Step 2: Aggregate with salted key
result = salted_df.groupBy("salted_key").agg(...)

# Step 3: Remove salt and final aggregation
final_result = result.withColumn(
    "key",
    split(col("salted_key"), "_")[0]
).groupBy("key").agg(...)
Interview Tip
💡 Salting is the go-to solution for data skew. Understanding this technique demonstrates advanced Spark knowledge.
🟡 Intermediate

Cache and Persist

Cache

💡
Definition
Cache keeps data in memory for reuse across multiple actions.
Python 
df.cache()  # Same as df.persist(StorageLevel.MEMORY_ONLY)

When to use: When you plan to use the same DataFrame multiple times.

Persist

💡
Definition
Persist allows you to choose different storage levels for caching.
Python 
from pyspark import StorageLevel

# Memory only
df.persist(StorageLevel.MEMORY_ONLY)

# Memory and disk
df.persist(StorageLevel.MEMORY_AND_DISK)

# Memory only serialized
df.persist(StorageLevel.MEMORY_ONLY_SER)

Storage Levels

Storage Level Description
MEMORY_ONLY Store as deserialized objects in JVM heap
MEMORY_AND_DISK Spill to disk if doesn't fit in memory
MEMORY_ONLY_SER Store as serialized objects (compact)
DISK_ONLY Store only on disk

Unpersist

Python 
df.unpersist()
Interview Tip
💡 Cache/persist when you reuse a DataFrame. Unpersist when done to free memory.
🟡 Intermediate

Performance Optimization Checklist

✅
Checklist
Use this interactive checklist to ensure your Spark jobs are optimized. Click each item to track your progress.
🔴 Advanced

Catalyst Optimizer

💡
Definition
Catalyst is Spark's query optimization framework that automatically optimizes Spark SQL queries.
Catalyst Optimization Flow
Query
→
Logical Plan
→
Catalyst Optimizer
→
Optimized Plan
→
Physical Execution

Key Optimizations

Predicate Pushdown

Push filters as close to the data source as possible to reduce the amount of data read and processed.
Filter Early Reduce I/O

Column Pruning

Only read columns that are needed for the query, dramatically reducing memory usage and I/O.
Select Only Needed Reduce Memory

Constant Folding

Evaluate constant expressions at compile time rather than runtime. For example, WHERE age > 5 + 3 becomes WHERE age > 8.
Compile Time Simplify

Join Reordering

Optimize the order of joins to minimize intermediate result sizes. Smaller tables are joined first when possible.
Cost-Based Minimize Shuffles
Interview Tip
💡 Catalyst Optimizer automatically optimizes your queries. Understanding it helps you write better Spark code.
🔴 Advanced

Adaptive Query Execution (AQE)

💡
Definition
AQE can use runtime information to adapt and optimize execution plans during the query.
AQE Optimization Flow
Initial Plan
→
Execution
→
Runtime Statistics
→
Adaptive Optimization
→
Improved Execution

Key Features

Coalescing Post-Shuffle Partitions

Automatically combines small partitions after a shuffle to reduce overhead and improve utilization.
Reduce Partitions Better Utilization

Converting Sort-Merge Joins to Broadcast Joins

If one table turns out to be small at runtime, AQE can switch from sort-merge to broadcast join for better performance.
Join Strategy Runtime Switch

Optimizing Skew Joins

Detects and handles data skew by splitting large partitions and redistributing work more evenly.
Skew Detection Load Balancing

Configuration

Python 
# Enable AQE
spark.conf.set("spark.sql.adaptive.enabled", "true")
Interview Tip
💡 AQE is one of the most powerful features in modern Spark. It can dynamically optimize your query at runtime.