🔍 Ctrl+K
🔴 Advanced

Real-World Spark Problems & Solutions

In production Spark environments, you'll encounter various performance and reliability issues. Here are the most common problems and their solutions.

🚨

Data Skewness

Certain partitions contain significantly more data than others, causing uneven workload distribution.

Symptoms

  • One task takes much longer than others
  • OOM errors on specific executors
  • Overall job is slow despite having resources
Solutions ▼
Salting Technique

Add a random prefix to skewed keys to distribute data more evenly across partitions.

Repartition

Use repartition() to redistribute data more evenly before processing.

Broadcast Join

For small-large joins, broadcast the smaller DataFrame to avoid shuffle entirely.

AQE Skew Join Optimization

Enable AQE to automatically detect and handle skew at runtime.

Python 
from pyspark.sql.functions import rand, lit

# Salt the skewed key with random prefix
salted_df = skewed_df.withColumn(
    "salted_key",
    concat(col("key"), lit("_"), (rand() * 10).cast("int"))
)

# Repartition by salted key for even distribution
salted_df = salted_df.repartition(200, "salted_key")
🚨

Out-of-Memory (OOM)

Executor or driver runs out of memory during processing.

Symptoms

  • java.lang.OutOfMemoryError
  • Job fails with memory-related errors
Solutions ▼
Increase Memory

Increase driver/executor memory configuration.

Repartition

Reduce partition size by increasing partition count.

Coalesce

Reduce partitions where appropriate to avoid overhead.

Persist Appropriately

Use MEMORY_AND_DISK to spill to disk when needed.

Avoid collect()

Never collect() on large datasets - it brings everything to driver memory.

Broadcast Joins

Use broadcast joins to avoid large data movement across the network.

Python 
# Increase memory configuration
spark.conf.set("spark.executor.memory", "8g")
spark.conf.set("spark.driver.memory", "4g")

# Use MEMORY_AND_DISK persistence
df.persist(StorageLevel.MEMORY_AND_DISK)
🚨

Slow Jobs

Spark jobs take longer than expected to complete.

Diagnosis Steps

  1. Check Spark UI for long-running stages
  2. Look for large shuffles
  3. Check data skew
  4. Check partition count
  5. Check joins strategy
  6. Check caching/persistence
Solutions ▼
Cache/Persist

Cache or persist datasets that are reused multiple times.

Reduce Wide Transformations

Minimize unnecessary shuffles (wide transformations).

Repartition

Repartition when appropriate for parallelism.

Coalesce After Aggregation

Coalesce after aggregation when appropriate to reduce partitions.

Broadcast Joins

Use broadcast joins for small-large joins to avoid shuffles.

Optimize Data Skew

Detect and handle data skew using salting or AQE.

🚨

Small File Problem

Processing thousands of very small files creates overhead and degrades performance.

Why It's a Problem

  • Each file requires a task to process
  • Excessive task scheduling overhead
  • Poor I/O utilization
Solutions ▼
Compact Files

Merge small files before processing.

Use coalesce()

Reduce partitions by combining small files.

Use repartition()

Repartition to combine small files into evenly-sized partitions.

Optimized Formats

Use Parquet or ORC for better file management.

Configure maxRecordsPerFile

Control maximum records per output file during writes.

Python 
# Coalesce small files
df = df.coalesce(10)

# Or use repartition for even distribution
df = df.repartition(10)
🚨

Driver Failure

The driver program fails, usually due to excessive data being brought back to it.

Why It Happens

  • Using collect() on very large datasets
  • Too many variables/references in driver
  • Driver memory too low
Solutions ▼
Avoid collect()

Never collect() on large datasets.

Use take() or show()

Use take() for limited data or show() for display.

Increase Driver Memory

Increase spark.driver.memory configuration.

Broadcast Variables

Use broadcast variables for shared read-only data.

Avoid Accumulating Results

Don't accumulate large results in the driver.

Python 
# BAD - can cause driver OOM
all_data = df.collect()

# GOOD - use take for limited data
sample = df.take(100)

# GOOD - use show for display
df.show(20)
🚨

Slow Joins

Join operations are taking too long to complete.

Solutions ▼
Broadcast Small DataFrame

Broadcast the smaller DataFrame when appropriate.

Check for Data Skew

Verify join keys are not skewed.

Sort-Merge Join

Use sort-merge join for large-large joins.

Pre-partition by Join Key

Partition data by join key before joining.

Filter Before Joining

Reduce data size by filtering before the join.

Python 
from pyspark.sql.functions import broadcast

# Use broadcast for small-large joins
result = large_df.join(broadcast(small_df), "key")
🟡 Intermediate

Spark Execution Modes

Spark supports multiple execution modes for running applications. Understanding when to use each is important for deployment decisions.

Local Mode

Used primarily for development and testing. Runs Spark on a single machine with multiple threads.
Development Testing Debugging
master("local[*]")

Standalone Mode

Spark's own cluster manager. Simple to set up for small to medium clusters.
Small Clusters Simple Setup

YARN Mode

Cluster resource manager commonly associated with Hadoop environments. Leverages existing Hadoop infrastructure.
Enterprise Hadoop
master("yarn")

Kubernetes Mode

Runs Spark workloads on Kubernetes clusters. Containerized deployment.
Cloud-Native Containers
master("k8s://...")

Comparison Table

Mode Use Case Complexity Resource Management
Local Dev/Test Simple Single machine
Standalone Small clusters Medium Spark native
YARN Enterprise/Hadoop Medium Hadoop YARN
Kubernetes Cloud-native Complex Kubernetes
Interview Tip
Knowing when to use each execution mode and their trade-offs is a common interview topic.