Interview Preparation
Master PySpark interviews with these carefully curated questions organized by difficulty level. Each question includes a brief answer to help you prepare.
Next: Beginner Questions →Beginner Questions
Intermediate Questions
Advanced Questions
Real-World Spark Interview Simulator
Practice these real-world scenarios to prepare for system design and troubleshooting interviews.
Scenario: Your Spark job is taking 40 minutes instead of 5 minutes.
Open the Spark UI and navigate to the running or completed job. Look at the DAG visualization and stage details to understand where time is being spent.
Identify stages that take significantly longer than others. These are the bottleneck stages that need investigation.
Large shuffle read/write indicates expensive data movement. Check if the shuffle partitions count is appropriate (default 200 may be too high or too low).
Look at task metrics - if some tasks take much longer than others, data skew is likely the cause. Check for keys with disproportionately large amounts of data.
Too few partitions lead to underutilization; too many lead to overhead. Optimal partition size is 128MB-256MB per partition.
Verify if broadcast joins are being used where possible. Large-large joins using sort-merge may be slower than broadcast joins for small-large combinations.
Determine if data is being recomputed multiple times. Cache or persist data that is reused across multiple actions.
Apply the appropriate fix: repartition data, use broadcast joins, cache frequently accessed data, filter early, or tune configuration parameters.
Scenario: One task in your Spark job takes 10x longer than others.
Check the Spark UI task metrics. The slow task processes a partition with an disproportionately large amount of data, likely due to a single key with many records.
Run a groupBy on the suspect key and count records per key. Identify if a few keys dominate the dataset.
Add a random prefix to the skewed key to distribute it across multiple partitions. This spreads the load evenly.
Perform the aggregation using the salted key instead of the original key. This ensures parallel processing across multiple partitions.
Strip the salt prefix from the keys and perform a final aggregation to get the correct results.
Enable Adaptive Query Execution which can automatically detect and optimize skew joins in Spark 3.0+.
Scenario: Your Spark executor throws OutOfMemoryError.
Look at the failed tasks in the Spark UI. Identify which task failed and what data it was processing.
Check if certain partitions are significantly larger than others. This is often the root cause of OOM errors.
Adjust spark.executor.memory and spark.executor.memoryOverhead to allocate more memory per executor.
Use repartition() to split large partitions into smaller ones. Aim for 128MB-256MB per partition.
Instead of MEMORY_ONLY, use MEMORY_AND_DISK to spill data to disk when memory is insufficient.
collect() brings all data to the driver which can cause driver OOM. Use take(), show(), or write to storage instead.
Broadcast join avoids shuffling large DataFrames and reduces memory pressure on executors.
Scenario: Your Spark driver crashes when calling collect().
Locate where collect() is being used in your code. This is typically the root cause of driver OOM.
Use take(n) to get only a subset of records, or show() to display a preview. This avoids bringing all data to the driver.
Adjust spark.driver.memory to allocate more memory to the driver process.
For shared lookup data, use broadcast variables instead of collecting data to the driver.
Avoid accumulating results in driver-side variables. Write results directly to storage using write operations.
Check driver metrics in Spark UI to track memory usage and identify when memory pressure builds up.
Scenario: Your join operation is taking too long.
Look at the query plan in the Spark UI to see which join strategy is being used (broadcast, sort-merge, or shuffle hash).
If one side of the join is small enough (typically < 10MB), use broadcast join to avoid expensive shuffles.
Skewed join keys cause uneven partition sizes. Some tasks will take much longer, slowing the overall join.
Repartition both DataFrames by the join key before joining to reduce shuffle overhead.
Apply filters as early as possible to reduce the amount of data being joined.
Force broadcast join using .hint("broadcast") or the broadcast() function if you know one side is small.
Adaptive Query Execution can automatically convert sort-merge to broadcast joins based on runtime statistics.
Scenario: Your Spark job processes thousands of small files.
Many small files cause excessive task overhead, slow job startup, and underutilized resources. Each file creates a separate task.
Use coalesce() to reduce the number of partitions and combine small files into larger ones.
Columnar formats like Parquet and ORC handle small files better and support predicate pushdown for faster queries.
Set spark.sql.files.maxRecordsPerFile to limit the number of records per output file during writes.
Use combineInputFormat or similar to merge small files at the input level before processing.
Set up periodic compaction jobs to merge small files into larger, optimized files.