Apache Spark is a unified analytics engine designed for large-scale data processing. It provides a simple programming interface for distributed computing and handles both batch processing and real-time streaming workloads.

Key characteristics of Spark:

• In-memory computation — Data is kept in RAM across the cluster, making Spark up to 100x faster than disk-based engines like Hadoop MapReduce.
• Distributed processing — Data is split into partitions across multiple nodes, allowing parallel execution.
• Multi-language support — APIs are available for Python (PySpark), Scala, Java, R, and SQL.
• Rich ecosystem — Includes Spark SQL for structured data, Spark Streaming for real-time data, MLlib for machine learning, and GraphX for graph processing.

Spark was developed at UC Berkeley's AMPLab in 2009 and later donated to the Apache Software Foundation. It has become the most widely used engine for big data processing in production environments worldwide.

PySpark is the Python API for Apache Spark. It allows Python developers to leverage Spark's distributed computing capabilities without writing Scala or Java code. PySpark provides a high-level abstraction over Spark's core RDD API and includes the DataFrame and Dataset APIs for working with structured data.

With PySpark you can:

• Process terabytes of data across a cluster using familiar Python syntax.
• Use Spark SQL to run SQL queries on large datasets.
• Build machine learning pipelines with Spark MLlib.
• Stream real-time data with Structured Streaming.
• Integrate with popular Python libraries like Pandas and NumPy via Pandas UDFs.

PySpark communicates with the JVM-based Spark engine through a lightweight wrapper, so Python code is translated into Spark jobs that run on the cluster.

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("MyFirstApp") \
    .master("local[*]") \
    .getOrCreate()

print(spark.version)

SparkSession.builder is the entry point for creating a Spark application. The .master("local[*]") call tells Spark to run locally using all available CPU cores. The .getOrCreate() method returns an existing SparkSession or creates a new one if none exists. After creating the session, you can read data, apply transformations, and run actions on distributed datasets.

🟡 Intermediate

RDD — Resilient Distributed Dataset

RDD: Resilient, Distributed, Dataset

☆ Mark as Complete

An RDD is the fundamental data structure in Spark. It is an immutable, partitioned collection of elements that can be operated on in parallel. Every RDD is characterized by three core properties:

Resilient: An RDD can recover lost partitions through its lineage graph. If a partition is lost due to a node failure, Spark recomputes it by replaying the sequence of transformations that originally created it.

Distributed: The data stored in an RDD is spread across multiple nodes in the cluster. Each node holds a portion of the data called a partition, enabling parallel processing.

Dataset: A collection of partitioned data with primitive values or values of values (e.g., tuples, objects). The data is split into partitions that are processed independently.

Additional properties of RDDs:

• Immutability — Once created, an RDD cannot be modified. Each transformation produces a new RDD.
• Fault tolerance — Lost partitions are recomputed automatically via the lineage graph, without requiring data replication.
• Lazy evaluation — Transformations are not executed immediately. Spark builds a plan and only computes when an action is called.
• In-memory computation — Data is processed in RAM, which eliminates disk I/O overhead and dramatically improves performance.
• Parallel processing — Multiple partitions are processed simultaneously across the cluster's nodes.

rdd = spark.sparkContext.parallelize(
    [1, 2, 3, 4, 5]
)

print(rdd.collect())  # [1, 2, 3, 4, 5]
print(rdd.count())    # 5

parallelize() creates an RDD from a local Python collection. The data is distributed across partitions in the cluster. The collect() action returns all elements as a list, and count() returns the total number of elements. Note that collect() brings all data to the driver, so it should only be used on small result sets.

Apache Spark provides three core APIs for working with distributed data. Understanding their differences is one of the most asked interview questions.

Comparison Table

# RDD - no schema
rdd = spark.sparkContext.parallelize([(1, "Alice"), (2, "Bob")])
rdd.map(lambda x: x[1].upper()).collect()

# DataFrame - with schema, optimized
df = spark.createDataFrame([(1, "Alice"), (2, "Bob")], ["id", "name"])
df.filter(df.id > 1).show()
df.explain()  # shows Catalyst plan

# Dataset - Scala only (type-safe)
# case class Person(id: Int, name: String)
# val ds = spark.createDataset(Seq(Person(1,"Alice")))

Lazy evaluation means that Spark does not execute transformations immediately when they are called. Instead, Spark records the sequence of transformations as a lineage graph and only performs computation when an action (such as collect(), count(), or show()) is invoked.

This approach gives Spark the opportunity to optimize the entire execution plan before any data is actually processed. The Catalyst optimizer can reorder operations, combine steps, and eliminate unnecessary work.

Consider this transformation — it is recorded but not executed:

# Transformation - NOT executed yet
filtered_df = df.filter(df.salary > 50000)

The following action triggers execution. Spark reads the lineage, applies optimizations, and runs the computation:

# Action - TRIGGERS execution
filtered_df.show()

Lazy evaluation provides several benefits: it reduces unnecessary computation, enables query plan optimization, and allows Spark to minimize data movement across the cluster by fusing operations together.

Spark operations are divided into two categories: transformations and actions. Understanding the distinction between them is essential for writing efficient Spark code and for explaining how Spark's execution model works.

Transformations are lazy operations that create a new dataset from an existing one. They build up the lineage graph but do not trigger any computation. Transformations can be narrow (each input partition contributes to at most one output partition, like map() and filter()) or wide (input partitions contribute to multiple output partitions, like reduceByKey() and join(), which require a shuffle).

Actions are operations that trigger Spark to execute the lineage graph and produce a result. They either return data to the driver (collect(), count(), take()) or write data to external storage (save()). When an action is called, Spark optimizes the full chain of transformations and executes them in an efficient order.