Spark divides a dataset into partitions so that data can be processed in parallel. Each partition is a logical chunk of data that is processed by a single task.

Key relationships:

• 1 Partition → 1 Task — Each partition is processed by exactly one task.
• 1 Executor Core → 1 Task at a time — A core can process one partition concurrently.
• More partitions = more parallelism (but also more overhead).

Default partition count: When you read a file, Spark creates one partition per HDFS block (default 128 MB) or one partition per file. When using DataFrames, the default number of partitions is determined by spark.sql.shuffle.partitions (default 200).

Divides rows across partitions. Each partition contains a subset of the total rows.

Use case: Used by default in Spark for parallel processing of large datasets.

Divides columns across partitions. Each partition contains a subset of the total columns.

Use case: Useful when you frequently access only a subset of columns.

Determines partition assignment by applying a hash function to the partition key.

Records with the same key are sent to the same partition, ensuring co-location of related data.

from pyspark.sql.functions import hash, col

df = spark.createDataFrame([
    ("Alice", 100), ("Bob", 200), ("Charlie", 300)
], ["name", "amount"])

# Hash partition by name
df.repartition(4, "name")

Use case: Commonly used for joins and aggregations to co-locate related data.

Divides data according to value ranges. Each partition covers a specific range of values.

from pyspark.sql.functions import col

# Range partition by age
df.repartitionByRange(3, col("age"))

Use case: Useful for range-based queries and maintaining sorted order.

Records are distributed sequentially across partitions in a circular manner.

# Round-robin distribution (default repartition behavior)
df = df.repartition(4)

Use case: Used when there is no specific partitioning key and you want even distribution.

Custom partitioning allows developers to define their own partitioning logic according to application requirements.

from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

def custom_partitioner(key):
    # Custom logic to determine partition
    if key.startswith("A"):
        return 0
    elif key.startswith("B"):
        return 1
    else:
        return 2

Use case: When standard partitioning strategies don't meet your specific data distribution needs.