Install & Run the PySpark Notebooks

1. Prerequisites

• Python 3.8+ — download
• Java 8 or newer (JDK) — PySpark needs a JVM. Install a JDK and set JAVA_HOME.

Check versions:

python --version
java -version

2. Install dependencies

From the project root (where requirements.txt lives):

# (optional but recommended) create a virtual environment
python -m venv .venv
# Windows:
.venv\Scripts\activate
# macOS / Linux:
source .venv/bin/activate

# install PySpark + Jupyter
pip install -r requirements.txt

requirements.txt contains: pyspark, jupyterlab.

3. Launch Jupyter and open a notebook

jupyter lab

This opens http://localhost:8888 in your browser. Then:

• In the file browser, go to notebooks/pyspark/.
• Open any NN-*.ipynb (e.g. 01-rdd-dataframe.ipynb).
• Menu Run → Run All Cells (or click play on each cell).

You'll see real PySpark output (runs in local mode — no cluster needed).

4. Verify PySpark works

In a notebook cell, run:

from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").appName("test").getOrCreate()
df = spark.createDataFrame([(1, "alice"), (2, "bob")], ["id", "name"])
df.show()
spark.stop()

Troubleshooting

• "java not found" / JVM error: install a JDK and set JAVA_HOME to its path, then restart your terminal.
• "pyspark: command not found": make sure your virtual environment is activated and pip install finished without errors.
• Notebook download instead of opening: you opened the .ipynb from GitHub directly. Download it, then open via jupyter lab as above.
• Low memory: use master("local[2]") instead of local[*] to limit cores.

Related

SQL → PySpark live converter  ·  Back to lessons