Bookmarks

🟢 Easy

Why Python for PySpark?

For Data Engineering, Python is the control plane: you write PySpark transforms in Python, handle API ingestion, and glue AWS services. Here are six key features — and why they matter in a pipeline.

1. English-like (40% English)

df.filter(col('salary') > 50000) reads like English — faster onboarding for Spark jobs.

2. Interpreted

Line-by-line execution → instant feedback in notebooks (python -m pyspark, Databricks). No compile step.

3. Dynamically Typed

x=1 then x=[1,2,3] then x='praveen' — no declaration. Fast for prototyping, but needs validation in ETL.

4. Large Community

Libraries (pandas, boto3, pyspark) fix bugs faster than building from scratch.

5. Cross-Platform

Same code on Windows/Mac/Linux — critical moving from local to EMR/Glue.

6. OOP

Class = blueprint (e.g. class BankAccount). Model pipelines as reusable objects.
💡 How Python handles it
Python hides binary — you write x='praveen' and the interpreter handles translation. No need to think about bits.
🟢 Easy

Variables & Naming

Variable names are how you label data — Python enforces 4 rules to keep the parser simple. Break a rule and you get SyntaxError before code even runs. In PySpark, the same rules apply to DataFrame column names (df.withColumn("123col", ...) fails).

Valid vs invalid — why each rule exists
✅ Valid
first_name — letters + underscore
_myvar — can start with _
MYVAR — capitals allowed (constants)
myvar123 — letters, digits, _
Why? Only a-z A-Z 0-9 _ — other symbols like - % & are operators.
❌ Invalid — SyntaxError
123myvar — cannot start with digit (parser thinks it's a number)
%myvar, my-var — symbols are operators
def = 10 — def is a keyword, reserved
Case-sensitive: first_name and First_Name are two different variables. praveen ≠ Praveen ≠ PRAVEEN. Bug source #1 in ETL.
Python — try it
# Good
first_name = 'praveen'
phone_number = 112131
_myvar = 'ok'
MYVAR = 'ok'      # constant style, still valid
myvar123 = 'ok'

# Bad — uncomment to see SyntaxError
# 123myvar = 'bad'   # cannot start with number
# %myvar = 'bad'     # only a-z A-Z 0-9 _
# my-var = 'bad'
# my&var = 'bad'

# Case-sensitive & keyword trap
first_name = 'praveen'
First_Name = 'different variable!'  # not the same!
print(first_name, First_Name)  # praveen different variable!
# def = 10  # SyntaxError: keywords like def, for, if cannot be reused
# Use: my_def, my_for instead
Output
praveen different variable!
✅ PEP8 in practice
first_name not firstName; max 79 chars per line improves PR diffs. Two words → join with _. PySpark hygiene: bad column name 123col also fails in Spark SQL.
🟢 Easy

Comments, Docstrings & Indentation

Python uses 3 ways to add notes — pick the right one: # for quick human notes (ignored), '''triple quotes''' for multi-line notes, and docstring (first string inside a function) for tools like help() . Indentation (spaces after :) is not style — it's syntax; wrong indent = IndentationError.

Which note to use when?
# comment
Human only — interpreter skips the line. Use for “why”.
# printing name praveen
'''docstring'''
Multi-line — also ignored unless it's the first line of a function.
""" prints name """
def docstring
For tools — help(fn) shows it. Use for “what this function does”.
def printname(): """return..."""
Python — all 3 in action
# 1) Single-line comment — ignored completely
print('praveen')  # → praveen (comment after code also ignored)

# 2) Multi-line note via triple quotes — useful for long todo
"""
the code below prints the name praveen
this code is amazing — it handles ETL logging
"""
print('praveen')

# 3) Docstring — first string inside function = documentation
def printname(name):
    """Return a greeting for the given name."""
    return f"hello {name}"

help(printname)  # shows: Return a greeting for the given name.
print(printname('praveen'))  # hello praveen

# Tip: Ctrl+/ in VS Code comments/uncomments a whole block
Output
praveen
praveen
Help on function printname:
    Return a greeting for the given name.
Python — indentation matters
for i in ['praveen', 'kumar']:
    print(i)  # inside loop
print('done')  # outside — dedent ends block
Output
praveen
kumar
done