Bookmarks

🟡 Intermediate

Exception Handling — try / except / else / finally

What: Exception handling is the safety net around risky code — try the operation, except catches known failures (ZeroDivision, ValueError, FileNotFound), else runs only if nothing failed, and finally always runs (close handles, release connections). Why it matters: In ETL every stage is a minefield: a zero divisor in a rate calc, a bad string in an int column, a missing landing file — wrapping them with specific except guards keeps the pipeline alive and logs a clean error instead of crashing at 3 AM. How to remember: raise Exception('insufficient balance') is you throwing the error; except (ValueError, ZeroDivisionError) as e is you catching it — be specific, never bare except:.

try / except / else / finally — pipeline guard flow
try
risky code
x = 10/0
→
except
on error → handle
ZeroDivision / ValueError
→
else
no error →
success path
→
finally
always → close
file / connection
Catch specific → except FileNotFoundError:  |  Custom gate → if amount > balance: raise Exception('insufficient')
Python
try:
    x = 10/0
except ZeroDivisionError:
    print('you cannot divide by 0')
else:
    print('no error')
finally:
    print('i will execute no matter what')

try:
    y = int('abc')
except (ValueError, ZeroDivisionError) as e:
    print('Error:', e)

def withdraw(amount):
    balance = 1000
    if amount > balance:
        raise Exception('insufficient balance')
    return balance - amount
try:
    print(withdraw(10000))
except Exception as e:
    print('Error:', e)

try:
    file = open('/tmp/sample.csv','r')
except FileNotFoundError:
    print('file not found')
else:
    print('file read successfully')
finally:
    print('im just printing')
Output
you cannot divide by 0
i will execute no matter what
Error: invalid literal for int() with base 10: 'abc'
Error: insufficient balance
file not found
im just printing
🟡 Intermediate

File Handling — r / w / a

What: File handling in Python is open(path, mode) where mode decides the contract — 'w' creates/overwrites, 'r' reads, 'a' appends — and with open(...) as f: auto-closes the handle even if an error occurs. Why it matters: Before Spark's spark.read.csv touches a 5 GB log, Python often pre-checks or samples it with open; using /tmp/t3.txt keeps examples portable across local and cluster nodes, and a generator that yields lines streams the file without loading it all into RAM. How to remember: w = write (fresh slate), a = add (keep old), r = read (look only) — and always prefer with over manual close().

File modes — w / r / a with /tmp/t3.txt
'w' — write
f.write('hello world ')
f.write('welcome')
creates or overwrites
'r' — read
f.read() → "hello..."
for line in f: ...
read whole or line-by-line
'a' — append
f.write('\nnew line')
keeps old, adds at end
Always with open(...) as f: auto-close  |  Large file: def readlog(path): with open(path) as fh: for line in fh: yield line streams without OOM
Spark: spark.read.csv('/data/...') at scale Python: pre-check sample before Spark load
Python
with open('/tmp/t3.txt','w') as f:
    f.write('hello world ')
    f.write('welcome to python')
with open('/tmp/t3.txt','r') as f:
    print(f.read())  # hello world welcome to python
with open('/tmp/t3.txt','a') as f:
    f.write('\nthis is a new line appended')
with open('/tmp/t3.txt','r') as f:
    for line in f: print(line.strip())
import os; os.remove('/tmp/t3.txt'); print('removed')

# Generator-style line reader — 5GB log without OOM
def readlogfile(path):
    with open(path,'r') as fh:
        for line in fh:
            yield line
Output
hello world welcome to python
hello world welcome to python
this is a new line appended
removed
🔴 Advanced

Regex — phones, emails, dates, names

What: Regular expressions are pattern languages for text — \b\d{10}\b means "exactly 10 digits as a whole word", \b\d{2}[-/]\d{2}[-/]\d{4}\b matches dates like 16-05-1994, and [A-Z][a-z]+ finds Capitalized Words. Why it matters: In ETL you extract PII from free-text logs: phones, emails, postcodes, and HTML snippets — one re.findall(pattern, txt) replaces 20 lines of manual string slicing, and {2,} quantifiers make the rules resilient. How to remember: \b = word edge, \d = digit, {n} = exactly n, {2,} = two or more — compose them left to right like LEGO.

Regex kit — building blocks to full patterns
phone
\b\d{10}\b
6363402404
email
...+@...+.[A-Z]{2,}
praveen.b@example.com
date
\d{2}[-/]\d{2}[-/]\d{4}
16-05-1994
name
[A-Z][a-z]+\s[A-Z][a-z]+
Praveen Kumar
Quantifiers: {10} exactly 10   {6} postcode   {2,} TLD ≥2 chars  |  Use re.findall(pattern, txt) to extract all matches
Python
import re
txt = 'Praveen Kumar bought a Tomato Ketchup and he was born on 16-05-1994. His email is praveen.b@example.com and Phone Is 6363402404 and 1231231231.'

print(re.findall(r'\b[A-Z][a-z]+\s[A-Z][a-z]+\b', txt))  # ['Praveen Kumar','Tomato Ketchup']
print(re.findall(r'\b\d{10}\b', txt))  # ['6363402404','1231231231']
print(re.findall(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b', txt))  # ['praveen.b@example.com']
print(re.findall(r'\b\d{2}[-/]\d{2}[-/]\d{4}\b', txt))  # ['16-05-1994']
print(re.findall(r'\b\d{6}\b', '560001 is Bangalore'))  # ['560001']
print(re.findall(r'<p>(.*?)</p>', '<p>hello world</p>'))  # ['hello world']
Output
['Praveen Kumar', 'Tomato Ketchup']
['6363402404', '1231231231']
['praveen.b@example.com']
['16-05-1994']
['560001']
['hello world']
🟡 Intermediate

JSON & APIs — dumps/loads & HTTP codes

What: JSON is the lingua franca of APIs and streaming — json.loads(string) parses text into a Python dict/list (load string), while json.dumps(object) serialises a dict back to a JSON string (dump string). Why it matters: Every REST response, Kinesis payload, and Spark spark.read.json(...) row is JSON; mastering the round-trip plus HTTP codes (2xx success, 4xx client error, 5xx server error) and verbs (GET read, POST create, PUT/PATCH/DELETE) is table stakes for ingesting external data. How to remember: Loads goes Left (string → object), Dumps goes Direction outward (object → string) — and Python True/None become JSON true/null.

JSON round-trip + HTTP map
string (JSON text)
'{"name":"praveen",
"age":30}'
type: str
json.loads() → ← json.dumps()
Python object
{'name':'praveen',
'age':30}
type: dict
HTTP
2xx success3xx redirect 4xx client err5xx server err
GET read · POST create · PUT full · PATCH partial · DELETE remove
Python
import json
s = '{"name":"praveen","age":30,"city":"bangalore"}'
obj = json.loads(s); print(obj, type(obj))

obj2 = {'name':'praveen','age':30,'city':'bangalore'}
s2 = json.dumps(obj2); print(s2, type(s2))

x = {'name':'John','married':True,'children':('Ann','Billy'),'pets':None,
     'cars':[{'model':'BMW 230','mpg':27.5},{'model':'Ford Edge','mpg':24.1}]}
print(json.dumps(x))
# Status: 1xx info, 2xx success, 3xx redirect, 4xx client, 5xx server
# GET=read, POST=create, PUT=full update, PATCH=partial, DELETE=remove
Output
{'name': 'praveen', 'age': 30, 'city': 'bangalore'} <class 'dict'>
{"name": "praveen", "age": 30, "city": "bangalore"} <class 'str'>
{"name": "John", "married": true, ...}
🔴 Advanced

Decorators & Generators

What: A decorator wraps a function with extra behaviour — @loginrequired checks login before viewprofile() runs — and a generator uses yield to produce values one-by-one instead of building a huge list. Why it matters: Decorators add cross-cutting concerns (auth, timing, retry) without touching core logic, while generators stream a 5 GB log line-by-line with constant memory — both are essential for production Spark helpers that must scale without OOM. How to remember: Decorator = gift wrapper around the function; Generator = faucet that drips one line at a time when you call next().

Decorator (wrap) & Generator (stream)
Decorator — wrapping
@mydecorator
def idli(): print('idli is hot')
before → idli is hot → after
wrapper() runs before/after func()
Generator — streaming
def countup(n):  yield i
0→1→2one at a time
list(countup(3)) → [0,1,2]
5 GB file: for line in fh: yield line — never loads all
Python
def loginrequired(func):
    def wrapper(user):
        if user['loggedin']: return func(user)
        else: print('please login before accessing profile')
    return wrapper

@loginrequired
def viewprofile(user): print('welcome how are you ?')

viewprofile({'name':'praveen','loggedin':True})   # welcome
viewprofile({'name':'praveen','loggedin':False})  # please login...

def mydecorator(func):
    def wrapper():
        print('before')
        func()
        print('after')
    return wrapper
@mydecorator
def idli(): print('idli is hot')
idli()  # before / idli is hot / after

def readfile(path):
    with open(path,'r') as f:
        for line in f: yield line
def countup(n):
    for i in range(n): yield i
print(list(countup(3)))  # [0,1,2]
Output
welcome how are you ?
please login before accessing profile
before
idli is hot
after
[0, 1, 2]