Bookmarks

🟢 Easy

Functions — reusable logic

What: A function is a named, reusable block — def greet(name): takes input (parameters), does work, and optionally sends back a value with return. Why it matters: Without functions every ETL notebook repeats the same postcode, phone, or date validator; with them you write once and reuse across Spark UDFs and pipeline steps. How to remember: print shows, return hands back — a function that only prints returns None, so Spark UDFs must return to produce a column.

Function anatomy — definition vs call
def postcodevalidator(code): ← definition (recipe)
return 'valid' if len(code)==6 and code.isdigit() else 'invalid'
postcodevalidator('560001') ← call (use) → 'valid'
print() → shows, returns None return → gives value to caller c=0 default → optional param
Python
def printhello():
    print('hello')
printhello()  # hello

def greet(name):
    print(f'hello {name}, welcome to ICICI')
greet('praveen')

def add(a,b,c=0):
    return a+b+c
print(add(1,2))    # 3
print(add(1,2,3))  # 6

def square_print(x):
    print(x*x)
def square_return(x):
    return x*x
print(square_print(6))   # 36 then None
print(square_return(6))  # 36

def postcodevalidator(code):
    return 'valid' if len(code)==6 and code.isdigit() else 'invalid'
print(postcodevalidator('560001'))  # valid
print(postcodevalidator('56A001'))  # invalid
Output
hello
hello praveen, welcome to ICICI
3
6
36
None
36
valid
invalid
🟡 Intermediate

*args & **kwargs — variable arguments

What: *args collects extra positional arguments into a tuple, **kwargs collects extra keyword arguments into a dict — letting one function accept any number of inputs. Why it matters: Think of *args as Spark's *cols (df.select(*cols)) and **kwargs as Spark config dicts (spark.conf.set(**settings)); both let you pass a flexible bundle without rewriting the caller. How to remember: Single star * = tuple of loose values, double star ** = dict of named values — stars "pack" on entry, and "unpack" when you call with *list or **dict.

Packing vs unpacking — stars in action
*args → tuple
def addnumbers(*args):
(1, 2, 3, 4)
addnumbers(1,2,3,4) packed
**kwargs → dict
def printdetails(**kwargs):
{name: praveen, age: 25}
name='praveen', age=25 packed
*args, **kwargs
def printinfo(*args, **kwargs):
(1,2,3) + {name: praveen}
both at once
Python
def addnumbers(*args):
    return sum(args)
print(addnumbers(1,2))     # 3
print(addnumbers(1,2,3,4)) # 10

def printdetails(**kwargs):
    for k,v in kwargs.items():
        print(f'{k}={v}')
printdetails(name='praveen', age=25, city='bangalore')

def printinfo(*args, **kwargs):
    print('args', args)
    print('kwargs', kwargs)
printinfo(1,2,3, name='praveen')

def configuration(**settings):
    print('debug on' if settings.get('debug') else 'debug off')
configuration(debug=True, version='1.0', region='IN')
Output
3
10
name=praveen
age=25
city=bangalore
args (1, 2, 3)
kwargs {'name': 'praveen'}
debug on
🟡 Intermediate

Lambda — one-line anonymous functions

What: A lambda is a one-line anonymous function — lambda x,y: x+y is shorthand for a tiny def without a name, useful when the function is the argument itself. Why it matters: In PySpark, rdd.map(lambda x: x*x) or filter(lambda i: i%2==0) applies that tiny logic to every partition element without defining a full function elsewhere; it is the inline workhorse of map / filter / reduce. How to remember: Read lambda x: x*x as "take x, return x*x" — no return keyword, the expression is the return value.

lambda vs def — same logic, different form
def style
def add(x,y):
  return x+y
named, multi-line
⇄
lambda style
lambda x,y: x+y
anonymous, one expression
map(lambda i: i*i, x) → squares filter(lambda i: i%2==0, x) → evens reduce(lambda a,b: a+b, x) → fold
Python
add = lambda x,y: x+y
print(add(1,2))  # 3
print((lambda x: x*x)(1000))  # 1000000
print((lambda x: 'even' if x%2==0 else 'Odd')(11))  # Odd
print((lambda x: x[-1])('praveen'))  # a
x = [1,2,3,4,5,6,7,8,9]
print(list(map(lambda i: i*i, x)))          # squares
print(list(filter(lambda i: i%2==0, x)))    # evens
from functools import reduce
print(reduce(lambda a,b: a+b, [1,2,3,4,5])) # 15 sum
print(reduce(lambda a,b: a if a>b else b, [10,50,30,90,20]))  # 90 max
Output
3
1000000
Odd
a
[1, 4, 9, 16, 25, 36, 49, 64, 81]
[2, 4, 6, 8]
15
90
🔴 Advanced

Comprehensions

What: A comprehension builds a list, dict, or set in one expression — [i*i for i in x if i%2==0] compresses a 3-line loop into a single readable line. Why it matters: It is the Python twin of Spark's select + filter: use the OAC mental model — Output, Action (loop), Condition — to read any comprehension left-to-right without confusion. How to remember: Say it aloud: "give me i*i for each i in x if i is even" — Output first, then where it comes from, then the filter.

OAC — Output · Action · Condition
[ O: i*i A: for i in x C: if i%2==0 ]
list [i*i for i in x]
[1,4,9,16,25,36]
dict {i:i*i for i in range(5)}
{0:0,1:1,2:4,...}
set {i for i in range(10) if i%2==0}
{0,2,4,6,8}
Spark equivalent: [i*i for i in x if i%2==0] ≈ df.filter(col('i')%2==0).select((col('i')*col('i')).alias('sq'))  |  Flatten: [j for row in mat for j in row]
Python
x = [1,2,3,4,5,6]
print([i*i for i in x])                    # [1,4,9,16,25,36]
print([i*i for i in x if i%2==0])          # [4,16,36]
print([i for i in x if i%2==0])            # [2,4,6]

mat = [[1,2,3],[4,5,6],[7,8,9]]
print([j for row in mat for j in row])     # [1,2,3,4,5,6,7,8,9]

def flatten(lst):
    out=[]
    for i in lst:
        out.extend(flatten(i)) if isinstance(i,list) else out.append(i)
    return out
print(flatten([1,[2,3],[4,[5,6],[7,[8,9]]]]))  # [1,2,3,4,5,6,7,8,9]

print({i:i*i for i in range(5)})           # {0:0,1:1,2:4,3:9,4:16}
names = ['hema','sneha','praveen']
print({n:len(n) for n in names})           # {'hema':4,'sneha':5,'praveen':5}
print({i for i in range(10) if i%2==0})    # {0,2,4,6,8}
print([len(w) for w in ['data','engineering','is','hot']])  # [4,11,2,3]
Output
[1, 4, 9, 16, 25, 36]
[4, 16, 36]
[2, 4, 6]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}
{'hema': 4, 'sneha': 5, 'praveen': 5}
{0, 2, 4, 6, 8}
[4, 11, 2, 3]