Classes & Objects — the blueprint
What: A class is a blueprint (like an architect's plan), and an object is what you build from it — class Car defines what a car has (brand, model, speed) and what it does (drive(), stop()), while car1 = Car('mercedes',...) is the real car parked outside. Why it matters: PySpark's SparkSession.builder.getOrCreate() is exactly this pattern: the class is the factory, spark is the object you call spark.read.csv(...) on. How to remember: __init__ is the constructor — it runs automatically when you "build" the object and wires self.brand = brand as its personal data.
car1.drive() → mercedes e220 is driving
.getgrade() → C (logic inside object)
class SparkSession blueprint → spark = SparkSession.builder.getOrCreate() object → spark.read methodclass Car:
def __init__(self, brand, model, speed, colour):
self.brand, self.model, self.speed, self.colour = brand, model, speed, colour
def drive(self):
print(f'{self.brand} {self.model} is driving')
def stop(self):
print(f'{self.brand} {self.model} has stopped')
car1 = Car('mercedes','e220','150','white')
car1.drive() # mercedes e220 is driving
car1.stop()
class Student:
def __init__(self, name, marks):
self.name, self.marks = name, marks
def getgrade(self):
return 'A' if self.marks>=90 else 'B' if self.marks>=80 else 'C'
print(Student('praveen',70).getgrade()) # Cmercedes e220 is driving
mercedes e220 has stopped
CEncapsulation — hide balance, expose methods
What: Encapsulation hides sensitive data behind controlled methods — __balance is private (name-mangled to _BankAccount__balance), so outsiders cannot do acc.__balance = 999999; they must use deposit() / withdraw() / get_balance() which enforce rules. Why it matters: It mirrors Spark's design: you never touch internal partitions directly, you use the safe API (filter, map); similarly, setmarks() validates 0 ≤ m ≤ 100 before storing, preventing bad data at the gate. How to remember: Double-underscore __ = lock on the drawer, methods = the only keys with validation.
_BankAccount__balanceacc.withdraw(2000) → "insufficient"
acc.get_balance() → 1500
acc._BankAccount__balance (proof of mangling — but never do this in real code!)class BankAccount:
def __init__(self, balance):
self.__balance = balance # private — mangled
def deposit(self, amount):
self.__balance += amount
def withdraw(self, amount):
if amount <= self.__balance:
self.__balance -= amount
else:
print('insufficient balance')
def get_balance(self):
return self.__balance
acc = BankAccount(1000)
acc.deposit(500); print(acc.get_balance()) # 1500
acc.withdraw(2000) # insufficient balance
print(acc.get_balance()) # 1500
print(acc._BankAccount__balance) # 1500 — mangling proof
class Student2:
def __init__(self): self.__marks = 0
def setmarks(self, m):
if 0 <= m <= 100: self.__marks = m
else: print('invalid marks')
def getmarks(self): return self.__marks
s=Student2(); s.setmarks(95); print(s.getmarks()) # 95
s.setmarks(200) # invalid marks1500
insufficient balance
1500
1500
95
invalid marksAbstraction — rulebook, not engine
What: Abstraction is a rulebook: an abstract base class (class Payment(ABC)) declares what every payment must do (pay() + authenticate()) without saying how — concrete classes like CreditCard and UPI must fill in the details or Python refuses to instantiate them. Why it matters: In Data Engineering you define one contract for all sinks — "every sink must implement write(df)" — so S3, Delta, and JDBC writers are interchangeable, and a missing write fails fast at startup, not midnight in production. How to remember: ABC + @abstractmethod = "you cannot exist until you fulfil this promise."
pay(amount) @abstractmethod
authenticate()
pay(100) → credit card ✓
pay(100) → UPI ✓
for p in [CreditCard(), UPI()]: p.pay(100) works because both obey the same rulebook.from abc import ABC, abstractmethod
class Payment(ABC):
@abstractmethod
def pay(self, amount): pass
@abstractmethod
def authenticate(self): pass
class CreditCard(Payment):
def authenticate(self): print('verifying card')
def pay(self, amount):
self.authenticate()
print(f'Amount {amount} paid using credit card')
class UPI(Payment):
def authenticate(self): print('verifying UPI')
def pay(self, amount):
self.authenticate()
print(f'Amount {amount} paid using UPI')
for p in [CreditCard(), UPI()]:
p.pay(100)
# verifying card / Amount 100 paid using credit card
# verifying UPI / Amount 100 paid using UPI
# class Bad(Payment): pass # TypeError: Can't instantiate abstract classverifying card
Amount 100 paid using credit card
verifying UPI
Amount 100 paid using UPIInheritance — 5 types
What: Inheritance lets a child class reuse and extend a parent — Single (Child → Parent), Multiple (Child inherits from Mother and Father), Multilevel (Grandparent → Parent → Child), and Hybrid combos. Why it matters: In Spark you extend base readers/writers or mix behaviours (a class that is both a Logger and a MetricsEmitter); understanding MRO (Method Resolution Order) tells you which parent's method wins when two parents define the same show(). How to remember: Python's MRO is left-to-right, depth-first — class D(B,C) checks B before C, and D.mro() literally prints the search order.
▲
Child
↘ ↙
Child2
▼
Parent2
▼
Child3
↘ ↙
D(B,C)
D.mro() = [D, B, C, A, object] | Conflict: class C3(A2,B2): pass → C3().show() calls A2 (left first)# Single: Child → Parent
class Parent:
def greet(self): print('hello from parent')
class Child(Parent):
def hello(self): print('hello from child')
Child().greet() # hello from parent
# Multiple: Child(Mother, Father)
class Mother:
def skill(self): print('cooking')
class Father:
def talent(self): print('singing')
class Child2(Mother, Father):
def hobby(self): print('dancing')
Child2().talent() # singing
# Multilevel: Grandparent → Parent → Child
class Grandparent:
def greet(self): print('hello from grandparent')
class Parent2(Grandparent): pass
class Child3(Parent2): pass
Child3().greet()
# Hybrid + MRO
class A:
def show(self): print('A')
class B(A): pass
class C(A): pass
class D(B,C): pass
print(D.mro()) # [D, B, C, A, object]
D().show() # A
class A2:
def show(self): print('A2')
class B2:
def show(self): print('B2')
class C3(A2, B2): pass
C3().show() # A2 (left first)hello from parent
singing
hello from grandparent
[<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <class 'object'>]
A
A2Polymorphism — same name, many forms
What: Polymorphism means "same name, many forms" — the same .sound() or .start() call does something different depending on the object's class (Dog barks, Cat meows, Cow moos). Why it matters: A pipeline can loop over [S3Sink(), DeltaSink(), JDBCSink()] and call sink.write(df) without caring which one — duck typing says "if it has write, it is a sink." How to remember: Python has no true method overloading; the trick is default params — def add(a,b=0,c=0) handles add(1), add(1,2), and add(1,2,3) as one flexible method.
def add(a,b=0,c=0):add(1) → 1
add(1,2) → 3
add(1,2,3) → 6
class Dog:
def sound(self): print('bark')
class Cat:
def sound(self): print('meow')
class Cow:
def sound(self): print('moo')
for a in [Dog(), Cat(), Cow()]:
a.sound() # bark meow moo
class Vehicle:
def start(self): print('start the vehicle')
class Car(Vehicle):
def start(self): print('start the car')
class Bike(Vehicle):
def start(self): print('start the bike')
for v in [Car(), Bike()]:
v.start()
class Math:
def add(self, a,b=0,c=0): return a+b+c
m=Math()
print(m.add(1)) # 1
print(m.add(1,2)) # 3
print(m.add(1,2,3)) # 6bark
meow
moo
start the car
start the bike
1
3
6