Lists — ordered, mutable, allows duplicates
What: A list is Python's workhorse — ordered, mutable (you can change it), and it allows duplicates — think of it as a Spark array column you can reshape row by row. Why it matters: ETL stages constantly build lists: collecting file paths, batching rows, or nesting results. Mixing up append vs extend silently creates the wrong shape — a nested list instead of a flat one — and downstream explode() or joins break. How to remember: append adds the whole object as one entry (nest), extend iterates and adds each element (flatten) — see the box diagram below.
append(x) → one slot, extend([x,y]) → two slots | Spark: array_append vs array_unioninsert(1,'tomato') splices at a position, remove('kiwi') deletes first match, pop() removes last, and list(set(...)) de-duplicates — all building blocks for manual bucketing before you let Spark do it at scale.x = [1,2,3,4,'praveen',True,1.014]
print(x[::-1]) # reversed
a = ['apple','banana','cherry']
a.append('coconut') # ['apple','banana','cherry','coconut']
a.append(['orange','berry']) # nests! [..., ['orange','berry']]
print(a)
a = ['apple','banana','cherry']
b = ['orange','watermelon']
a.extend(b) # flattens: [...,'orange','watermelon']
print(a)
print([1,2,3] + [4,5,6]) # [1,2,3,4,5,6]
a.insert(1, 'tomato') # at index 1
print(a)
x = ['apple','banana',['mango','cherry',['watermelon','pencil','praveen'],'coconut'],'mouse']
print(x[2][2][2]) # praveen
x = ['apple','banana','kiwi','mango','orange','kiwi']
x.remove('kiwi') # first kiwi only
print(x)
x.pop() # last
print(x)
x.pop(0)
print(x)
print(list(set(['apple','banana','apple','mango']))) # dedup, order varies[1.014, True, 'praveen', 4, 3, 2, 1]
['apple', 'banana', 'cherry', 'coconut', ['orange', 'berry']]
['apple', 'banana', 'cherry', 'orange', 'watermelon']
[1, 2, 3, 4, 5, 6]
['apple', 'tomato', 'banana', 'cherry', 'orange', 'watermelon']
praveen
['apple', 'banana', 'mango', 'orange', 'kiwi']
['apple', 'banana', 'mango', 'orange']
['banana', 'mango', 'orange']
['mango', 'banana', 'apple'] (order varies)Tuples — ordered, immutable
What: A tuple looks like a list — (1,2,'praveen') — but it is immutable, meaning once created you cannot add, remove, or change elements. Why it matters: In Data Engineering immutability is a safety feature: tuples are used as fixed records like (employee_id, name) in RDDs, as dictionary keys, or as broadcast lookup entries where accidental mutation would corrupt joins. How to remember: List = editable draft, Tuple = sealed envelope — if you need to "change" it you must open (convert to list), edit, and reseal (convert back to tuple).
a,b,c = (1,2,3)x = (1,2,3,4,4,5,'praveen','apple')
print(x[-1]) # apple
# x.remove(1) # AttributeError — no remove
x = (1,2,3,4,5)
y = list(x); y.append(10); x = tuple(y)
print(x) # (1, 2, 3, 4, 5, 10)
a, b, c = (1,2,3)
print(a, b, c) # 1 2 3apple
(1, 2, 3, 4, 5, 10)
1 2 3Dictionaries — JSON-like (key:value)
What: A dictionary is Python's JSON — {key: value} pairs that store structured records like {'fname':'praveen','city':'bangalore','phonenumbers':[123...,987...]}. Why it matters: Every Spark JSON row, every API response, and every nested config is a dict of dicts — students['student1']['sports']['morning'] → 'cricket' is the same path you query with get_json_object or col('student.sports.morning') in Spark. How to remember: Keys are labels on drawers, values are what is inside; x['phonenumbers'][1] means open drawer "phonenumbers" then pick index 1.
spark.read.json("students.json") creates the same nested schema — use x.get('missing','not found') to avoid KeyError.keys / values / items
list(x.keys()) labels, list(x.values()) contents, list(x.items()) pairs — like Spark schema vs rows.get() vs []
x['missing'] throws KeyError, x.get('missing','not found') returns default — always use get in ETL.pop / popitem
pop('pancard') removes by key, popitem() removes last inserted — LIFO cleanup.x = {'fname':'praveen','lname':'kumar','age':25, 'city':'bangalore','salary':10000,'phonenumbers':[123456789,987654321]}
print(x['phonenumbers'][1]) # 987654321
print(list(x.keys()))
print(list(x.values()))
print(list(x.items()))
x['pancard'] = '2131321'
x['pancard'] = '21313213212313' # overwrite
x.pop('pancard'); x.popitem()
print(x)
students = {
'student1': {'fname':'praveen','age':25,'courses':['math','science'], 'sports':{'morning':'cricket','evening':'golf'}},
'student2': {'fname':'ajay','age':26}
}
print(students['student1']['sports']['morning']) # cricket
print(x.get('missing','not found'))987654321
['fname', 'lname', 'age', 'city', 'salary', 'phonenumbers']
['praveen', 'kumar', 25, 'bangalore', 10000, [123456789, 987654321]]
[('fname', 'praveen'), ...]
{... without pancard}
cricket
not foundSets — unordered, no duplicates
What: A set is an unordered bag that keeps only unique values — {'apple','banana','mango','mango'} automatically collapses to three items, and you cannot index it like a list. Why it matters: In Data Engineering sets are your "distinct" gate before an expensive shuffle or join: collect keys into a set to see cardinality, then list(set(ids)) to deduplicate, just like df.dropDuplicates() or SELECT DISTINCT in Spark. How to remember: Set = Venn circle — overlap is intersection, everything together is union, subtract is A - B.
A.union(B) → {1,2,3,4,5}A.intersection(B) → {3}A - B → {1,2} B - A → {4,5}list(set([1,2,3,4,4,4,5,1])) → [1,2,3,4,5] (order varies) | Spark: df.select('id').distinct().count()myset = {'apple','banana','mango','mango'}
print(myset) # {'apple','banana','mango'}
myset.add('orange'); print(myset)
myset.remove('banana'); print(myset)
A={1,2,3}; B={3,4,5}
print(A.union(B)) # {1,2,3,4,5}
print(A.intersection(B)) # {3}
print(A - B) # {1,2}
print(list(set([1,2,3,4,4,4,5,1,2,3]))) # [1,2,3,4,5] (order varies){'apple', 'banana', 'mango'}
{'apple', 'banana', 'mango', 'orange'}
{'apple', 'mango', 'orange'}
{1, 2, 3, 4, 5}
{3}
{1, 2}
[1, 2, 3, 4, 5]