Bookmarks

🟢 Easy

The Four Core Types

Every PySpark column has a type. Master Python's four and Spark's IntegerType / StringType / BooleanType / DoubleType will feel familiar.

Python
# Integer: whole numbers
x = 10; print(type(x))          # <class 'int'>
# Float: decimals
x = 1.04; print(type(x))        # <class 'float'>
# Boolean: True=1, False=0 (capital T/F)
print(True + True)               # 2
print(True == 1, False == 0)     # True True
# String: quoted text
x = 'praveen'; y = "praveen"; z = "bangalore's"
print(type(z))                  # <class 'str'>
Output
<class 'int'>
<class 'float'>
2
True True
<class 'str'>
🟡 Intermediate

String Slicing & Indexing

Slicing rule: s[start : stop : step]. start is inclusive, stop is exclusive, step is stride. Negative indices count from the end (-1 = last char). Omit a part and Python uses the default: s[:4] = from start, s[2:] = to end, s[::2] = every 2nd. In Spark you slice partition paths like /data/year=2024/month=09/day=15/ the same way.

String as boxes — indices for 'bangalore' (length 9)
b
a
n
g
a
l
o
r
e
0
1
2
3
4
5
6
7
8
-9
-8
-7
-6
-5
-4
-3
-2
-1
x[1:3] → a n (indices 1,2)  |  x[-3] → o  |  x[::-1] → reverse
Python
x = 'bangalore'
print(x[1:3])    # 'an'  (index 1 to 2, 3 exclusive)
print(x[::2])    # 'bn aoe' — positions 0,2,4,6,8
print(x[::-1])   # 'erolagnab' — step -1 = reverse
print(x[-3])     # 'o' — 3rd from end
print(x[:4])     # 'bang' — first 4 chars
print(x[2:])     # 'ngalore' — from index 2 to end
print(x[-4:-1])  # 'lor' — slice with negatives

# Concatenation — only string + string
x = 'praveen'; y = 'kumar'
print(x + ' ' + y)  # praveen kumar
print('1' + 'praveen')  # '1praveen' — not 1+praveen as number
Output
an
bn aoe
erolagnab
o
bang
ngalore
praveen kumar
1praveen
🟡 Intermediate

String Methods — before/after

In ETL, strings arrive dirty: ' praveen ' with spaces, 'HELLO' in wrong case, 'my,name,is,praveen' as CSV. You clean in a pipeline: strip → case → replace → split/join → validate. Each method below shows raw → call → cleaned.

Cleaning pipeline — from raw to trusted
raw: ' praveen 'strip()'praveen'upper()/isalpha()trusted
Python — clean step by step
# 1) Trim edges — user typed extra spaces
x = ' praveen '
print(repr(x.strip()))           # 'praveen' — front/back spaces removed
# why strip before isalpha? ' praveen '.isalpha() is False (space is not alpha)

# 2) Normalize case
print('hello'.replace('ll','k')) # 'heko' — replace substring
print('praveen'.upper())         # 'PRAVEEN' — for case-insensitive compare
print('PRAVEEN'.lower())         # 'praveen'
print('praveen'.capitalize())    # 'Praveen' — first letter up, rest low
print('praveen patel'.endswith('e'))  # False — check suffix

# 3) Check content type
print('praveen'.isalpha())       # True — only letters?
print('6363402404'.isdecimal())  # True — only digits 0-9?

# 4) Split & re-join — CSV ↔ list
print('my,name,is,praveen'.split(','))  # ['my','name','is','praveen'] — delimiter ','
print('|'.join(['my','name','is','praveen']))  # my|name|is|praveen — new delimiter '|'

# 5) Real validation chain — Indian mobile: 10 digits, all numeric, starts 6/7/8/9
x = '6363402404'
valid = len(x)==10 and x.isdecimal() and x[0] in '6789'
print(valid)  # True — valid Indian mobile
# try: x=' 6363402404 ' → strip first, then validate

# 6) Re-assemble sentence
words = ['hey','wassup','how','are','you?']
print(' '.join(words))  # hey wassup how are you?
Output
'praveen'
heko
PRAVEEN
praveen
Praveen
False
True
True
['my', 'name', 'is', 'praveen']
my|name|is|praveen
True
hey wassup how are you?
🟡 Intermediate

Find & Text Grouping — with visuals

Searching inside text and re-grouping characters are daily ETL tasks — e.g. finding a city name inside a free-text address, or splitting an alphanumeric ID like AbCdeFG123KL into its upper/lower/digit parts. This section explains how find() works, why the old workbook's manual length math was fragile, and how to do it cleanly.

1) str.find(sub) — what it really does

find() scans left-to-right and returns the index of the first occurrence, or -1 if not found. It does not return the text — you use the index to slice it out.

Visual: find + slice
index: 0    19        28
'my name is praveen, i m from bangalore, bangalore is beautiful'
x.find('bangalore') → 19  |  x[19 : 19+9] → 'bangalore'

Returns index

x.find('bangalore')19 (not the word). Use index to slice.

-1 = not found

x.find('mumbai')-1. Always check before slicing.

Find vs index()

find() returns -1 silently; index() raises ValueError. In ETL, prefer find().

2) Manual length calc (old) vs clean version

The workbook did this:

Python — old manual math (fragile)
x = 'my name is praveen, i m from bangalore, bangalore is beautiful'
i = x.find('bangalore')
i_len = len('bangalore')  # 9
xlen = 27 + i             # magic 27? breaks if text changes
print(x[27:xlen])         # works only for this exact sentence

Problem: magic number 27 assumes the word starts at 27 — if the address changes, it breaks. Clean version: start from the index you just found, add len(word):

Python — clean, robust
x = 'my name is praveen, i m from bangalore, bangalore is beautiful'
word = 'bangalore'
i = x.find(word)          # 19 — computed, not guessed
print(i)                  # 19
print(x[i : i + len(word)])  # x[19:28] → 'bangalore' — works for any sentence
# also works with slice length directly
print(x[i : i+9])         # same, but len(word) is self-documenting
Output
19
bangalore
bangalore
✅ PySpark parallel
Same idea in Spark: F.instr(col('address'), 'bangalore') gives position, F.substring(col('address'), pos, length) extracts. Never hard-code positions.

3) Grouping characters by type — isupper / islower / isnumeric

Data often arrives mixed: AbCdeFG123KL could be a product code where you need to separate capitals, small letters, and digits for validation. The workbook had the code but no why or visual. Here is the bucket view:

Buckets — where each character goes
UPPER
A C F G K L
c.isupper() → True
lower
b d e
c.islower() → True
digits
1 2 3
c.isnumeric() → True
s = 'A b C d e F G 1 2 3 K L'  →  ACFGKL + bde + 123 = ACFGKLbde123
Python — grouping
s = 'AbCdeFG123KL'
# Three buckets — each character tested once
upper = ''.join(c for c in s if c.isupper())   # 'ACFGKL' — capitals
lower = ''.join(c for c in s if c.islower())   # 'bde'    — small
digits = ''.join(c for c in s if c.isnumeric())# '123'    — numbers
print(upper, lower, digits)
print(upper + lower + digits)  # 'ACFGKLbde123' — re-assembled by type

# Why this order? In many validation rules you want capitals first,
# then lower, then digits — e.g. generating a normalized ID.

# PySpark equivalent (for a DataFrame column `code`):
# F.regexp_extract(col('code'), '[A-Z]+', 0)  → capitals
# F.regexp_extract(col('code'), '[a-z]+', 0)  → lower
# F.regexp_extract(col('code'), '[0-9]+', 0)  → digits
Output
ACFGKL bde 123
ACFGKLbde123
💡 When to use
Cleaning phone numbers (if c.isnumeric()), separating product codes, or building a new ID where type order matters. In Spark, do this per-row with F.when(c.isUpper()) logic or regex.