The Four Core Types
Every PySpark column has a type. Master Python's four and Spark's IntegerType / StringType / BooleanType / DoubleType will feel familiar.
# Integer: whole numbers
x = 10; print(type(x)) # <class 'int'>
# Float: decimals
x = 1.04; print(type(x)) # <class 'float'>
# Boolean: True=1, False=0 (capital T/F)
print(True + True) # 2
print(True == 1, False == 0) # True True
# String: quoted text
x = 'praveen'; y = "praveen"; z = "bangalore's"
print(type(z)) # <class 'str'><class 'int'>
<class 'float'>
2
True True
<class 'str'>String Slicing & Indexing
Slicing rule: s[start : stop : step]. start is inclusive, stop is exclusive, step is stride. Negative indices count from the end (-1 = last char). Omit a part and Python uses the default: s[:4] = from start, s[2:] = to end, s[::2] = every 2nd. In Spark you slice partition paths like /data/year=2024/month=09/day=15/ the same way.
x = 'bangalore'
print(x[1:3]) # 'an' (index 1 to 2, 3 exclusive)
print(x[::2]) # 'bn aoe' — positions 0,2,4,6,8
print(x[::-1]) # 'erolagnab' — step -1 = reverse
print(x[-3]) # 'o' — 3rd from end
print(x[:4]) # 'bang' — first 4 chars
print(x[2:]) # 'ngalore' — from index 2 to end
print(x[-4:-1]) # 'lor' — slice with negatives
# Concatenation — only string + string
x = 'praveen'; y = 'kumar'
print(x + ' ' + y) # praveen kumar
print('1' + 'praveen') # '1praveen' — not 1+praveen as numberan
bn aoe
erolagnab
o
bang
ngalore
praveen kumar
1praveenString Methods — before/after
In ETL, strings arrive dirty: ' praveen ' with spaces, 'HELLO' in wrong case, 'my,name,is,praveen' as CSV. You clean in a pipeline: strip → case → replace → split/join → validate. Each method below shows raw → call → cleaned.
# 1) Trim edges — user typed extra spaces
x = ' praveen '
print(repr(x.strip())) # 'praveen' — front/back spaces removed
# why strip before isalpha? ' praveen '.isalpha() is False (space is not alpha)
# 2) Normalize case
print('hello'.replace('ll','k')) # 'heko' — replace substring
print('praveen'.upper()) # 'PRAVEEN' — for case-insensitive compare
print('PRAVEEN'.lower()) # 'praveen'
print('praveen'.capitalize()) # 'Praveen' — first letter up, rest low
print('praveen patel'.endswith('e')) # False — check suffix
# 3) Check content type
print('praveen'.isalpha()) # True — only letters?
print('6363402404'.isdecimal()) # True — only digits 0-9?
# 4) Split & re-join — CSV ↔ list
print('my,name,is,praveen'.split(',')) # ['my','name','is','praveen'] — delimiter ','
print('|'.join(['my','name','is','praveen'])) # my|name|is|praveen — new delimiter '|'
# 5) Real validation chain — Indian mobile: 10 digits, all numeric, starts 6/7/8/9
x = '6363402404'
valid = len(x)==10 and x.isdecimal() and x[0] in '6789'
print(valid) # True — valid Indian mobile
# try: x=' 6363402404 ' → strip first, then validate
# 6) Re-assemble sentence
words = ['hey','wassup','how','are','you?']
print(' '.join(words)) # hey wassup how are you?'praveen'
heko
PRAVEEN
praveen
Praveen
False
True
True
['my', 'name', 'is', 'praveen']
my|name|is|praveen
True
hey wassup how are you?Find & Text Grouping — with visuals
Searching inside text and re-grouping characters are daily ETL tasks — e.g. finding a city name inside a free-text address, or splitting an alphanumeric ID like AbCdeFG123KL into its upper/lower/digit parts. This section explains how find() works, why the old workbook's manual length math was fragile, and how to do it cleanly.
1) str.find(sub) — what it really does
find() scans left-to-right and returns the index of the first occurrence, or -1 if not found. It does not return the text — you use the index to slice it out.
Returns index
x.find('bangalore') → 19 (not the word). Use index to slice.-1 = not found
x.find('mumbai') → -1. Always check before slicing.Find vs index()
find() returns -1 silently; index() raises ValueError. In ETL, prefer find().2) Manual length calc (old) vs clean version
The workbook did this:
x = 'my name is praveen, i m from bangalore, bangalore is beautiful'
i = x.find('bangalore')
i_len = len('bangalore') # 9
xlen = 27 + i # magic 27? breaks if text changes
print(x[27:xlen]) # works only for this exact sentenceProblem: magic number 27 assumes the word starts at 27 — if the address changes, it breaks. Clean version: start from the index you just found, add len(word):
x = 'my name is praveen, i m from bangalore, bangalore is beautiful'
word = 'bangalore'
i = x.find(word) # 19 — computed, not guessed
print(i) # 19
print(x[i : i + len(word)]) # x[19:28] → 'bangalore' — works for any sentence
# also works with slice length directly
print(x[i : i+9]) # same, but len(word) is self-documenting19
bangalore
bangaloreF.instr(col('address'), 'bangalore') gives position, F.substring(col('address'), pos, length) extracts. Never hard-code positions.3) Grouping characters by type — isupper / islower / isnumeric
Data often arrives mixed: AbCdeFG123KL could be a product code where you need to separate capitals, small letters, and digits for validation. The workbook had the code but no why or visual. Here is the bucket view:
s = 'AbCdeFG123KL'
# Three buckets — each character tested once
upper = ''.join(c for c in s if c.isupper()) # 'ACFGKL' — capitals
lower = ''.join(c for c in s if c.islower()) # 'bde' — small
digits = ''.join(c for c in s if c.isnumeric())# '123' — numbers
print(upper, lower, digits)
print(upper + lower + digits) # 'ACFGKLbde123' — re-assembled by type
# Why this order? In many validation rules you want capitals first,
# then lower, then digits — e.g. generating a normalized ID.
# PySpark equivalent (for a DataFrame column `code`):
# F.regexp_extract(col('code'), '[A-Z]+', 0) → capitals
# F.regexp_extract(col('code'), '[a-z]+', 0) → lower
# F.regexp_extract(col('code'), '[0-9]+', 0) → digitsACFGKL bde 123
ACFGKLbde123if c.isnumeric()), separating product codes, or building a new ID where type order matters. In Spark, do this per-row with F.when(c.isUpper()) logic or regex.