🟢 Easy
SCD Types Overview
| Type | Behavior |
|---|---|
| SCD 0 | Fixed, no updates allowed |
| SCD 1 | Overwrite on top |
| SCD 2 | New row with start_date, end_date, flag |
| SCD 3 | Add new column for previous value |
| SCD 4 | Historical data in different table |
| SCD 6 | Hybrid (1+2+3) |
🔴 Advanced
SCD 2 Example (from file)
| id | name | address | startdate | enddate | flag |
|---|---|---|---|---|---|
| 1 | indumati | 123 mg road | 01012025 | 03082025 | N |
| 1 | indumati | 234 mg road | 03082025 | 31129999 | Y |
Python
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit, current_date
spark = SparkSession.builder.appName('myapplication').getOrCreate()
newdf = customerdf.withColumn('startdate',current_date()).withColumn('enddate',lit('9999-12-31')).withColumn('FLAG',lit('Y'))SCD 3 example from file: id | name | address | job title | previous job title — tracks previous value in new column.