Advanced

Snowflake - Cloud Data Warehouse

Snowflake is a cloud-native data warehouse that can be built on AWS, Azure, or GCP. It uses a multi-cluster architecture with separate storage, compute, and services layers.

Advanced

Architecture

Storage Layer

  • Data is stored in cloud object storage (S3 on AWS, Blob on Azure)
  • Data is stored in micro partitions (compressed columnar format)
  • Automatic indexing and metadata tracking
  • Supports structured and semi-structured data (JSON, Avro, Parquet, CSV, ORC)
  • Data is automatically partitioned (16-64 MB)
  • Columnar compressed, encrypted (AES 256)

Compute Layer

  • Compute is completely separated from storage
  • Follows MPP architecture (Massively Parallel Processing)
  • Virtual Warehouse (cluster) - individual computers
  • Can work in parallel on same data without interfering with each other
  • Supports Autoscaling

Cloud Service Layer

Coordination layer: authentication, metadata management, query optimization, access control. The brain of Snowflake.

  • Authentication: SSO, OAuth, MFA
  • Metadata management: tracks data, partitions, and clustering
Advanced

Advantages

  1. Separation of storage from compute - Pay only for what you use and scale independently.
  2. Concurrency and multi-cluster - Multiple teams can query the same data without any interference.
  3. Zero infrastructure management - Fully managed, no servers or patching.
  4. Native semi-structured data - Can manage JSON, Avro, Parquet with SQL queries natively.
  5. Automatic optimization - Automatic vacuum, distribution key is also automatic.
  6. Security - Very secure and supports cross-region replication.
Advanced

Limitations

  1. Per-second billing - Can be expensive.
  2. Less customizable - Not much control over nodes or servers.
  3. Sign up/sign out - Can be a lengthy process.
  4. No manual optimization - It is all taken care by Snowflake optimizer.
Advanced

Components of Snowflake — Key Components

Components of Snowflake: Storage Layer, Compute (Virtual Warehouse), Cloud Service Layer, plus Snowpipe, Streams, Tasks, Time Travel, Failsafe, Data Sharing.

Component Description
Snowpipe Continuous realtime data ingestion. Loads data as soon as it arrives.
Streams CDC in Snowflake. Captures changes to data.
Tasks Scheduler. Schedules when SQL queries or pipelines should get executed.
Time Travel Retrieves historical data (up to 90 days).
Failsafe 7 days disaster recovery protection.
Data Sharing Share data across multiple accounts.