Partitioning Strategies: Optimizing Data Layout for Performance

Why Partitioning Matters

Partitioning determines how data is physically organized on disk or across cluster nodes. Good partitioning enables partition pruning — skipping irrelevant data during queries — which can reduce I/O by 10-100x.

Key trade-offs:

• Too few partitions — large files, poor parallelism, OOM errors
• Too many partitions — high metadata overhead, small file problem
• Wrong partition key — data skew, uneven resource utilization

Time-Based Partitioning

The most common pattern for analytical workloads. Partition by date (and optionally hour) for efficient time-range queries.

# Good: partition by date for daily queries
df.write.partitionBy("year", "month", "day") \
 .parquet("s3://lake/transactions/")

# Query only reads relevant partitions
SELECT * FROM transactions 
WHERE year = 2026 AND month = 7 AND day = 11

# iceberg: hidden partitioning with transform
ALTER TABLE transactions ADD PARTITION FIELD 
 days(ts) -- automatic daily partitioning

Rule of thumb: Target 128MB-1GB per partition file. For daily partitions processing 1TB/day, this means ~1000 partitions per day.

Hash Partitioning for Joins

When joining large datasets on a common key, hash-partitioning both datasets by that key ensures co-located data and minimizes network shuffles.

# Pre-bucket by customer_id for efficient joins
df.write \
 .bucketBy(256, "customer_id") \
 .sortBy("customer_id") \
 .saveAsTable("transactions_bucketed")

# Delta Lake: Z-ORDER for multi-dimensional pruning
OPTIMIZE transactions
ZORDER BY (customer_id, transaction_date)

Monitoring Partition Health

Regularly check for partition skew and small files:

# Check partition sizes
df.groupBy("date").count().show()

# Iceberg: check for small files
SELECT path, file_size_in_bytes 
FROM table.files(table_name) 
ORDER BY file_size_in_bytes ASC
LIMIT 20

# Fix: compact small files
ALTER TABLE transactions EXECUTE optimize 
WHERE date = '2026-07-11'