PySpark Fundamentals: Distributed Data Processing at Scale

What is PySpark?

PySpark is the Python API for Apache Spark — a unified analytics engine for large-scale data processing. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Key advantages over single-machine processing:

• In-memory computation — up to 100x faster than Hadoop MapReduce for certain workloads
• Lazy evaluation — builds an optimized execution plan before running
• Unified API — same codebase for batch, streaming, SQL, and ML workloads
• Horizontal scaling — add nodes to handle larger datasets

SparkSession and DataFrames

The SparkSession is your entry point to Spark functionality. It creates a bridge between your Python code and the distributed Spark engine.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, avg, count, when

# Initialize SparkSession
spark = SparkSession.builder \
 .appName("AcaciaFund Analytics") \
 .config("spark.sql.shuffle.partitions", "200") \
 .getOrCreate()

# Read data from various sources
df = spark.read.parquet("s3://bucket/transactions/")
df_json = spark.read.json("s3://bucket/market-data/")
df_csv = spark.read.option("header", True).csv("s3://bucket/reports/")

# Basic transformations
result = df \
 .filter(col("amount") > 10000) \
 .groupBy("category") \
 .agg(
 count("*").alias("transaction_count"),
 avg("amount").alias("avg_amount")
 ) \
 .orderBy(col("avg_amount").desc())

result.show()

Transformations and Actions

Spark operations fall into two categories:

• Transformations (lazy) — filter(), select(), groupBy(), join(), withColumn()
• Actions (eager) — show(), collect(), count(), write()

The Catalyst optimizer automatically optimizes your query plan, combining operations and eliminating unnecessary shuffles.

# Window functions for time-series analysis
from pyspark.sql.window import Window
from pyspark.sql.functions import lag, lead, dense_rank

window_spec = Window.partitionBy("asset_class").orderBy("date")

df_with_returns = df \
 .withColumn("prev_close", lag("close", 1).over(window_spec)) \
 .withColumn("daily_return", 
 (col("close") - col("prev_close")) / col("prev_close")) \
 .withColumn("volatility_30d", 
 avg("daily_return").over(
 Window.partitionBy("asset_class")
 .orderBy("date")
 .rowsBetween(-30, -1)
 )
 )

Performance Optimization

Key techniques for production PySpark workloads:

• Partitioning — Control data distribution with repartition() and coalesce()
• Caching — Use cache() or persist() for DataFrames reused across actions
• Broadcast joins — Automatically broadcasts small tables to avoid shuffles
• Bucketing — Pre-partition data by join keys for efficient joins

# Cache frequently accessed data
transactions = spark.read.parquet("s3://transactions/").cache()

# Broadcast join for small dimension table
from pyspark.sql.functions import broadcast
enriched = transactions.join(broadcast(risk_scores), "customer_id")

# Optimize partition count for output
result.repartition(200, "date") \
 .write \
 .partitionBy("date") \
 .parquet("s3://output/enriched-transactions/")