Spark Data Skew: How to Detect and Fix It

Data skew happens when a small number of partitions hold far more data than the rest. Spark runs a stage as parallel tasks, and the stage only finishes when its slowest task finishes, so one oversized partition can keep a whole cluster waiting.

What data skew looks like

Skew usually appears after a wide transformation such as a join, groupBy or window, where rows are redistributed by key. If a handful of keys are very common, their partitions grow much larger than the others.

Symptoms in the Spark UI

Open the stage detail page and compare the task duration and shuffle read size percentiles. A max value many times higher than the median is the classic sign of skew.

Symptoms in your job

Typical signs are a stage stuck at the last few tasks, disk spills on a few executors, and out-of-memory errors that disappear when you add memory.

Why skew is expensive

Skewed tasks run longer, spill to disk and hold executors while the rest of the cluster sits idle. You pay for the whole cluster for the duration of the slowest task.

[[cta:benchmark]]

How to fix data skew

Enable Adaptive Query Execution

In Spark 3.x, set spark.sql.adaptive.enabled and spark.sql.adaptive.skewJoin.enabled to true. AQE detects oversized shuffle partitions at runtime and splits them into smaller tasks.

Broadcast the smaller side of a join

If one side of a join fits in memory, a broadcast join avoids the shuffle entirely, and with it the skewed partition.

Salt hot keys

Add a random suffix to the most frequent keys so their rows spread across several partitions, then aggregate twice: once on the salted key and once on the original key.

Filter or isolate outliers

Null or default keys are a common cause of skew. Filter them out, or process them in a separate branch of the job.

Key takeaways

Check stage percentiles in the Spark UI first, turn on AQE, and use broadcast joins or salting for the keys AQE cannot handle.

[[cta:demo]]

‍