Note 01 1 of 4
Spark Dataframe window function
A Scala reference for working with window functions in Spark SQL.
Collection
Spark & Data Engineering
Learn DataFrame operations, SQL tuning, Spark optimization, and structured streaming.
4 notes in this path
Note 02 2 of 4
Spark SQL
Configure Spark SQL and tune distributed queries for more efficient execution.
Note 03 3 of 4
Spark Optimization
Data locality is how close data is to the code processing it. There are several levels of locality based on the data’s current location. In order from closest to farthest:
Note 04 4 of 4
Spark Structured Streaming
Recently reading a blog Structured Streaming in PySpark It's implemented in Databricks platform. Then I try to implement in my local Spark. Some tricky issue happened during my work.