Azure Data Factory (Data Flow)
Recently I'm working in Azure to implement ETL jobs. The main tool is ADF (Azure Data Factory). This post show some solutions to resolve issue in my work.
2020#
Spark run faster and faster
- Cluster Optimization
- Parameters Optimization
- Code Optimization
Spark Structured Streaming
Spark Structured Streaming
Recently reading a blog Structured Streaming in PySpark It's implemented in Databricks platform. Then I try to implement in my local Spark. Some tricky issue happened during my work.
Batch Normalization
Batch Normalization is one of important parts in our NN.
Why need Normalization
This paper title tells me the reason Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift - accelerating traning - reduce internal covariate shift