Blog#

August 12, 2024
in ML
1 min read

Model Registry

July 18, 2021
in k8s
2 min read

Setup Minikube

November 18, 2020
in Azure
4 min read

Azure Data Factory (Data Flow)

Recently I'm working in Azure to implement ETL jobs. The main tool is ADF (Azure Data Factory). This post show some solutions to resolve issue in my work.

March 1, 2020
in spark
1 min read

Spark Dataframe window function

scala ref

create dataframe

February 21, 2020
in spark
2 min read

Spark run faster and faster

Cluster Optimization
Parameters Optimization
Code Optimization

February 21, 2020
in spark
4 min read

Spark SQL

February 11, 2020
in airflow
1 min read

Airflow

February 11, 2020
in ML
1 min read

Whitening Transformation

February 8, 2020
in spark
8 min read

Spark Structured Streaming

Recently reading a blog Structured Streaming in PySpark It's implemented in Databricks platform. Then I try to implement in my local Spark. Some tricky issue happened during my work.

February 4, 2020
in ML
1 min read

Batch Normalization

Batch Normalization is one of important parts in our NN.

Why need Normalization

This paper title tells me the reason Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift - accelerating traning - reduce internal covariate shift