Recently I'm working in Azure to implement ETL jobs. The main tool is ADF (Azure Data Factory). This post show some solutions to resolve issue in my...
Archive year
2020
scala ref create dataframe
```txt master MASTERURL --> 运行模式 例：spark://host:port, mesos://host:port, yarn, or local.
PROCESSLOCAL data is in the same JVM as the running code. This is the best locality possible NODELOCAL data is on the same node. Examples might be in...
import airflow from airflow.models import DAG from airflow.operators.pythonoperator import PythonOperator
Whitening Transformation
Recently reading a blog Structured Streaming in PySpark It's implemented in Databricks platform. Then I try to implement in my local Spark. Some...
Batch Normalization is one of important parts in our NN.
Vanilla gradient descent, aka batch gradient descent, computes the gradient of the cost function w.r.t. to the parameters θ