Ideas into practice

AutoGen HttpClient

Now we can add our proxy config in our client, which will resolve connection issue.

How to execute python modules

We can use internal runpy to execute different moduls in our project.

Model Registry

Problem: How to introduce ml-based production/features to cross-functional teams.

Setup Minikube

Set up Minikube on macOS, deploy a sample application, and submit a Spark job to a local Kubernetes cluster.

Azure Data Factory (Data Flow)

Recently I'm working in Azure to implement ETL jobs. The main tool is ADF (Azure Data Factory). This post show some solutions to resolve issue in my work.

Spark Dataframe window function

A Scala reference for working with window functions in Spark SQL.

Spark SQL

Spark submit options and tuning notes covering executors, memory, parallelism, and resource allocation on YARN.

Spark Optimization

Data locality is how close data is to the code processing it. There are several levels of locality based on the data’s current location. In order from closest to farthest:

Airflow

Airflow DAG examples for scheduling Python tasks, configuring retries and queues, and separating task implementations.

Whitening transformation