AI ENGINEERING
Use Databricks Models with VS Code Copilot and Copilot CLI
How to use one local proxy to connect Databricks Model Serving to both VS Code Copilot Custom Endpoint and GitHub Copilot CLI.
Topic / 9 articles
DATA ENGINEERING
Blog posts in the DATA ENGINEERING category
DATA ENGINEERING
PySpark Dataframe Transformation
Notes on moving from Scala to PySpark, using Python dataclasses for data models, and creating DataFrames from Row objects.
DATA ENGINEERING
Databricks Wheel Job
Recently I successfully deploy my python wheel to Databricks Cluster. Here are some tips if you plan to deploy pyspark.
DATA ENGINEERING
Azure Data Factory (Data Flow)
Recently I'm working in Azure to implement ETL jobs. The main tool is ADF (Azure Data Factory). This post show some solutions to resolve issue in my work.
DATA ENGINEERING
Spark Dataframe window function
A Scala reference for working with window functions in Spark SQL.
DATA ENGINEERING
Spark SQL
Spark submit options and tuning notes covering executors, memory, parallelism, and resource allocation on YARN.
DATA ENGINEERING
Spark Optimization
Data locality is how close data is to the code processing it. There are several levels of locality based on the data’s current location. In order from closest to farthest:
DATA ENGINEERING
Airflow
Airflow DAG examples for scheduling Python tasks, configuring retries and queues, and separating task implementations.
DATA ENGINEERING
Spark Structured Streaming
Recently reading a blog Structured Streaming in PySpark It's implemented in Databricks platform. Then I try to implement in my local Spark. Some tricky issue happened during my...