Skip to content

Setup Minikube#

Installation#

brew upgrade 
brew install minikube
brew install kubectl
# minikube kubectl -- get pods -A

Start/Stop Cluster#

#minikube start
minikube start --driver=hyperkit
minikube stop

Interact with Cluster#

minikube dashboard --alsologtostderr

Deploy application#

kubectl create deployment balanced --image=k8s.gcr.io/echoserver:1.4  
kubectl expose deployment balanced --type=LoadBalancer --port=8080

Spark on K8s#

Official Reference

Build Spark image#

  • brew install spark
  • build image via docker-image-tool.sh 
    sh /usr/local/Cellar/apache-spark/3.1.2/bin/docker-image-tool.sh -m -t spark-docker build
  • check image 
    minikube ssh
docker image ls

Submit a sample job#

  • Find master node of K8s 
    kubectl cluster-info
  • submit job 
    # spark-submit --master k8s://https://192.168.64.2:8443 --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=3 --conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v2.4.5 --conf spark.kubernetes.namespace=default local:///usr/local/opt/apache-spark/libexec/examples/jars/spark-examples_2.12-2.4.5.jar

bin/spark-submit \
--master k8s://https://192.168.99.100:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=512m \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=512m \
--conf spark.executor.cores=1 \
--conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v2.4.5 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar