Spark
Advanced
Dynamic Allocation
In 3.0, Spark has introduced a beta feature where dynamic allocation can be run without external shuffle service. This is achieved by adding intelligence within Spark Dynamic Scaler to track the location of shuffle data and removing executors accordingly. This feature can be enabled using spark.dynamicAllocation.shuffleTracking.enabled.
"options": {
"type": "spark",
"conf" = @{
"spark.driver.cores" = "1"
"spark.executor.instances" = "1"
"spark.executor.memory" = "512m"
"spark.executor.cores" = "1"
"spark.dynamicAllocation.maxExecutors" = "3"
"spark.dynamicAllocation.enabled" = "true"
"spark.dynamicAllocation.shuffleTracking.enabled" = "true"
}
}