DevOps(6)Spark Deployment on VM 2
DevOps(6)Spark Deployment on VM 2
1. Workers Sync from Master
under SPARK_HOME/conf/spark-env.sh
It works in cluster mode.
#SCALA_HOME=/opt/scala/scala-2.10.3
export SPARK_WORKER_MEMORY=768m
#SPARK_CLASSPATH='/opt/profiles-spark/lib/*.jar'
export SPARK_JAVA_OPTS="-Dbuild.env=sparkvm"
#export SPARK_DAEMON_JAVA_OPTS="-Dbuild.env=sparkvm"
#jobJar='/opt/profiles-spark/lib/*.jar'
#export JAVA_OPTS="-Dbuild.env=sparkvm"
export USER=carl
It requires we extend the serialization interface.
2. Pre-place the binary files
Did not try that. I do not like this way.
References:
http://sillycat.iteye.com/blog/2166583
相关推荐
学习web前端 2020-10-27
bingdata 2020-10-15
chunyang 2020-09-30
JustHaveTry 2020-08-26
jaybeat 2020-08-20
sayhaha 2020-08-10
白开水 2020-07-19
liushen0 2020-07-17
loganwz 2020-07-05
李少阳 2020-06-16
maicowxp 2020-06-16
btqszl 2020-06-15
花落兰亭 2020-06-09
lrcoop 2020-06-03
Wyt00 2020-05-30
nebulali 2020-05-27
today0 2020-05-27
LCFlxfldy 2020-05-15