DevOps(6)Spark Deployment on VM 2

DevOps(6)Spark Deployment on VM 2
 
1. Workers Sync from Master
under SPARK_HOME/conf/spark-env.sh
 
It works in cluster mode.
#SCALA_HOME=/opt/scala/scala-2.10.3
export SPARK_WORKER_MEMORY=768m
#SPARK_CLASSPATH='/opt/profiles-spark/lib/*.jar'
export SPARK_JAVA_OPTS="-Dbuild.env=sparkvm"
#export SPARK_DAEMON_JAVA_OPTS="-Dbuild.env=sparkvm"
#jobJar='/opt/profiles-spark/lib/*.jar'
#export JAVA_OPTS="-Dbuild.env=sparkvm"
export USER=carl
 
It requires we extend the serialization interface.
 
2. Pre-place the binary files
Did not try that. I do not like this way.
 
 
References:
http://sillycat.iteye.com/blog/2166583
 

相关推荐