大数据高可用集群环境安装与配置（09）——安装Spark高可用集群

1. 获取spark下载链接

2. 执行命令下载并安装

cd /usr/local/src/
wget http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
tar -zxvf spark-2.4.4-bin-hadoop2.7.tgz
mv spark-2.4.4-bin-hadoop2.7 /usr/local/spark
cd /usr/local/spark/conf
mv spark-env.sh.template spark-env.sh

3. 修改spark-env.sh配置

vi spark-env.sh

在尾部添加下面配置，绑定hadoop的配置文件路径

export JAVA_HOME=/usr/local/java/jdk
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/Hadoop
export SPARK_HOME=/usr/local/spark

export SPARK_MASTER_PORT=7077
# 非高可用集群配置
# export SPARK_MASTER_IP=master
# 高可用集群配置
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=master:2181,master-backup:2181 -Dspark.deploy.zookeeper.dir=/spark"

4. 修改slaves配置

mv slaves.template slaves

vi slaves

删除里面的localhost，添加下面配置

node1
node2
node3

5. 指定spark的主节点

mv spark-defaults.conf.template spark-defaults.conf
vi spark-defaults.conf

添加下面配置

spark.master spark://master:7077,master-backup:7077

6. 修改服务器系统环境变量

7. 安装插件，配置pyspark访问hbase

8. 将spark同步到其他服务器上

rsync -avz /usr/local/spark/ master-backup:/usr/local/spark/
rsync -avz /usr/local/spark/ node1:/usr/local/spark/
rsync -avz /usr/local/spark/ node2:/usr/local/spark/
rsync -avz /usr/local/spark/ node3:/usr/local/spark/

# 将phoenix-5.0.0-HBase-2.0-server.jar同步到其他两台服务器上
rsync -avz /usr/local/hbase/lib/phoenix-5.0.0-HBase-2.0-server.jar master-backup:/usr/local/hbase/lib/
rsync -avz /usr/local/hbase/lib/phoenix-5.0.0-HBase-2.0-server.jar node1:/usr/local/hbase/lib/
rsync -avz /usr/local/hbase/lib/phoenix-5.0.0-HBase-2.0-server.jar node2:/usr/local/hbase/lib/
rsync -avz /usr/local/hbase/lib/phoenix-5.0.0-HBase-2.0-server.jar node3:/usr/local/hbase/lib/

9. 启动spark

10. 测试master切换