大数据学习5(基于yarn的高可用集群搭建)
NN-1 | NN-2 | DN | ZK | ZKFC | JNN | RS | NM(和DN在一起) | |
node0001 | * | * | * | |||||
node0002 | * | * | * | * | * | * | ||
node0003 | * | * | * | * | * | |||
node0004 | * | * | * | * |
1,修改配置文件
cd /opt/sxt/hadoop-2.6.5/etc/hadoop
mv mapred-site.xml.template mapred-site.xml
vi mapred-site.xml (插入)
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
vi yarn-site.xml
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>cluster1</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>node0003</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>node0004</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>node0002:2181,node0003:2181,node0004:2181</value> </property>
2,分发
scp mapred-site.xml yarn-site.xml node0002:`pwd`
scp mapred-site.xml yarn-site.xml node0003:`pwd`
scp mapred-site.xml yarn-site.xml node0004:`pwd`
3,两个ResourceManager相互免密钥
node0003:cd .ssh
ssh-keygen -t dsa -P ‘‘ -f ./id_dsa
cat id_dsa.pub >> authorized_keys
scp id_dsa.pub node0004:`pwd`/node0003.pub
node0004:cd .ssh
cat node0003.pub >> authorized_keys
ssh-keygen -t dsa -P ‘‘ -f ./id_dsa
cat id_dsa.pub >> authorized_keys
scp id_dsa.pub node0003:`pwd`/node0004.pub
node0003:
cat node0004.pub >> authorized_keys
相互访问没问题ok
4,启动
zkServer.sh start(启动zookeeper,002,003,004)
start-dfs.sh (启动hdfs,mapreduce基于hdfs,0001)
start-yarn.sh (启动yarn,0001)
jps 查看 0002,0003,0004 都有NodeManager进程
yarn-daemon.sh start resourcemanager (启动resourcemanager ,0003,0004)
jps 查看 0003,0004 都有ResourceManager 进程
5,浏览器访问
node0003:8088
6,停止
node0003,node0004: yarn-daemon.sh stop resourcemanager (停止resourcemanager)
node0001: stop-yarn.sh (停止nodemanager)
node0001: stop-dfs.sh
node0002,node0003,node0004: zkServer.sh stop