hadoop 2.7.1 高可用安装部署
Hadoop集群规划
目标：创建2个NameNode，做高可用，一个NameNode挂掉，另一个能够启动；一个运行Yarn，3台DataNode，3台Zookeeper集群，做高可用。
在hadoop2中通常由两个NameNode组成，一个处于active状态，另一个处于standby状态。Active NameNode对外提供服务，而Standby NameNode则不对外提供服务，仅同步active namenode的状态，以便能够在它失败时快速进行切换。
这里还配置了一个zookeeper集群，用于ZKFC（DFSZKFailoverController）故障转移，当Active NameNode挂掉了，会自动切换Standby NameNode为standby状态
安装我都把hadoop和Zookeeper放到了/soft/下面；
http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz
下面关于Hadoop的文章您也可能喜欢，不妨看看：
0、系统环境安装
操作系统CentOS6.5,64位操作系统，采用最小化安装，为了能够实现目标，采用VMware 虚拟机来搭建6台服务器，所以建议服务器至少有8G内存；vmware使用的是VMware® Workstation 11.0.0 build-2305329；
网络配置如下：
虚拟机配置如下：
1、同步机器时间
<span class="pln">yum install </span><span class="pun">-</span><span class="pln">y ntp </span><span class="com">#安装时间服务</span><span class="pln">ntpdate us</span><span class="pun">.</span><span class="pln">pool</span><span class="pun">.</span><span class="pln">ntp</span><span class="pun">.</span><span class="pln">org </span><span class="com">#同步时间</span>
2、设置主机IP
<span class="pln">vi </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">sysconfig</span><span class="pun">/</span><span class="pln">network</span><span class="pun">-</span><span class="pln">scripts</span><span class="pun">/</span><span class="pln">ifcfg</span><span class="pun">-</span><span class="pln">eth0</span>
<span class="pln">DEVICE</span><span class="pun">=</span><span class="pln">eth0</span><span class="pln">IPADDR</span><span class="pun">=</span><span style="color: #ff0000;"><span class="lit">192.168</span><span class="pun">.</span><span class="lit">8.101</span></span><span class="pln">NETMASK</span><span class="pun">=</span><span class="lit">255.255</span><span class="pun">.</span><span class="lit">255.0</span><span class="pln">GATEWAY</span><span class="pun">=</span><span class="lit">192.168</span><span class="pun">.</span><span class="lit">8.2</span><span class="pln">HWADDR</span><span class="pun">=</span><span class="lit">00</span><span class="pun">:</span><span class="lit">0C</span><span class="pun">:</span><span class="lit">29</span><span class="pun">:</span><span class="lit">56</span><span class="pun">:</span><span class="lit">63</span><span class="pun">:</span><span class="pln">A1</span><span class="pln">TYPE</span><span class="pun">=</span><span class="typ">Ethernet</span><span class="pln">UUID</span><span class="pun">=</span><span class="pln">ecb7f947</span><span class="pun">-</span><span class="lit">8a93</span><span class="pun">-</span><span class="lit">488c</span><span class="pun">-</span><span class="pln">a118</span><span class="pun">-</span><span class="pln">ffb011421cac</span><span class="pln">ONBOOT</span><span class="pun">=</span><span class="pln">yes</span><span class="pln">NM_CONTROLLED</span><span class="pun">=</span><span class="pln">yes</span><span class="pln">BOOTPROTO</span><span class="pun">=</span><span class="pln">none</span>
然后重启网络服务
<span class="pln">service network restart</span>
查看ip配置
<span class="pln">ifconfig eth0</span>
<span class="pln">eth0 </span><span class="typ">Link</span><span class="pln"> encap</span><span class="pun">:</span><span class="typ">Ethernet</span><span class="typ">HWaddr</span><span class="lit">00</span><span class="pun">:</span><span class="lit">0C</span><span class="pun">:</span><span class="lit">29</span><span class="pun">:</span><span class="lit">6C</span><span class="pun">:</span><span class="lit">20</span><span class="pun">:</span><span class="lit">2B</span><span class="pln">inet addr</span><span class="pun">:</span><span style="color: #ff0000;"><span class="lit">192.168</span><span class="pun">.</span><span class="lit">8.101</span></span><span class="typ">Bcast</span><span class="pun">:</span><span class="lit">192.168</span><span class="pun">.</span><span class="lit">8.255</span><span class="typ">Mask</span><span class="pun">:</span><span class="lit">255.255</span><span class="pun">.</span><span class="lit">255.0</span><span class="pln">inet6 addr</span><span class="pun">:</span><span class="pln"> fe80</span><span class="pun">::</span><span class="lit">20c</span><span class="pun">:</span><span class="lit">29ff</span><span class="pun">:</span><span class="pln">fe6c</span><span class="pun">:</span><span class="lit">202b</span><span class="pun">/</span><span class="lit">64</span><span class="typ">Scope</span><span class="pun">:</span><span class="typ">Link</span><span class="pln">UP BROADCAST RUNNING MULTICAST MTU</span><span class="pun">:</span><span class="lit">1500</span><span class="typ">Metric</span><span class="pun">:</span><span class="lit">1</span><span class="pln">RX packets</span><span class="pun">:</span><span class="lit">777</span><span class="pln"> errors</span><span class="pun">:</span><span class="lit">0</span><span class="pln"> dropped</span><span class="pun">:</span><span class="lit">0</span><span class="pln"> overruns</span><span class="pun">:</span><span class="lit">0</span><span class="pln"> frame</span><span class="pun">:</span><span class="lit">0</span><span class="pln">TX packets</span><span class="pun">:</span><span class="lit">316</span><span class="pln"> errors</span><span class="pun">:</span><span class="lit">0</span><span class="pln"> dropped</span><span class="pun">:</span><span class="lit">0</span><span class="pln"> overruns</span><span class="pun">:</span><span class="lit">0</span><span class="pln"> carrier</span><span class="pun">:</span><span class="lit">0</span><span class="pln">collisions</span><span class="pun">:</span><span class="lit">0</span><span class="pln"> txqueuelen</span><span class="pun">:</span><span class="lit">1000</span><span class="pln">RX bytes</span><span class="pun">:</span><span class="lit">70611</span><span class="pun">(</span><span class="lit">68.9</span><span class="typ">KiB</span><span class="pun">)</span><span class="pln"> TX bytes</span><span class="pun">:</span><span class="lit">49955</span><span class="pun">(</span><span class="lit">48.7</span><span class="typ">KiB</span><span class="pun">)</span>
这就说明我们配置的IP地址是成功的
注意：
使用vmware克隆了5台虚拟机，配置网卡的时候出现报错：”Bring up interface eth0：Device eth0 does not seem to be present，delaying initialization”
解决步骤：
第一步 删除文件70-persistent-net.rules
<span class="pln">rm </span><span class="pun">-</span><span class="pln">f </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">udev</span><span class="pun">/</span><span class="pln">rules</span><span class="pun">.</span><span class="pln">d</span><span class="pun">/</span><span class="lit">70</span><span class="pun">-</span><span class="pln">persistent</span><span class="pun">-</span><span class="pln">net</span><span class="pun">.</span><span class="pln">rule</span>
第二步 修改ifcfg-eth0
<span class="pln">vim </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">sysconfig</span><span class="pun">/</span><span class="pln">network</span><span class="pun">-</span><span class="pln">scripts</span><span class="pun">/</span><span class="pln">ifcfg</span><span class="pun">-</span><span class="pln">eth0</span>
删除或者注释MAC地址的配置
第三步：重启服务器
<span class="pln">reboot</span>
这样就可以设置新的ip地址了
3、设置主机名
把一台主机的名称改为：hadoop01
<span class="pln">vi /etc/sysconfig/network<br></span>
修改hostname就可以了：
<span class="pln">NETWORKING</span><span class="pun">=</span><span class="pln">yes</span><span class="pln">HOSTNAME</span><span class="pun">=</span><span class="pln"><span style="color: #ff0000;">hadoop01</span></span><span class="pln"><span style="color: #ff0000;">NETWORKING_IPV6=no<br></span></span>
要修改hosts
<span class="pln">vi </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">hosts</span>
<span class="lit">127.0</span><span class="pun">.</span><span class="lit">0.1</span><span class="pln"> localhost</span><span class="pln">192.168.8.101 hadoop01 192.168.8.102 hadoop02 192.168.8.103 hadoop03 192.168.8.104 hadoop04 192.168.8.105 hadoop05 192.168.8.106 hadoop06</span>
关闭ipv6
1、查看系统是否开启ipv6
a）通过网卡属性查看
命令：ifconfig
注释：有 “inet6 addr：。。。。。。。“ 的表示开启了ipv6功能
b）通过内核模块加载信息查看
命令：lsmod | grep ipv6
2、ipv6关闭方法
在/etc/modprobe.d/dist.conf结尾添加
<span class="pln">alias net</span><span class="pun">-</span><span class="pln">pf</span><span class="pun">-</span><span class="lit">10</span><span class="pln"> off</span><span class="pln">alias ipv6 off</span>
可用vi等编辑器，也可以通过命令：
cat <<EOF>>/etc/modprobe.d/dist.conf
alias net-pf-10 off
alias ipv6 off
EOF
关闭防火墙
<span class="pln">chkconfig iptables stop</span><span class="pln">chkconfig iptables off</span>
改好后重启服务器：
<span class="pln">reboot</span>
hadoop02-hadoop06都需要设置
4、安装JDK
将压缩包解压到指定目录，然后编辑环境变量
<span class="pln">vi </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">proflie</span>
在文件后面追加环境变量
<span class="pln">export JAVA_HOME</span><span class="pun">=/</span><span class="pln">soft</span><span class="pun">/</span><span class="pln">jdk1</span><span class="pun">.</span><span class="lit">7.0</span><span class="pln">_80</span><span class="pun">/</span><span class="pln">export PATH</span><span class="pun">=</span><span class="pln">$JAVA_HOME</span><span class="pun">/</span><span class="pln">bin</span><span class="pun">:</span><span class="pln">$JAVA_HOME</span><span class="pun">/</span><span class="pln">jre</span><span class="pun">/</span><span class="pln">bin</span><span class="pun">:</span><span class="pln">$PATH</span>
刷新
<span class="pln">source </span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">profile</span>
可以用java -version测试安装是否正确
相关推荐
飞鸿踏雪0 2020-06-12
changjiang 2020-11-16
minerd 2020-10-28
WeiHHH 2020-09-23
Aleks 2020-08-19
WeiHHH 2020-08-17
飞鸿踏雪0 2020-07-26
tomli 2020-07-26
deyu 2020-07-21
strongyoung 2020-07-19
eternityzzy 2020-07-19
Elmo 2020-07-19
飞鸿踏雪0 2020-07-09
飞鸿踏雪0 2020-07-04
xieting 2020-07-04
WeiHHH 2020-06-28
genshengxiao 2020-06-26