Hadoop2.7.1 HA安装部署详解
Hadoop集群规划
目标：创建2个NameNode，做高可用，一个NameNode挂掉，另一个能够启动；一个运行Yarn，3台DataNode，3台Zookeeper集群，做高可用。
在 hadoop2中通常由两个NameNode组成，一个处于active状态，另一个处于standby状态。Active NameNode对外提供服务，而Standby NameNode则不对外提供服务，仅同步active namenode的状态，以便能够在它失败时快速进行切换。
这里还配置了一个zookeeper集群，用于ZKFC（DFSZKFailoverController）故障转移，当Active NameNode挂掉了，会自动切换Standby NameNode为standby状态
安装我都把hadoop和Zookeeper放到了/soft/下面；
0、系统环境安装
操作系统CentOS6.5,64位操作系统，采用最小化安装，为了能够实现目标，采用VMware 虚拟机来搭建6台服务器，所以建议服务器至少有8G内存；vmware使用的是VMware® Workstation 11.0.0 build-2305329；
网络配置如下：
虚拟机配置如下：
1、同步机器时间
<span class="pln">yum install <span class="pun">-<span class="pln">y ntp <span class="com">#安装时间服务</span></span></span></span><span class="pln">ntpdate us<span class="pun">.<span class="pln">pool<span class="pun">.<span class="pln">ntp<span class="pun">.<span class="pln">org <span class="com">#同步时间</span></span></span></span></span></span></span></span>
2、设置主机IP
<span class="pln">vi <span class="pun">/<span class="pln">etc<span class="pun">/<span class="pln">sysconfig<span class="pun">/<span class="pln">network<span class="pun">-<span class="pln">scripts<span class="pun">/<span class="pln">ifcfg<span class="pun">-<span class="pln">eth0</span></span></span></span></span></span></span></span></span></span></span></span></span>
<span class="pln">DEVICE<span class="pun">=<span class="pln">eth0</span></span></span><span class="pln">IPADDR<span class="pun">=<span style="color: #ff0000;"><span class="lit">192.168<span class="pun">.<span class="lit">8.101</span></span></span></span></span></span><span class="pln">NETMASK<span class="pun">=<span class="lit">255.255<span class="pun">.<span class="lit">255.0</span></span></span></span></span><span class="pln">GATEWAY<span class="pun">=<span class="lit">192.168<span class="pun">.<span class="lit">8.2</span></span></span></span></span><span class="pln">HWADDR<span class="pun">=<span class="lit">00<span class="pun">:<span class="lit">0C<span class="pun">:<span class="lit">29<span class="pun">:<span class="lit">56<span class="pun">:<span class="lit">63<span class="pun">:<span class="pln">A1</span></span></span></span></span></span></span></span></span></span></span></span></span><span class="pln">TYPE<span class="pun">=<span class="typ">Ethernet</span></span></span><span class="pln">UUID<span class="pun">=<span class="pln">ecb7f947<span class="pun">-<span class="lit">8a93<span class="pun">-<span class="lit">488c<span class="pun">-<span class="pln">a118<span class="pun">-<span class="pln">ffb011421cac</span></span></span></span></span></span></span></span></span></span></span><span class="pln">ONBOOT<span class="pun">=<span class="pln">yes</span></span></span><span class="pln">NM_CONTROLLED<span class="pun">=<span class="pln">yes</span></span></span><span class="pln">BOOTPROTO<span class="pun">=<span class="pln">none</span></span></span>
然后重启网络服务
<span class="pln">service network restart</span>
查看ip配置
<span class="pln">ifconfig eth0</span>
<span class="pln">eth0 <span class="typ">Link<span class="pln"> encap<span class="pun">:<span class="typ">Ethernet<span class="pln"> <span class="typ">HWaddr<span class="pln"> <span class="lit">00<span class="pun">:<span class="lit">0C<span class="pun">:<span class="lit">29<span class="pun">:<span class="lit">6C<span class="pun">:<span class="lit">20<span class="pun">:<span class="lit">2B</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="pln">inet addr<span class="pun">:<span style="color: #ff0000;"><span class="lit">192.168<span class="pun">.<span class="lit">8.101</span></span></span></span><span class="pln"> <span class="typ">Bcast<span class="pun">:<span class="lit">192.168<span class="pun">.<span class="lit">8.255<span class="pln"> <span class="typ">Mask<span class="pun">:<span class="lit">255.255<span class="pun">.<span class="lit">255.0</span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="pln">inet6 addr<span class="pun">:<span class="pln"> fe80<span class="pun">::<span class="lit">20c<span class="pun">:<span class="lit">29ff<span class="pun">:<span class="pln">fe6c<span class="pun">:<span class="lit">202b<span class="pun">/<span class="lit">64<span class="pln"> <span class="typ">Scope<span class="pun">:<span class="typ">Link</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="pln">UP BROADCAST RUNNING MULTICAST MTU<span class="pun">:<span class="lit">1500<span class="pln"> <span class="typ">Metric<span class="pun">:<span class="lit">1</span></span></span></span></span></span></span><span class="pln">RX packets<span class="pun">:<span class="lit">777<span class="pln"> errors<span class="pun">:<span class="lit">0<span class="pln"> dropped<span class="pun">:<span class="lit">0<span class="pln"> overruns<span class="pun">:<span class="lit">0<span class="pln"> frame<span class="pun">:<span class="lit">0</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="pln">TX packets<span class="pun">:<span class="lit">316<span class="pln"> errors<span class="pun">:<span class="lit">0<span class="pln"> dropped<span class="pun">:<span class="lit">0<span class="pln"> overruns<span class="pun">:<span class="lit">0<span class="pln"> carrier<span class="pun">:<span class="lit">0</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="pln">collisions<span class="pun">:<span class="lit">0<span class="pln"> txqueuelen<span class="pun">:<span class="lit">1000</span></span></span></span></span></span><span class="pln">RX bytes<span class="pun">:<span class="lit">70611<span class="pln"> <span class="pun">(<span class="lit">68.9<span class="pln"> <span class="typ">KiB<span class="pun">)<span class="pln"> TX bytes<span class="pun">:<span class="lit">49955<span class="pln"> <span class="pun">(<span class="lit">48.7<span class="pln"> <span class="typ">KiB<span class="pun">)</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
这就说明我们配置的IP地址是成功的
注意：
使用vmware克隆了5台虚拟机，配置网卡的时候出现报错：”Bring up interface eth0：Device eth0 does not seem to be present，delaying initialization”
解决步骤：
第一步 删除文件70-persistent-net.rules
<span class="pln">rm <span class="pun">-<span class="pln">f <span class="pun">/<span class="pln">etc<span class="pun">/<span class="pln">udev<span class="pun">/<span class="pln">rules<span class="pun">.<span class="pln">d<span class="pun">/<span class="lit">70<span class="pun">-<span class="pln">persistent<span class="pun">-<span class="pln">net<span class="pun">.<span class="pln">rule</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
第二步 修改ifcfg-eth0
<span class="pln">vim <span class="pun">/<span class="pln">etc<span class="pun">/<span class="pln">sysconfig<span class="pun">/<span class="pln">network<span class="pun">-<span class="pln">scripts<span class="pun">/<span class="pln">ifcfg<span class="pun">-<span class="pln">eth0</span></span></span></span></span></span></span></span></span></span></span></span></span>
删除或者注释MAC地址的配置
第三步：重启服务器
<span class="pln">reboot</span>
这样就可以设置新的ip地址了
3、设置主机名
把一台主机的名称改为：hadoop01
<span class="pln">vi /etc/sysconfig/network<br></span>
修改hostname就可以了：
<span class="pln">NETWORKING<span class="pun">=<span class="pln">yes</span></span></span><span class="pln">HOSTNAME<span class="pun">=<span class="pln"><span style="color: #ff0000;">hadoop01</span></span></span></span><span class="pln"><span style="color: #ff0000;">NETWORKING_IPV6=no<br></span></span>
要修改hosts
<span class="pln">vi <span class="pun">/<span class="pln">etc<span class="pun">/<span class="pln">hosts</span></span></span></span></span>
<span class="lit">127.0<span class="pun">.<span class="lit">0.1<span class="pln"> localhost</span></span></span></span><span class="pln">192.168.8.101 hadoop01 192.168.8.102 hadoop02 192.168.8.103 hadoop03 192.168.8.104 hadoop04 192.168.8.105 hadoop05 192.168.8.106 hadoop06</span>
关闭ipv6
1、查看系统是否开启ipv6
a）通过网卡属性查看
命令：ifconfig
注释：有 “inet6 addr：。。。。。。。“ 的表示开启了ipv6功能
b）通过内核模块加载信息查看
命令：lsmod | grep ipv6
2、ipv6关闭方法
在/etc/modprobe.d/dist.conf结尾添加
<span class="pln">alias net<span class="pun">-<span class="pln">pf<span class="pun">-<span class="lit">10<span class="pln"> off</span></span></span></span></span></span><span class="pln">alias ipv6 off</span>
可用vi等编辑器，也可以通过命令：
cat <<EOF>>/etc/modprobe.d/dist.conf
alias net-pf-10 off
alias ipv6 off
EOF
关闭防火墙
<span class="pln">chkconfig iptables stop</span><span class="pln">chkconfig iptables off</span>
改好后重启服务器：
<span class="pln">reboot</span>
hadoop02-hadoop06都需要设置
相关推荐
yaodilu 2020-05-10
SXIAOYI 2020-09-16
有心就有方向 2012-09-03
zfyaixue 2013-06-14
pigsmall 2020-11-19
Ladyseven 2020-07-25
whileinsist 2020-06-24
gufudhn 2020-06-12
冰蝶 2020-06-05
LinuxAndroidAI 2020-06-04
supperme 2020-05-28
e度空间 2020-04-27
云端漂移 2020-04-09
peterwzc 2020-03-17
ebuild 2013-05-14
donghedonghe 2013-05-31
tdeclipse 2011-02-28
linuxprobe0 2013-04-15