hadoop2.2.0 centos 编译安装详解
搭建环境:Centosx6.464bit
1、安装JDK
我这里用的是64位机，要下载对应的64位的JDK,下载地址：http://www.oracle.com/technetwork/cn/java/javase/downloads/jdk7-downloads-1880260-zhs.html，选择对应的JDK版本，解压JDK，然后配置环境变量,
[html]viewplaincopy
vi/etc/profile
注：这里有的人喜欢配置在当前用户里，我这里是配置的全局。
[html]viewplaincopy
exportPATH
exportJAVA_HOME=/opt/jdk1.7
exportPATH=$PATH:$JAVA_HOME/bin
source/etc/profile
测试下JDK是否安装成功:java-version
[html]viewplaincopy
javaversion"1.7.0_45"
Java(TM)SERuntimeEnvironment(build1.7.0_45-b18)
JavaHotSpot(TM)64-BitServerVM(build24.45-b08,mixedmode)
2、编译前的准备(maven)
maven官方下载地址，可以选择源码编码安装，这里就直接下载编译好的就可以了
[html]viewplaincopy
wgethttp://mirror.bit.edu.cn/apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.zip
解压文件后，同样在/etc/profie里配置环境变量
[html]viewplaincopy
exportMAVEN_HOME=/opt/maven3.1.1
exportPATH=$PATH:$MAVEN_HOME/bin
验证配置是否成功:mvn-version
[html]viewplaincopy
ApacheMaven3.1.1(0728685237757ffbf44136acec0402957f723d9a;2013-09-1723:22:22+0800)
Mavenhome:/opt/maven3.1.1
Javaversion:1.7.0_45,vendor:OracleCorporation
Javahome:/opt/jdk1.7/jre
Defaultlocale:en_US,platformencoding:UTF-8
OSname:"linux",version:"2.6.32-358.el6.x86_64",arch:"amd64",family:"unix"
3、编译hadoop
这个地方你将会遇到各式各样的头疼问题
首先官方下载hadoop源码
[html]viewplaincopy
wgethttp://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
如果是你32bit的机器，可以直接下载官方已经编译好的包，64bit的机子跑编译好的包跑不了。
由于maven国外服务器可能连不上，先给maven配置一下国内镜像，在maven目录下，conf/settings.xml,在<mirrors></mirros>里添加，原本的不要动
[html]viewplaincopy
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
同样，在<profiles></profiles>内新添加
[html]viewplaincopy
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>localprivatenexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>localprivatenexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
编译clean
[html]viewplaincopy
cdhadoop2.2.0-src
mvncleaninstall–DskipTests
发现异常
[html]viewplaincopy
[ERROR]Failedtoexecutegoalorg.apache.hadoop:hadoop-maven-plugins:2.2.0:protoc(compile-protoc)onprojecthadoop-common:org.apache.maven.plugin.MojoExecutionException:'protoc--version'didnotreturnaversion->[Help1]
[ERROR]
[ERROR]Toseethefullstacktraceoftheerrors,re-runMavenwiththe-eswitch.
[ERROR]Re-runMavenusingthe-Xswitchtoenablefulldebuglogging.
[ERROR]
[ERROR]Formoreinformationabouttheerrorsandpossiblesolutions,pleasereadthefollowingarticles:
[ERROR][Help1]http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR]Aftercorrectingtheproblems,youcanresumethebuildwiththecommand
[ERROR]mvn<goals>-rf:hadoop-common
hadoop2.2.0编译需要protoc2.5.0的支持，所以还要下载protoc，下载地址:https://code.google.com/p/protobuf/downloads/list，要下载2.5.0版本噢
对protoc进行编译安装前先要装几个依赖包:gcc,gcc-c++,make如果已经安装的可以忽略
[html]viewplaincopy
yuminstallgcc
yumintallgcc-c++
yuminstallmake
安装protoc
[html]viewplaincopy
tar-xvfprotobuf-2.5.0.tar.bz2
cdprotobuf-2.5.0
./configure--prefix=/opt/protoc/
make&&makeinstall
安装完配置下环境变量，就不多说了，跟上面过程一样。
别急，还不要着急开始编译安装，不然又是各种错误,需要安装cmake,openssl-devel,ncurses-devel依赖
[html]viewplaincopy
yuminstallcmake
yuminstallopenssl-devel
yuminstallncurses-devel
目前的2.2.0的SourceCode压缩包解压出来的code有个bug需要patch后才能编译。否则编译hadoop-auth会提示下面错误：
[ERROR]Failedtoexecutegoalorg.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile(default-testCompile)onprojecthadoop-auth:Compilationfailure:Compilationfailure:
[ERROR]/home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]cannotaccessorg.mortbay.component.AbstractLifeCycle
[ERROR]classfilefororg.mortbay.component.AbstractLifeCyclenotfound
Patch:https://issues.apache.org/jira/browse/HADOOP-10110
ok,现在可以进行编译了，
[html]viewplaincopy
mvnpackage-Pdist,native-DskipTests-Dtar
现在可以拿出你的手机，玩会游戏了，慢慢等吧！
[html]viewplaincopy
[INFO]------------------------------------------------------------------------
[INFO]ReactorSummary:
[INFO]
[INFO]ApacheHadoopMain................................SUCCESS[3.709s]
[INFO]ApacheHadoopProjectPOM.........................SUCCESS[2.229s]
[INFO]ApacheHadoopAnnotations.........................SUCCESS[5.270s]
[INFO]ApacheHadoopAssemblies..........................SUCCESS[0.388s]
[INFO]ApacheHadoopProjectDistPOM....................SUCCESS[3.485s]
[INFO]ApacheHadoopMavenPlugins.......................SUCCESS[8.655s]
[INFO]ApacheHadoopAuth................................SUCCESS[7.782s]
[INFO]ApacheHadoopAuthExamples.......................SUCCESS[5.731s]
[INFO]ApacheHadoopCommon..............................SUCCESS[1:52.476s]
[INFO]ApacheHadoopNFS.................................SUCCESS[9.935s]
[INFO]ApacheHadoopCommonProject......................SUCCESS[0.110s]
[INFO]ApacheHadoopHDFS................................SUCCESS[1:58.347s]
[INFO]ApacheHadoopHttpFS..............................SUCCESS[26.915s]
[INFO]ApacheHadoopHDFSBookKeeperJournal.............SUCCESS[17.002s]
[INFO]ApacheHadoopHDFS-NFS............................SUCCESS[5.292s]
[INFO]ApacheHadoopHDFSProject........................SUCCESS[0.073s]
[INFO]hadoop-yarn.......................................SUCCESS[0.335s]
[INFO]hadoop-yarn-api...................................SUCCESS[54.478s]
[INFO]hadoop-yarn-common................................SUCCESS[39.215s]
[INFO]hadoop-yarn-server................................SUCCESS[0.241s]
[INFO]hadoop-yarn-server-common.........................SUCCESS[15.601s]
[INFO]hadoop-yarn-server-nodemanager....................SUCCESS[21.566s]
[INFO]hadoop-yarn-server-web-proxy......................SUCCESS[4.754s]
[INFO]hadoop-yarn-server-resourcemanager................SUCCESS[20.625s]
[INFO]hadoop-yarn-server-tests..........................SUCCESS[0.755s]
[INFO]hadoop-yarn-client................................SUCCESS[6.748s]
[INFO]hadoop-yarn-applications..........................SUCCESS[0.155s]
[INFO]hadoop-yarn-applications-distributedshell.........SUCCESS[4.661s]
[INFO]hadoop-mapreduce-client...........................SUCCESS[0.160s]
[INFO]hadoop-mapreduce-client-core......................SUCCESS[36.090s]
[INFO]hadoop-yarn-applications-unmanaged-am-launcher....SUCCESS[2.753s]
[INFO]hadoop-yarn-site..................................SUCCESS[0.151s]
[INFO]hadoop-yarn-project...............................SUCCESS[4.771s]
[INFO]hadoop-mapreduce-client-common....................SUCCESS[24.870s]
[INFO]hadoop-mapreduce-client-shuffle...................SUCCESS[3.812s]
[INFO]hadoop-mapreduce-client-app.......................SUCCESS[15.759s]
[INFO]hadoop-mapreduce-client-hs........................SUCCESS[6.831s]
[INFO]hadoop-mapreduce-client-jobclient.................SUCCESS[8.126s]
[INFO]hadoop-mapreduce-client-hs-plugins................SUCCESS[2.320s]
[INFO]ApacheHadoopMapReduceExamples..................SUCCESS[9.596s]
[INFO]hadoop-mapreduce..................................SUCCESS[3.905s]
[INFO]ApacheHadoopMapReduceStreaming.................SUCCESS[7.118s]
[INFO]ApacheHadoopDistributedCopy....................SUCCESS[11.651s]
[INFO]ApacheHadoopArchives............................SUCCESS[2.671s]
[INFO]ApacheHadoopRumen...............................SUCCESS[10.038s]
[INFO]ApacheHadoopGridmix.............................SUCCESS[6.062s]
[INFO]ApacheHadoopDataJoin...........................SUCCESS[4.104s]
[INFO]ApacheHadoopExtras..............................SUCCESS[4.210s]
[INFO]ApacheHadoopPipes...............................SUCCESS[9.419s]
[INFO]ApacheHadoopToolsDist..........................SUCCESS[2.306s]
[INFO]ApacheHadoopTools...............................SUCCESS[0.037s]
[INFO]ApacheHadoopDistribution........................SUCCESS[21.579s]
[INFO]ApacheHadoopClient..............................SUCCESS[7.299s]
[INFO]ApacheHadoopMini-Cluster........................SUCCESS[7.347s]
[INFO]------------------------------------------------------------------------
[INFO]BUILDSUCCESS
[INFO]------------------------------------------------------------------------
[INFO]Totaltime:11:53.144s
[INFO]Finishedat:FriNov2216:58:32CST2013
[INFO]FinalMemory:70M/239M
[INFO]------------------------------------------------------------------------
直到看到上面的内容那就说明编译完成了。
编译后的路径在:hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0
[html]viewplaincopy
[root@localhostbin]#./hadoopversion
Hadoop2.2.0
SubversionUnknown-rUnknown
Compiledbyrooton2013-11-22T08:47Z
Compiledwithprotoc2.5.0
Fromsourcewithchecksum79e53ce7994d1628b240f09af91e1af4
Thiscommandwasrunusing/data/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar
可以看出hadoop的版本
[html]viewplaincopy
[[email protected]]#filelib//native/*
lib//native/libhadoop.a:currentararchive
lib//native/libhadooppipes.a:currentararchive
lib//native/libhadoop.so:symboliclinkto`libhadoop.so.1.0.0'
lib//native/libhadoop.so.1.0.0:<spanstyle="color:#ff0000;">ELF64-bitLSBsharedobject,x86-64,version1</span>(SYSV),dynamicallylinked,notstripped
lib//native/libhadooputils.a:currentararchive
lib//native/libhdfs.a:currentararchive
lib//native/libhdfs.so:symboliclinkto`libhdfs.so.0.0.0'
lib//native/libhdfs.so.0.0.0:<spanstyle="color:#ff0000;">ELF64-bitLSBsharedobject,x86-64,version1</span>(SYSV),dynamicallylinked,notstripped
注意红色字体部分，如果下载官网的编译好的包，这里显示的是32-bit。
hadoop编译成功，下面可以来部署集群。
5、部署集群准备
两台以上机器，修改hostname,ssh免登陆，关闭防火墙等
5.1、创建新用户
[html]viewplaincopy
useraddhadoop
suhadoop
注意以下操作有些需要root权限
5.2、修改主机名
[html]viewplaincopy
vi/etc/sysconfig/network
[html]viewplaincopy
hostnamemaster
注销一下系统
[html]viewplaincopy
[root@master~]#
变成master了,修改生效
5.3、修改hosts
[html]viewplaincopy
vi/etc/hosts
新增你的主机IP和HOSTNAME
192.168.10.10master
192.168.10.11slave1
5.4、ssh免登陆
查看ssh
[html]viewplaincopy
[root@localhostdata]#rpm-qa|grepssh
libssh2-1.4.2-1.el6.x86_64
openssh-5.3p1-84.1.el6.x86_64
openssh-server-5.3p1-84.1.el6.x86_64
缺少openssh-clients,
[html]viewplaincopy
yuminstallopenssh-clients
修改/etc/ssh/sshd_config
RSAAuthenticationyes
PubkeyAuthenticationyes
AuthorizedKeysFile.ssh/authorized_keys
把这三行放开保存
然后servicesshdrestart
现在开始配置无密登录
[html]viewplaincopy
[hadoop@master~]$cd/home/hadoop/
[hadoop@master~]$ssh-keygen-trsa
一路回车
[html]viewplaincopy
[hadoop@master~]$cd.ssh/
[[email protected]]$cpid_rsa.pubauthorized_keys
[[email protected]]$chmod600authorized_keys
把authorized_keys复制到其他要无密的机器上
[html]viewplaincopy
[[email protected]][email protected]:/home/hadoop/.ssh/
记得这里是以要以root权限过去，不然会报权限错误
一般情况到这里就可以无密登录了，可是我怎么还是需要密码，经过一翻搜寻才知道这是centos6.4版本的问题，《关于centosssh无密登录失败的记录》
[html]viewplaincopy
[[email protected]]$sshslave1
Lastlogin:MonNov2514:49:252013frommaster
[hadoop@slave1~]$
看到已经变成slave1了，说明成功鸟
6、开始集群配置工作
配置之前在要目录下创建三个目录，用来放hadooop文件和日志数据
[html]viewplaincopy
[hadoop@master~]$mkdir-pdfs/name
[hadoop@master~]$mkdir-pdfs/data
[hadoop@master~]$mkdir-ptemp
把之前编译成功的版本移到hadoop目录下，注意目录权限问题
下面就开始配置文件
6.1hadoop-env.sh
找到JAVA_HOME，把路径改为实际地址
6.2yarn-env.sh
同6.1
6.3slave
配置所有slave节点
6.4core-site.xml
[html]viewplaincopy
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>//系统分布式URL
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hadoop/temp</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
注意fs.defaultFS为2.2.0新的变量，代替旧的：fs.default.name
6.5、hdfs-site.xml
配置namenode、datanode的本地目录信息
[html]viewplaincopy
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/dfs/data,/mnt/d1,/mnt/d2,/mnt/d3<tableborder="1"cellspacing="0"cellpadding="0"style="background:black;"><tbody><tr><tdvalign="top"><spanstyle="background-color:rgb(240,240,240);"></value></span></td></tr></tbody></table></property>
<spanstyle="white-space:pre"></span><property>
<name>dfs.replication</name>
<value>3</value>
</property>
<spanstyle="white-space:pre"></span><property>
<name>dfs.webhdfs.enabled</name>
<spanstyle="white-space:pre"></span><value>true</value>
</property>
新的：dfs.namenode.name.dir，旧：dfs.name.dir，新：dfs.datanode.name.dir，旧：dfs.data.dir
dfs.replication确定datablock的副本数目，hadoop基于rackawareness(机架感知)默认复制3份分block,（同一个rack下两个，另一个rack下一份，按照最短距离确定具体所需block,一般很少采用跨机架数据块，除非某个机架down了）
6.6、mapred-site.xml
配置其使用Yarn框架执行map-reduce处理程序
这个地方需要把mapred-site.xml.template复制重新命名
[html]viewplaincopy
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value><spanstyle="font-family:Consolas,'CourierNew',Courier,mono,serif;line-height:18px;">master</span>:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
新的计算框架取消了实体上的jobtracker,故不需要再指定mapreduce.jobtracker.addres，而是要指定一种框架，这里选择yarn.备注2：hadoop2.2.还支持第三方的计算框架，但没怎么关注过。
配置好以后将$HADOOP_HOME下的所有文件，包括hadoop目录分别copy到其它3个节点上。
6.7、yarn-site.xml
配置ResourceManager,NodeManager的通信端口，WEB监控端口等
[html]viewplaincopy
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
[html]viewplaincopy
<tablecellspacing="0"cellpadding="0"class="t1"><tbody><tr><tdvalign="middle"class="td1"><pclass="p1"><property></p><pclass="p1"><name>yarn.nodemanager.resource.memory-mb</name>//配置内存</p><pclass="p1"><value>15360</value></p><pclass="p1"></property></p></td></tr></tbody></table>
[html]viewplaincopy
<tablecellspacing="0"cellpadding="0"class="t1"><tbody><tr><tdvalign="middle"class="td1"><pclass="p1"><spanstyle="background-color:rgb(255,255,255);">到这里基本配置好了，把所有复制到其他的slave节点。</span></p></td></tr></tbody></table>
7、启动hadoop
这里你可以进行环境变量设置，不举例了
7.1、格式化namenode
[html]viewplaincopy
[hadoop@masterhadoop]$cd/home/hadoop/hadoop-2.2.0/bin/
[hadoop@masterbin]$./hdfsnamenode-format
7.2、启动hdfs
[html]viewplaincopy
[hadoop@masterbin]$cd../sbin/
[hadoop@mastersbin]$./start-dfs.sh
这时候在master中输入jps应该看到namenode和secondarynamenode服务启动，slave中看到datanode服务启动
7.3、启动yarn
[html]viewplaincopy
[hadoop@mastersbin]$./start-yarn.sh
master中应该有ResourceManager服务，slave中应该有nodemanager服务
查看集群状态：./bin/hdfsdfsadmin–report
查看文件块组成：./bin/hdfsfsck/-files-blocks
查看各节点状态:http://192.168.10.10:50070
查看resourcemanager上cluster运行状态:http://192.168.10.11:8088
8、安装中要注意的事项
8.1、注意版本，机器是32bit还是64位
8.2、注意依赖包的安装
8.3、写配置文件注意”空格“，特别是从别的地方copy的时候
8.4、关闭所有节点的防火墙
如果有看到类似"noroutetohost"这样的异常，基本就是防火墙没关
记得关的时候要切换到root帐号
[html]viewplaincopy
（1）重启后永久性生效：
开启：chkconfigiptableson
关闭：chkconfigiptablesoff
（2）即时生效，重启后失效：
开启：serviceiptablesstart
关闭：serviceiptablesstop
8.5、开启datanode后自动关闭
基本是因为namenode和datanode的clusterID不一致，可以参考《解决hadoop集群中datanode启动后自动关闭的问题》
其他一些特殊异常只能google之了
8.6nodatanodetostop
删除/tmp目录下的
adoop-daemon.sh代码，脚本是通过pid文件来停止hadoop服务的，而集群配置是使用的默认配置，pid文件位于/tmp目录下，对比/tmp目录下hadooppid文件中的进程id和psax查出来的进程id，发现两个进程id不一致，终于找到了问题的根源。
赶紧去更新hadoop的配置吧！
修改hadoop-env.sh中的：HADOOP_PID_DIR=hadoop安装路径
9、运行测试例子
[html]viewplaincopy
[hadoop@masterbin]$./yarnjar../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarrandomwriter/home/hadoop/dfs/input/
这里要注意不要用-jar,不然会报异常“Exceptioninthread"main"java.lang.NoClassDefFoundError:org/apache/hadoop/util/ProgramDriver”
[html]viewplaincopy
[hadoop@masterbin]$./yarnjar../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount/home/hadoop/dfs/input//home/hadoop/dfs/output/
在input下面新建两个文件
[html]viewplaincopy
$mkdir/dfs/input%echo‘hello,world’>>input/file1.in
$echo‘hello,ruby’>>input/file2.in
./bin/hadoopfs-mkdir-p/home/hadoop/dfs/input
./bin/hadoopfs–put/home/hadoop/dfs/input/home/hadoop/test/test_wordcount/in
查看wordcount的计算结果：
$bin/hadoopfs-cat/home/hadoop/test/test_wordcount/out/*
hadoop1
hello1
ruby