hadoop详细安装和配置

hadoop版本:hadoop-2.2.0-cdh5.0.0-beta-1

jdk版本:jdk-7u40-linux-x64

环境准备:

10.95.3.100master1

10.95.3.101master2

10.95.3.103slave1

10.95.3.104slave2

说明:本安装没有进行hdfsHA配置,NameNode和SecondaryNameNode都在master1上,其他三个节点作为DN节点。

安装步骤(JDK的安装省略):

1、添加hadoop用户

root身份登录master1

添加用户:useradd–d/home/hadoop-mhadoop

给hadoop用户设置密码:passwdhadoop

2、配置hosts

以root用户登录各台机器,vi/etc/hosts文件,修改如下:

#127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.95.3.100              master1
10.95.3.101              master2
10.95.3.103              slave1
10.95.3.104              slave2

3、配置master1到各台机器的免密码登录

以hadoop用户登录master1,进入/home/hadoop目录下,执行:

ssh-keygen-trsa

注意一定不要输入密码,这样,在/home/hadoop/.ssh目录下将会生成id_rsa,id_rsa.pub两个文件

然后再分别执行:

ssh-copy-id-i~/.ssh/id_rsa.pubhadoop@master1

ssh-copy-id-i~/.ssh/id_rsa.pubhadoop@master2

ssh-copy-id-i~/.ssh/id_rsa.pubhadoop@slave1

ssh-copy-id-i~/.ssh/id_rsa.pubhadoop@slave2

这样,就把master1的公钥加入到了master的authorized_keys文件中。可以测试下从master1ssh到其他各台机器是否不需要输入密码。

注意:文件夹的权限

sudochmod755.ssh/

sudochmod644.ssh/authorized_keys

4、解压hadoop-2.2.0-cdh5.0.0-beta-1.tar.gz,设置hadoop的安装目录为/dp/hadoop

设置环境变量:vi~/.bashrc

export HADOOP_HOME=/dp/hadoop
export HADOOP_PID_DIR=/dp/hadoop_pid_dir
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=/dp/hadoop
export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$ANT_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$FLUME_HOME/bin:$SQOOP_HOME/bin:$OOZIE_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:.

5、进入/dp/hadoop/etc/hadoop目录,修改配置文件

core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master1</value>
        </property>

        <property>
                <name>fs.trash.interval</name>
                <value>10080</value>
        </property>
        <property>
                <name>fs.trash.checkpoint.interval</name>
                <value>10080</value>
        </property>
        <property>
                <name>topology.script.file.name</name>
                <value>/dp/hadoop/etc/hadoop/rack.py</value>
                </property>
        <property>
                <name>topology.script.number.args</name>
                <value>6</value>
        </property>
        <property>
               <name>hadoop.security.group.mapping</name>
               <value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>
        </property>
<!--
        <property> 
                <name>hadoop.native.lib</name> 
                <value>false</value> 
                <description>Should native hadoop libraries, if present, be used.</description> 
        </property>
-->
<!--
        <property>
                <name>ha.zookeeper.quorum</name>
                <value>master:2181,slave2:2181,slave6:2181</value>
        </property>      
        
        <property>
                <name>hadoop.proxyuser.hadoop.hosts</name>
                <value>*</value>
        </property>
        <property>
                <name>hadoop.proxyuser.hadoop.groups</name>
                <value>*</value>
        </property>
-->
</configuration>

hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
        <property>
                <name>dfs.blocksize</name>
                <value>16m</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/dp/data/hadoop</value>
        </property>

        <property>
                <name>dfs.namenode.http-address</name>
                <value>master1:50070</value>
        </property>

        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>master1:50090</value>
        </property>
        <property>
                <name>dfs.webhdfs.enabled</name>
                <value>true</value>
        </property>
        <property>
                <name>dfs.datanode.max.xcievers</name>
                <value>1000000</value>
        </property>
        <property>
                <name>dfs.balance.bandwidthPerSec</name>
                <value>104857600</value>
                <description>
                        Specifies the maximum amount of bandwidth that each datanode
                        can utilize for the balancing purpose in term of
                        the number of bytes per second.
                </description>
        </property>
        <property>
                <name>dfs.hosts.exclude</name>
                <value>/dp/hadoop/etc/hadoop/excludes</value>
                <description>Names a file that contains a list of hosts that are
                        not permitted to connect to the namenode.  The full pathname of the
                        file must be specified.  If the value is empty, no hosts are
                        excluded.</description>
         </property>

上述的配置要手工创建excludes文件。

map-site.xml

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>master1:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>master1:19888</value>
        </property>

<!--        <property>
                <name>mapreduce.history.server.delegationtoken.renewer</name>
                <value>true</value>
        </property>
-->
        <property>
                <name>mapreduce.output.fileoutputformat.compress</name>
                <value>true</value>
        </property>
        <property>
                <name>mapreduce.output.fileoutputformat.compress.type</name>
                <value>BLOCK</value>
        </property>
        <property>
                <name>mapreduce.output.fileoutputformat.compress.codec</name>
                <value>org.apache.hadoop.io.compress.SnappyCodec</value>
        </property>
        <property>
                <name>mapreduce.map.output.compress</name>
                <value>true</value>
        </property>
        <property>
        <property>
                <name>mapreduce.map.output.compress.codec</name>
                <value>org.apache.hadoop.io.compress.SnappyCodec</value>
        </property>

yarn-site.xml

<configuration>
        <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>master1:8031</value>
        </property>
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>master1:8032</value>
        </property>
        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>master1:8030</value>
        </property>

        <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>master1:8033</value>
        </property>

        <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>master1:8088</value>
        </property>
        <property>
                <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
                <value>10000</value>
        </property>
        <property>
                <description>Classpath for typical applications.</description>
                <name>yarn.application.classpath</name>
                <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,
                        $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
                        $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
                        $YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*,
                        $YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
                <name>yarn.nodemanager.local-dirs</name>
                <value>/dp/data/yarn/local</value>
        </property>
        <property>
                <name>yarn.nodemanager.log-dirs</name>
                <value>/dp/data/yarn/logs</value>
        </property>
        <property>
                <description>Where to aggregate logs</description>
                <name>yarn.nodemanager.remote-app-log-dir</name>
                <value>/dp/data/yarn/logs</value>
        </property>
        <property>
                <name>yarn.app.mapreduce.am.staging-dir</name>
                <value>/user</value>
        </property>
        <property>
                <description>Amount of physical memory, in MB, that can be allocated
                  for containers.</description>
                <name>yarn.nodemanager.resource.memory-mb</name>
                <value>2048</value>
        </property>

6、配置机架感知,在/dp/hadoop/etc/hadoop下添加rack.datarack.py文件

rack.data

default /rack/default

10.95.3.101 master2 /rack/rack1
10.95.3.103 slave1  /rack/rack1
10.95.3.104 slave2  /rack/rack1

rack.py

#!/bin/env python

import sys,os,time

pwd = os.path.realpath( __file__ )
rack_file = os.path.dirname(pwd) + "/rack.data"

rack_list = [ l.strip().split() for l in open(rack_file).readlines() if len(l.strip().split()) > 1 ]
rack_map = {}
for item in rack_list:
        for host in item[:-1]:
                rack_map[host] = item[-1]
rack_map['default'] = 'default' in rack_map and rack_map['default'] or '/default/rack'
rack_result = [av in rack_map and rack_map[av] or rack_map['default'] for av in sys.argv[1:]]
#print rack_map, rack_result
print ' '.join( rack_result )

f = open('/tmp/rack.log','a+')
f.writelines( "[%s] %sn" % (time.strftime("%F %T"),str(sys.argv)))
f.close()

7、修改slaves文件:

master2
slave1
slave

8、将master1上配置好的hadoop拷贝到其他节点上去

9、执行start-dfs.sh启动hadoop,注意如果防火墙没关闭,有可能造成DN连接不上NN

10、执行hadoopdfsadmin-report命令,查看hadoop是否正常启动

相关推荐