yarn+hdfs

使用yum安装CDH5

时间2014-02-1009:02:00CSDN博客

原文http://blog.csdn.net/beckham008/article/details/19028853

安装ZooKeeper（集群模式）

NodeType:

node229,node452,node440

1.所有节点安装zookeeper,zookeeper-server

yuminstall-yzookeeperzookeeper-server

2.所有节点修改zookeeper配置文件

vi/etc/zookeeper/conf/zoo.cfg

增加节点的配置

server.1=node229:2888:3888

server.2=node452:2888:3888

server.3=node440:2888:3888

3.所有节点初始化zookeeper-server

每个节点的myid唯一

node229:servicezookeeper-serverinit--myid=1

node452:servicezookeeper-serverinit--myid=2

node440:servicezookeeper-serverinit--myid=3

4.所有节点启动zookeeper

servicezookeeper-serverstart

5.查看zookeeper状态

zookeeper-serverstatus

安装CDH（集群模式，HDFS+YARN）

NodeType:

namenode:node229

datanode:node229,node452,node440

yarn-resourcemanager:node452

yarn-nodemanager:node229,node452,node440

mapreduce-historyserver:node440

yarn-proxyserver:node440

node1:

yuminstallhadoop-hdfs-namenode

node2:

yuminstallhadoop-yarn-resourcemanager

node3:

yuminstallhadoop-mapreduce-historyserverhadoop-yarn-proxyserver

所有节点:

yuminstallhadoop-client

yuminstallhadoop-yarn-nodemanagerhadoop-hdfs-datanodehadoop-mapreduce

部署CDH

1.部署HDFS

(1)配置文件

core-site.xml

<property>

<name>fs.defaultFS</name>

<value>hdfs://node229:8020</value>

</property>

<property>

<name>fs.trash.interval</name>

<value>1440</value>

</property>

hdfs-site.xml

<property>

<name>dfs.permissions.superusergroup</name>

<value>hadoop</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>/hadoop/hdfs/namenode</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>/hadoop/hdfs/datanode</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

slaves

node209

node452

node440

(2)创建namenode和datanode文件夹

namenode:

mkdir-p/hadoop/hdfs/namenode

chown-Rhdfs:hdfs/hadoop/hdfs/namenode

chmod700/hadoop/hdfs/namenode

datanode:

mkdir-p/hadoop/hdfs/datanode

chown-Rhdfs:hdfs/hadoop/hdfs/datanode

chmod700/hadoop/hdfs/datanode

(3)格式化namenode

sudo-uhdfshadoopnamenode-format

(4)启动hdfs

namenode(node209):

servicehadoop-hdfs-namenodestart

datanode(node209,node452,node440):

servicehadoop-hdfs-datanodestart

(forxin`cd/etc/init.d;lshadoop-hdfs-*`;dosudoservice$xstart;done)

(5)查看hdfs状态

sudo-uhdfshdfsdfsadmin-report

sudo-uhdfshadoopfs-ls-R-h/

(6)创建HDFS临时文件夹

sudo-uhdfshadoopfs-mkdir/tmp

sudo-uhdfshadoopfs-chmod-R1777/tmp

http://101.227.253.62:50070

2.部署YARN

(1)配置YARN

mapred-site.xml:

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>node440:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>node440:19888</value>

</property>

yarn-site.xml

<property>

<name>yarn.resourcemanager.address</name>

<value>node452:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>node452:8030</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>node452:8088</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>node452:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>node452:8033</value>

</property>

<property>

<description>Classpathfortypicalapplications.</description>

<name>yarn.application.classpath</name>

<value>

$HADOOP_CONF_DIR,

$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,

$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,

$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,

$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*

</value>

</property>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.nodemanager.local-dirs</name>

<value>/hadoop/data/yarn/local</value>

</property>

<property>

<name>yarn.nodemanager.log-dirs</name>

<value>/hadoop/data/yarn/logs</value>

</property>

<property>

<name>yarn.aggregation.enable</name>

<value>true</value>

</property>

<property>

<description>Wheretoaggregatelogs</description>

<name>yarn.nodemanager.remote-app-log-dir</name>

<value>/var/log/hadoop-yarn/apps</value>

</property>

<property>

<name>yarn.app.mapreduce.am.staging-dir</name>

<value>/user</value>

</property>

(2)所有nodemanager创建本地目录

sudomkdir-p/hadoop/data/yarn/local

sudochown-Ryarn:yarn/hadoop/data/yarn/local

sudomkdir-p/hadoop/data/yarn/logs

sudochown-Ryarn:yarn/hadoop/data/yarn/logs

(3)创建HDFS目录

sudo-uhdfshadoopfs-mkdir-p/user/history

sudo-uhdfshadoopfs-chmod-R1777/user/history

sudo-uhdfshadoopfs-chownyarn/user/history

sudo-uhdfshadoopfs-mkdir-p/var/log/hadoop-yarn

sudo-uhdfshadoopfs-chownyarn:mapred/var/log/hadoop-yarn

(4)启动YARN

ResourceManager(node452):

sudoservicehadoop-yarn-resourcemanagerstart

NodeManager(node209,node452,node440):

sudoservicehadoop-yarn-nodemanagerstart

MapReduceJobHistoryServer(node440):

sudoservicehadoop-mapreduce-historyserverstart

(5)创建YARN的HDFS用户目录

sudo-uhdfshadoopfs-mkdir-p/user/$USER

sudo-uhdfshadoopfs-chown$USER/user/$USER

(6)测试

查看节点状态

yarnnode-all-list

hadoopjar/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jarrandomwriterinput

(7)关闭

sudoservicehadoop-yarn-resourcemanagerstop

sudoservicehadoop-yarn-nodemanagerstop

sudoservicehadoop-mapreduce-historyserverstop

http://101.227.253.63:8088/

安装和部署HBase

NodeType:

hbase-master:node229,node440

hbase-regionserver:node229,node452,node440

hbase-thrift:node440

hbase-rest:node229,node452,node440

1.安装HBase

(1)修改配置

/etc/security/limits.conf，增加配置

hdfs-nofile32768

hbase-nofile32768

hdfs-site.xml，增加配置

<property>

<name>dfs.datanode.max.xcievers</name>

<value>4096</value>

</property>

(2)安装HBase

hbase-master:

sudoyuminstallhbasehbase-master

hbase-regionserver:

sudoyuminstallhbasehbase-regionserver

hbase-thrift:

sudoyuminstallhbase-thrift

hbase-rest:

sudoyuminstallhbase-rest

(3)配置HBase

hbase-site.xml

<property>

<name>hbase.rest.port</name>

<value>60050</value>

</property>

<property>

<name>hbase.zookeeper.quorum</name>

<value>node229,node452,node440</value>

</property>

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

<property>

<name>hbase.tmp.dir</name>

<value>/hadoop/hbase</value>

</property>

<property>

<name>hbase.rootdir</name>

<value>hdfs://node229:8020/hbase/</value>

</property>

(4)创建本地目录

mkdir-p/hadoop/hbase

chown-Rhbase:hbase/hadoop/hbase

(5)创建hbase的HDFS目录

sudo-uhdfshadoopfs-mkdir/hbase/

sudo-uhdfshadoopfs-chownhbase/hbase

(6)启动HBase

hbase-master:

sudoservicehbase-masterstart

hbase-regionserver:

sudoservicehbase-regionserverstart

hbase-thrift:

sudoservicehbase-thriftstart

hbase-rest:

sudoservicehbase-reststart

http://101.227.253.62:60010