hadoop伪分布式环境搭建

下载安装包

Apache Hadoop 官网下载hadoop安装包 hadoop-2.9.1.tar.gz,并解压。

[root@d1 local]# pwd
/usr/local
[root@d1 local]# tar -zxvf hadoop-2.9.1.tar.gz
[root@d1 local]# ln -s hadoop-2.9.1 hadoop

修改配置

修改解压后的目录中的文件夹etc/hadoop下的xml配置文件(如果文件不存在,则自己创建)

hadoop-env.sh修改以下配置:

[root@d1 hadoop]# pwd
/usr/local/hadoop
[root@d1 hadoop]# vi etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64

slaves文件修改为以下配置:

[root@d1 hadoop]# vi etc/hadoop/slaves
d1

注:以下四个XML配置文件,需在标签<configuration>和</configuration>之间增加配置项。

[root@d1 hadoop]# vi etc/hadoop/mapred-site.xml
<configuration>
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
 </property>
</configuration>

core-site.xml(其中“d1”是在/etc/hosts中设置的host,如果未设置,则换为localhost):

[root@d1 hadoop]# vi etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
 <name>fs.default.name</name>
 <value>hdfs://d1:9000</value>
 </property>
 <property>
 <name>dfs.replication</name>
 <value>1</value>
 </property>
 <property>
 <name>hadoop.tmp.dir</name>
 <value>/hadoop/hadoop_tmp_dir</value>
 </property>
 <property>
 <name>dfs.namenode.name.dir</name>
 <value>/hadoop/dfs/name</value>
 </property>
 <property>
 <name>dfs.datanode.data.dir</name>
 <value>/hadoop/dfs/data</value>
 </property>
</configuration>

yarn-site.xml:

[root@d1 hadoop]# vi etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>
 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
 </property>
</configuration>

启动服务

格式化HDFS:

bin/hadoop namenode -format

启动HDFS:

sbin/start-dfs.sh

启动YARN:

sbin/start-yarn.sh

验证是否启动成功

[root@d1 hadoop]# jps
21697 Jps
20612 SecondaryNameNode
20308 NameNode
20933 NodeManager
20827 ResourceManager
20414 DataNode
[root@d1 hadoop]#

参照:http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

hadoop伪分布式环境搭建

相关推荐