MacBook Pro 安装Hadoop 2.7.1 伪分布式

飞鸿踏雪0

2020-02-01

配置ssh

允许本机远程连接：

系统偏好设置->共享->点击选中远程登录【否则报错：ssh: connect to host localhost port 22: Connection refused】

允许免密ssh

ssh-keygen -t rsa【一直回车到底】

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod og-wx ~/.ssh/authorized_keys

再次验证

[@wooluwakerdeMBP:~]$ ssh localhost

Last login: Sat Feb 1 16:29:20 2020 from ::1

安装java

[@wooluwakerdeMBP:~]$ java -version

java version "1.8.0_231"

Java(TM) SE Runtime Environment (build 1.8.0_231-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)

安装Hadoop

下载hadoop

　Hadoop 版本：hadoop-2.7.1.tar.gz

解压Hadoop

　　　　 tar -zxvf hadoop-2.7.1.tar.gz

修改配置文件 hadoop-env.sh

# 添加 JAVA_HOMEJAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home

修改配置文件 core-site.xml

# 配置namenode的主机名和端口号，hadoop临时目录
<configuration>
    
    <!-- 指定HDFS老大（namenode）的通信地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://0.0.0.0:9000</value>
    </property>
    <!-- 指定hadoop运行时产生文件的存储路径 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/walker/software/hadoop/hadoop-2.7.1/temp</value>
    </property>

</configuration>

修改配置文件 hdfs-site.xml
- 默认副本数3，修改为1
- dfs.namenode.name.dir指明fsimage存放目录，多个目录用逗号隔开。
- dfs.datanode.data.dir指定块文件存放目录，多个目录逗号隔开

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
　　 <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/Users/walker/software/hadoop/hadoop-2.7.1/tmp/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/Users/walker/software/hadoop/hadoop-2.7.1/tmp/hdfs/data</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:50070</value>
    </property>
</configuration>

修改配置文件 mapred-site.xml

<configuration>
    
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

</configuration>

修改配置文件 yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>

</configuration>

配置Hadoop环境变量

export HADOOP_HOME=/Users/walker/software/hadoop/hadoop-2.7.1
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

格式化NameNode

$ cd $HADOOP_HOME

$ bin/hdfs namenode -format

## 格式化成功后的结果

20/02/01 18:13:53 INFO common.Storage: Storage directory /Users/walker/software/hadoop/hadoop-2.7.1/tmp/hdfs/name has been successfully formatted.
20/02/01 18:13:53 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/02/01 18:13:53 INFO util.ExitUtil: Exiting with status 0
20/02/01 18:13:53 INFO namenode.NameNode: SHUTDOWN_MSG: 

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at wooluwakerdeMBP.lan/192.168.2.194

启动Hadoop

$ start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
20/02/01 18:17:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:27 WARN conf.Configuration: bad conf file: element not <property>
Starting namenodes on [0.0.0.0]
The authenticity of host ‘0.0.0.0 (0.0.0.0)‘ can‘t be established.
ECDSA key fingerprint is SHA256:RYiYsOpM6+P18wPsX0uqkxG/SrG2QtHZocO3PJrJOD8.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added ‘0.0.0.0‘ (ECDSA) to the list of known hosts.
0.0.0.0: starting namenode, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/hadoop-walker-namenode-wooluwakerdeMBP.lan.out
localhost: starting datanode, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/hadoop-walker-datanode-wooluwakerdeMBP.lan.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/hadoop-walker-secondarynamenode-wooluwakerdeMBP.lan.out
20/02/01 18:17:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
20/02/01 18:17:48 WARN conf.Configuration: bad conf file: element not <property>
starting yarn daemons
starting resourcemanager, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/yarn-walker-resourcemanager-wooluwakerdeMBP.lan.out
localhost: starting nodemanager, logging to /Users/walker/software/hadoop/hadoop-2.7.1/logs/yarn-walker-nodemanager-wooluwakerdeMBP.lan.out
$ jps
32960 Jps
32640 DataNode
32737 SecondaryNameNode
32567 NameNode
32920 NodeManager
32842 ResourceManager