Hadoop(一)：CentOS 安装hadoop

砚浅

2013-06-01

我的CSDN博客地址：http://blog.csdn.net/qidong7

为Hadoop集群安装SSH

一、验证是否安装了ssh

[hadoop-user@brain /]# which ssh
/usr/bin/ssh
[hadoop-user@brain /]# which sshd
/usr/sbin/sshd
[hadoop-user@brain /]# which ssh-keygen
/usr/bin/ssh-keygen

如果收到类似这样的错误信息

/usr/bin/which no ssh in (/usr/bin:/bin:/usr/bin...

您可以通过Linux安装包管理器安装OpenSSH(www.openssh.com)或者直接下载源码。

二、生成SSH密钥对

务必要避免输入口令，否则，主节点每次试图访问其他节点时，你都得手动输入这个口令

[hadoop-user@brain /]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop-user/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop-user/.ssh/id_rsa.
Your public key has been saved in /home/hadoop-user/.ssh/id_rsa.pub.
The key fingerprint is:
2e:e1:91:cf:85:48:f9:92:8f:2d:e5:36:92:15:e0:1a root@brain
The key's randomart image is:
+--[ RSA 2048]----+
|      .          |
|     . o         |
|    E + .        |
|     + = o       |
|    . B S .      |
|     . / .       |
|      B X        |
|       = .       |
|                 |
+-----------------+

生成密钥对之后，公钥的形式为：

[hadoop-user@brain /]# 
more  /home/hadoop-user/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0pmwcaf/xgUrDQfK4ZePQEqjXhYPSV2iUsRxunk20RxerLOpw5SNJ+3gb2jVSnuncAlABkS
yi8VCCJVmV4b0y6kCP6ap9jdcRJhxLB33zd5IyHMyo9zjdo8j2xRnFtv1SfaqkhIt8/By1E2+Si3dlIE61DhRGtdxGJNkVv0O4+8AANKcEk
C5LK8Hr+pEBBVlM9m9OHYJ9GdekQcMQQC/CtrZohzHznDm70tJ1EtnidYdUcBz31FuOfeJjuEapbkn6Az9HwDEQD8cemtED0QK/aW9FLjko
zid5nLjJ7ZJwmkmq5V3rLSE2AbqGnGtYSYCezJFFHmUpNyG35vYjpGj....

三、将密钥公布(本人的target为localhost)

[hadoop-user@brain hadoop-1.2.0]# scp ~/.ssh/id_rsa.pub root@localhost:~/master_key

[hadoop-user@brain/]#sshlocalhost

Lastlogin:SatJun118:12:592013from192.168.1.102

四、安装hadoop

1、解压

tarxzvfhadoop-1.2.0.tar.gz

2、配置Hadoop

2.1.配置JDK

hadoop-env.sh

exportJAVA_HOME=/usr/java/jdk1.7.0_17

2.2.配置conf/core-site.xml、conf/hdfs-site.xml、conf/mapred-site.xml文件

core-site.xml

<configuration>     
<property>  
       <name>fs.default.name</name>  
       <value>hdfs://localhost:9000</value>  
</property>  
<property>
       <name>hadoop.tmp.dir</name>
        <value>/usr/app/hadoop-hdfs</value>
        </property>
</configuration>

hdfs-site.xml

<configuration>     
   <property>  
         <name>dfs.replication</name>  
         <value>1</value>  
    </property>  
</configuration>

mapred-site.xml

<configuration>     
    <property>  
         <name>mapred.job.tracker</name>  
         <value>localhost:9001</value>  
    </property>  
</configuration>

3、启动hadoop

cd/home/brain/hadoop-1.2.0/bin

1.格式化文件系统

./hadoopnamenode–format

2.启动/关闭hadoop

启动所有任务start-all.sh/stop-all.sh

启动关闭HDFS:start-dfs.sh/stop-dfs.sh

启动关闭MapReduce:start-mapred.sh/stop-mapred.sh

3.用jps命令查看进程,确保有

namenode,

dataNode,

JobTracker,

TaskTracker

如果没有namenode,

1、先运行stop-all.sh

2、格式化namdenode，不过在这之前要先删除原目录，即core-site.xml下配置的<name>hadoop.tmp.dir</name>所指向的目录，然后运行hadoopnamenode-format

3、运行start-all.sh

4、检查执行

访问

http://192.168.1.77:50030(MapReduce的Web页面)

http://192.168.1.77:50070(HDfS的web页面)

本人安装过程中出现没有namenode，格式化后仍然没有，这时可以查看下日志，发现端口被占。

有问题查看日志HADOOP_HOME/logs/

另外如果访问

http://192.168.1.77:50030访问不到，可能是防火墙的原因，可以关闭防火墙，再次访问

serviceiptablesstop

如果是，修改防火墙配置，对50030端口不监控；

修改防火墙监控端口

vi/etc/sysconfig/iptables

中加

-ARH-Firewall-1-INPUT-mstate--stateNEW-mtcp-ptcp--dport50030-jACCEPT

就不对50030端口进行监控了

请关注微博http://weibo.com/u/3272022684

砚浅

0 关注 0 粉丝 0 动态

关注关注

安科网

Hadoop(一)：CentOS 安装hadoop

砚浅

砚浅

砚浅