hadoop(6)2.6.0 and Example
hadoop(6)2.6.0 and Example
1. Prepare
ProtocolBuffer Installation
> git clone https://github.com/google/protobuf.git
> ./autogen.sh
> ./configure --prefix=/home/carl/tool/protobuf-3.0.0
> make
> make check
> make install
Add that to path.
> protoc --version
libprotoc 3.0.0
Need to switch to 2.5.0 version
JDK Installation
> java -version
java version "1.6.0_45"
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_
MAVEN Installation
Unzip and place in the right directory, adding to path
> mvn --version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T11:29:23-06:00)
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T11:29:23-06:00)
Maven home: /opt/maven
Add this to stop the out of memory error
export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=128m"
Hadoop Src Build
> mvn package -Pdist -DskipTests -Dtar
Find the release dist file hadoop-2.6.0.tar.gz
I get “JAVA_HOME not set” Error message, I already have the JAVA_HOME in .profile, but I added it in
/opt/hadoop/etc/hadoop/hadoop-env.sh as well.
export JAVA_HOME="/usr/lib/jvm/java-8-oracle"
Configure the SSH on master
> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Format file system
Format the filesystem
>hdfs namenode -format
All the configuration should take reference from
http://sillycat.iteye.com/blog/2090186
It works. Change the etc/hadoop/slaves files to have a machine list there.
Command line to start the HDFS and YARN
> sbin/start-dfs.sh
> sbin/start-yarn.sh
Tips
Error Message:
autoreconf: not found
Solution:
>sudo apt-get install autoconf
Error Message:
Can't exec "libtoolize": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 345, <GEN7> line 6.
autoreconf: failed to run libtoolize: No such file or directory
autoreconf: failed to run libtoolize: No such file or directory
autoreconf: libtoolize is needed because this package uses Libtool
Solution:
> sudo apt-get install libtool
Error Message:
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/carl/install/protobuf':
configure: error: in `/home/carl/install/protobuf':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
Solution:
> sudo apt-get install g++
Error Message:
protoc version is 'libprotoc 3.0.0', expected version is '2.5.0'
Solution:
Switch to 2.5.0
References:
http://sillycat.iteye.com/blog/1556106 Hadoop doc and Installation on ubuntu with an old version
http://sillycat.iteye.com/blog/1556107 cluster of the old version
http://sillycat.iteye.com/blog/2084169 yet another resource negotiator 2.4.0
http://sillycat.iteye.com/blog/2090186 cluster 2.4.1
http://sillycat.iteye.com/blog/2090937 eclipse and example
相关推荐
changjiang 2020-11-16
minerd 2020-10-28
WeiHHH 2020-09-23
Aleks 2020-08-19
WeiHHH 2020-08-17
飞鸿踏雪0 2020-07-26
tomli 2020-07-26
deyu 2020-07-21
strongyoung 2020-07-19
eternityzzy 2020-07-19
Elmo 2020-07-19
飞鸿踏雪0 2020-07-09
飞鸿踏雪0 2020-07-04
xieting 2020-07-04
WeiHHH 2020-06-28
genshengxiao 2020-06-26
Hhanwen 2020-06-25