hadoop(6)2.6.0 and Example

hadoop(6)2.6.0 and Example
 
1. Prepare
ProtocolBuffer Installation
> ./autogen.sh 
> ./configure --prefix=/home/carl/tool/protobuf-3.0.0
> make
> make check
> make install
 
Add that to path.
> protoc --version
libprotoc 3.0.0
 
Need to switch to 2.5.0 version
 
JDK Installation
> java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_
 
MAVEN Installation
 
Unzip and place in the right directory, adding to path
> mvn --version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T11:29:23-06:00)
Maven home: /opt/maven
 
Add this to stop the out of memory error
export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=128m"
 
Hadoop Src Build
> mvn package -Pdist -DskipTests -Dtar
 
Find the release dist file  hadoop-2.6.0.tar.gz
 
I get “JAVA_HOME not set” Error message, I already have the JAVA_HOME in .profile, but I added it in 
/opt/hadoop/etc/hadoop/hadoop-env.sh as well.
export JAVA_HOME="/usr/lib/jvm/java-8-oracle"
 
Configure the SSH on master
> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
 
Format file system
Format the filesystem
>hdfs namenode -format
 
All the configuration should take reference from 
http://sillycat.iteye.com/blog/2090186
 
It works. Change the etc/hadoop/slaves files to have a machine list there.
 
Command line to start the HDFS and YARN
> sbin/start-dfs.sh
> sbin/start-yarn.sh
 
 
Tips
Error Message:
autoreconf: not found
Solution:
>sudo apt-get install autoconf
 
Error Message:
Can't exec "libtoolize": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 345, <GEN7> line 6.
autoreconf: failed to run libtoolize: No such file or directory
autoreconf: libtoolize is needed because this package uses Libtool
Solution:
> sudo apt-get install libtool
 
Error Message:
checking how to run the C++ preprocessor... /lib/cpp
configure: error: in `/home/carl/install/protobuf':
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
Solution:
> sudo apt-get install g++
 
Error Message:
protoc version is 'libprotoc 3.0.0', expected version is '2.5.0'
Solution:
Switch to 2.5.0
 
References:
http://sillycat.iteye.com/blog/1556106  Hadoop doc and Installation on ubuntu with an old version
http://sillycat.iteye.com/blog/1556107 cluster of the old version
http://sillycat.iteye.com/blog/2084169  yet another resource negotiator 2.4.0
http://sillycat.iteye.com/blog/2090186 cluster 2.4.1
http://sillycat.iteye.com/blog/2090937 eclipse and example
 
 
 
 
 

相关推荐