Hadoop/HBase 编译安装 snappy 压缩工具（采坑）

准备所需组件

1. Hadoop 2.7.3:
   https://github.com/apache/had...
2. Snappy 1.1.3: https://github.com/google/sna...
3. protobuf-2.5.0: https://github.com/google/pro...

安装依赖包

ant，gcc，gcc-c++，make，autoconf，automake，cmake，gzip，libssl-devel，openssl-devel，libtool，Java7+，maven 3+

大部分可直接 yum install 安装即可

安装 snappy

1. 解压 unzip snappy-1.1.3.zip
2. 进入目录 cd snappy-1.1.3
3. 编译安装 ./configure, make && make install；编译成功后，在 /usr/local/lib/ 下会有如下

# ls /usr/local/lib/
libsnappy.a libsnappy.la libsnappy.so libsnappy.so.1 libsnappy.so.1.3.0

安装 protobuf

1. 解压：unzip protobuf-2.5.0.zip
2. 进入目录：cd protobuf-2.5.0/
3. 编译安装：./configure， make && make install
4. 配置环境变量：export PATH=/usr/local/protoc/bin:$PATH
5. 立即生效：source /etc/profile
6. 查看版本： protoc –version

[root@SparkWorker3 data]# protoc --version
libprotoc 2.5.0

编译 hadoop 源码

1. 安装 jdk，maven，安装依赖库：

   yum -y install svn ncurses-devel gcc*
   yum -y install lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel

2. 解压 unzip hadoop-release-2.7.3-RC2.zip
3. 进入目录 cd hadoop-release-2.7.3-RC2/
4. 设置 maven 内存，export MAVEN_OPTS="-Xms256m -Xmx512m"
5. 编译 mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy
6. 时间会很长，一般报错都是缺少依赖
7. 结果如下

# ls hadoop-release-2.7.3-RC2/hadoop-dist/target/hadoop-2.7.3/lib/native
libhadoop.a         libhadooputils.a  libsnappy.a     libsnappy.so.1.3.0
libhadooppipes.a    libhdfs.a         libsnappy.la
libhadoop.so        libhdfs.so        libsnappy.so
libhadoop.so.1.0.0  libhdfs.so.0.0.0  libsnappy.so.1

hadoop 和hbase 添加 snappy

1. 拷贝库文件到 hadoop

   # cp -r /data/hadoop-compile/hadoop-release-2.7.3-RC2/hadoop-dist/target/hadoop-2.7.3/lib/native/* $HADOOP_HOME/lib/native/

2. 在 hbase 中创建目录

   # mkdir -p $HBASE_HOME/lib/native/Linux-amd64-64

3. 拷贝库文件到 Hbase

   # cp -r /data/hadoop-compile/hadoop-release-2.7.3-RC2/hadoop-dist/target/hadoop-2.7.3/lib/native/* $HBASE_HOME/lib/native/Linux-amd64-64/

4. $HADOOP_/HOME/etc/hadoop/hadoop-env.sh 添加如下环境变量：

   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/hadoop/hadoop-2.7.3/lib/native/:/usr/local/lib/

5. $HBASE_HOME/conf/hbase-env.sh 添加如下环境变量：

   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/hadoop/hadoop-2.7.3/lib/native/:/usr/local/lib/
   export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

修改配置文件

1. 添加 $HADOOP_HOOME/etc/hadoop/core-site.xml

   <property>  
         <name>io.compression.codecs</name>  
         <value>org.apache.hadoop.io.compress.GzipCodec,  
           org.apache.hadoop.io.compress.DefaultCodec,  
           org.apache.hadoop.io.compress.BZip2Codec,  
           org.apache.hadoop.io.compress.SnappyCodec  
         </value>  
      </property>

2. 添加 $HADOOP_HOOME/etc/hadoop/mapred-site.xml

   <property>    
       <name>mapred.output.compress</name>    
       <value>true</value>    
   </property>    
   <property>    
       <name>mapred.output.compression.codec</name>    
       <value>org.apache.hadoop.io.compress.SnappyCodec</value>    
   </property>    
   <property>    
       <name>mapred.compress.map.output</name>    
       <value>true</value>    
   </property>    
   <property>    
       <name>mapred.map.output.compression.codec</name>    
       <value>org.apache.hadoop.io.compress.SnappyCodec</value>    
   </property>

3. 添加 $HBASE_HOME/conf/hbase-site.xml

   <property>  
            <name>hbase.regionserver.codecs</name>  
            <value>snappy</value>  
    </property>

重启 hadoop、hbase，并测试

hadoop测试： $HADOOP_HOME/bin/hadoop checknative -a

# $HADOOP_HOME/bin/hadoop checknative -a
17/09/21 10:31:50 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/local/hadoop/hadoop-2.7.3/lib/native/libhadoop.so
zlib:    true /lib64/libz.so.1
snappy:  true /usr/local/hadoop/hadoop-2.7.3/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   false 
openssl: true /lib64/libcrypto.so
17/09/21 10:31:50 INFO util.ExitUtil: Exiting with status 1

hbase 创建表：

create 'snappytest', { NAME => 'info', COMPRESSION => 'snappy'}