Hadoop HBase Hive伪分布式环境搭建
Hadoop HBase Hive
启动：
$HADOOP_HOME/bin/start-all.sh
$HBASE_HOME/bin/start-hbase.sh
$HIVE_HOME/bin/hive start
环境配置
1、JDK安装
2、SSH配置
3、环境变量
/etc/profile
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_65
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib/dt.jar:$JAVA_HOME/lib/tools.jar:${JRE_HOME}/lib/rt.jar
export HADOOP_HOME=/usr/local/hadoop
export CLASSPATH=.:$CLASSPATH:$HADOOP_HOME/lib
export HBASE_HOME=/usr/local/hbase
export HIVE_HOME=/usr/local/hive
export CLASSPATH=$CLASSPATH:$HIVE_HOME/lib
export PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin:${HADOOP_HOME}/bin:${HBASE_HOME}/bin:$HIVE_HOME/bin:$PATH
Hadoop
Hadoop版本：1.1.2
目的
This document describes how to set up and configure a single-node Hadoop installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).
Prerequisites
Required Software
Required software for Linux and Windows include:
1. JavaTM 1.6.x, preferably from Sun, must be installed.
2. ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.
Additional requirements for Windows include:
1. Cygwin - Required for shell support in addition to the required software above.
Installing Software
If your cluster doesn't have the requisite software you will need to install it.
For example on Ubuntu Linux:
$ sudo apt-get install ssh
$ sudo apt-get install rsync
On Windows, if you did not install the required software when you installed cygwin, start the cygwin installer and select the packages:
· openssh - the Net category
下载
To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.
Prepare to Start the Hadoop Cluster
Unpack the downloaded Hadoop distribution. In the distribution, edit the file conf/hadoop-env.sh to define at least JAVA_HOME to be the root of your Java installation.
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_65
Try the following command:
$ bin/hadoop
This will display the usage documentation for the hadoop script.
伪分布式Pseudo-Distributed Operation
Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.
配置Configuration
Use the following:
hostname myhadoop
vi /etc/hostname
myhadoop
vi /etc/hosts
ip myhadoop
conf/core-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://myhadoop:9000</value>
</property>
</configuration>
conf/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/local/hadoop/hadoopdata/dfsname</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/hadoop/hadoopdata/dfsdata</value></property>
</configuration>
conf/mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>myhadoop:9001</value>
</property>
</configuration>
Setup passphraseless ssh
Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost
If you cannot ssh to localhost without a passphrase, execute the following commands:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
执行Execution
Format a new distributed-filesystem:
$ bin/hadoop namenode -format
Start the hadoop daemons:
$ bin/start-all.sh
The hadoop daemon log output is written to the ${HADOOP_LOG_DIR} directory (defaults to ${HADOOP_HOME}/logs).
Browse the web interface for the NameNode and the JobTracker; by default they are available at:
<!--[if !supportLists]-->· <!--[endif]-->NameNode - http://localhost:50070/
<!--[if !supportLists]-->· <!--[endif]-->JobTracker - http://localhost:50030/
Copy the input files into the distributed filesystem:
$ bin/hadoop fs -put conf input
Run some of the examples provided:
$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
Examine the output files:
Copy the output files from the distributed filesystem to the local filesytem and examine them:
$ bin/hadoop fs -get output output
$ cat output/*
or
View the output files on the distributed filesystem:
$ bin/hadoop fs -cat output/*
When you're done, stop the daemons with:
$ bin/stop-all.sh
HBase
HBase版本：0.94.7
hbase-site.xml
<property>
<name>hbase.rootdir</name>
<value>hdfs://myhadoop:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>myhadoop</value>
</property>
bin/start-hbase.sh
Hive
一、hive配置
cp hive-default.xml.template hive-site.xml
cp hive-log4j.properties.template hive-log4j.properties
cp hive-env.sh.template hive-env.sh
二、修改hive-env.sh
配置HADOOP_HOME路径。
三、修改hive-site.xml配置文件，把Hive的元数据存储到MySQL中
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://myhadoop:3306/hive_metadata?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/local/hive/lib/hive-hbase-handler-0.9.0.jar,file:///usr/local/hive/lib/hbase-0.94.7-security.jar,file:///usr/local/hive/lib/protobuf-java-2.4.0a.jar,file:///usr/local/hive/lib/zookeeper-3.4.5.jar</value>
</property>
删除/usr/local/hive/lib下的hbase-0.92.0.jar，hbase-0.92.0-tests.jar，zookeeper-3.4.3.jar
从hbase拷贝hbase-0.94.7-security.jar、zookeeper-3.4.5.jar和protobuf-java-2.4.0a.jar到hive/lib下。
三、修改hive-log4j.properties
#log4j.appender.EventCounter=org.apache.hadoop.metrics.jvm.EventCounter
log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
四、在hdfs上面，创建目录
$HADOOP_HOME/bin/hadoop fs -mkdrr /tmp
$HADOOP_HOME/bin/hadoop fs –mkdir /user/hive/warehouse
$HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp
$HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse
五、手动上传mysql的jdbc库到hive/lib
~ ls /home/cos/toolkit/hive-0.9.0/lib
mysql-connector-java-5.1.22-bin.jar
六、启动hive
方式1：
bin/hive start
方式2：
#启动metastore服务
~ bin/hive --service metastore &
Starting Hive Metastore Server
#启动hiveserver服务
~ bin/hive --service hiveserver &
Starting Hive Thrift Server
#启动hive客户端
~ bin/hive shell
Logging initialized using configuration in file:/root/hive-0.9.0/conf/hive-log4j.properties
Hive history file=/tmp/root/hive_job_log_root_201211141845_1864939641.txt
hive> show tables
OK
Hive函数、复杂类型访问操作
hive提供了复合数据类型：
Structs： structs内部的数据可以通过DOT（.）来存取，例如，表中一列c的类型为STRUCT{a INT; b INT}，我们可以通过c.a来访问域a
Maps（K-V对）：访问指定域可以通过["指定域名称"]进行，例如，一个Map M包含了一个group-》gid的kv对，gid的值可以通过M['group']来获取
Arrays：array中的数据为相同类型，例如，假如array A中元素['a','b','c']，则A[1]的值为'b'
Array：
建表：
create table class_test(name string, student_id_list array<INT>) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':';
导数据：
vim test.txt
034,1:2:3:4
035,5:6
036,7:8:9:10
LOAD DATA LOCAL INPATH '/opt/test.txt' INTO TABLE class_test ;
查询：
select student_id_list[3] from class_test;
Map：
建表：
create table employee(id string, perf map<string, int>) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' COLLECTION ITEMS TERMINATED BY ',' MAP KEYS TERMINATED BY ':';
导数据：
vim test2.txt
1 job:80,team:60,person:70
2 job:60,team:80
3 job:90,team:70,person:100
LOAD DATA LOCAL INPATH '/opt/test2.txt' INTO TABLE employee;
查询：
select perf['person'] from employee;
select perf['person'] from employee where perf['person'] is not null;
Struct使用
建表：
create table student_test(id INT, info struct<name:STRING, age:INT>) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':';
'FIELDS TERMINATED BY' ：字段与字段之间的分隔符
'COLLECTION ITEMS TERMINATED BY' ：一个字段各个item的分隔符
导入数据：
cat test3.txt
1,zhou:30
2,yan:30
3,chen:20
4,li:80
LOAD DATA LOCAL INPATH '/opt/test3.txt' INTO TABLE student_test;
查询：
select info.age from student_test;
查询每年9.30-10.07十一期间的身份证号：
select substr(rzsj,1,4) as year, sfzmhm from jnlk where substr(rzsj,6,5)>='09-30' and substr(rzsj,6,5)<='10-07' and rzsj is not null order by year,sfzmhm;
练习
Hbase创建表：
create 'blog','article','author'
插入hbase数据：
put 'blog','1','article:title','Head First HBase'
put 'blog','1','article:content','HBase is the Hadoop database. Use it when you need random, realtime read/write access to your Big Data.'
put 'blog','1','article:tags','Hadoop,HBase,NoSQL'
put 'blog','1','author:name','hujinjun'
put 'blog','1','author:nickname','一叶渡江'
put 'blog','10','article:tags','Hadoop'
put 'blog','10','author:nickname','heyun'
put 'blog','100','article:tags','hbase,nosql'
put 'blog','100','author:nickname','shenxiu'
hive:
CREATE EXTERNAL TABLE blog(key int,title string,content string,tags string,name string,nickname string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "article:title,article:content,article:tags,author:name,author:nickname") TBLPROPERTIES("hbase.table.name" = "blog");
hive> create table wyp (id int, name string, age int, tel string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE;
vim /opt/wyp.txt
1 wyp 25 13188888888888
2 test 30 13888888888888
3 zs 34 899314121
hive> load data local inpath '/opt/wyp.txt' into table wyp;
vim /opt/add.txt
5 wyp1 23 131212121212
6 wyp2 24 134535353535
7 wyp3 25 132453535353
8 wyp4 26 154243434355
$HADOOP_HOME/bin/hadoop fs -mkdir /wyp
$HADOOP_HOME/bin/hadoop fs -copyFromLocal /opt/add.txt /wyp/add.txt
hive> load data inpath '/wyp/add.txt' into table wyp;
hive> select * from wyp;
FAQ
hive hwi 启动错误
错误日志：
INFO hwi.HWIServer: HWI is starting up
WARN conf.HiveConf: DEPRECATED: Ignoring hive-default.xml found on the CLASSPATH at /usr/local/hive/conf/hive-default.xml
FATAL hwi.HWIServer: HWI WAR file not found at /usr/local/hive/usr/local/hive/lib/hive-hwi-0.9.0.war
解决方法：
这样的错误解决办法很简单，hive-site.xml中添加：
<property>
<name>hive.hwi.war.file</name>
<value>lib/hive-hwi-0.9.0.war</value>
<description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </description>
</property>
否则路径错误！