hadoop学习日记四 hive环境搭建

搭建hive环境

从官网http://hive.apache.org/下载hive-0.8.1-bin.tar.gz，下载完成后copy hive-0.8.1-bin.tar.gz到服务器目录下

执行tar -zxvf hive-0.8.1-bin.tar.gz 解压hive。

将hive加入环境变量，主要是方便hive命令的执行，命令如下

exprot HIVE_HOME=/home/hive-0.8.1
exprot PATH=$HIVE_HOME:$PATH

将conf下面的template文件copy一份，便于个性化配置

cp hive-default.xml.template hive-default.xml 
cp hive-default.xml.template hive-site.xml
cp hive-env.sh.template hive-env.sh
cp hive-log4j.properties.template hive-log4j.properties

在hive-env.sh中添加HADOOP_HOME的安装目录地址

在hive-log4j.properties中将log4j.appender.EventCounter的值修改为org.apache.hadoop.log.metrics.EventCounter，这样就不会报WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.的警告了。

以上一切完成后，执行

root@wenbo00:/home/hive-0.8.1-bin/conf# hive

进入hive命令行模式，然后执行

hive> show tables;
OK
Time taken: 6.909 seconds

出现以上结果表示安装成功。

执行建表语句

hive> create table invites(foo INT, bar STRING) partitioned by (ds STRING);
OK
Time taken: 5.918 seconds

查看结果

hive> show tables;
OK
invites
Time taken: 0.246 seconds 

加载数据

hive> LOAD DATA LOCAL INPATH '/home/wenbo/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2012-03-16 17:56:25');
 hive> LOAD DATA LOCAL INPATH '/home/wenbo/kv3.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2012-03-16 17:57:25');

查询数据

hive> select count(*) from invites;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201203160053_0004, Tracking URL = http://wenbo00:50030/jobdetails.jsp?jobid=job_201203160053_0004
Kill Command = /home/hadoop-1.0.1/libexec/../bin/hadoop job  -Dmapred.job.tracker=wenbo00:9001 -kill job_201203160053_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-03-16 03:03:25,739 Stage-1 map = 0%,  reduce = 0%
2012-03-16 03:03:37,819 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:38,825 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:39,837 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:40,852 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:41,870 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:42,879 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:43,885 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:44,898 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:45,907 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:46,914 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:47,926 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:48,933 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.29 sec
2012-03-16 03:03:49,949 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:50,958 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:51,964 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:52,978 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:53,997 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:55,016 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.74 sec
2012-03-16 03:03:56,029 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.74 sec
MapReduce Total cumulative CPU time: 9 seconds 740 msec
Ended Job = job_201203160053_0004
MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   Accumulative CPU: 9.74 sec   HDFS Read: 6342 HDFS Write: 4 SUCESS
Total MapReduce CPU Time Spent: 9 seconds 740 msec
OK
525
Time taken: 42.688 seconds

查询的时候利用mapreduce创建任务执行，可惜我这里的环境是三台虚拟机运行在一台windows7上面，无法发挥集群优势，才导致一个简单的查询就耗费了将近43秒。