记录一下window idea 如何直连sparksql 使用hive数据元数据
换了几个开发环境,每次都会遇到sparksql连不上元数据的情况,整理一下脱坑吧。。。。。
进入主题:
首先说一下几个文件吧,这些是我遇到的几个问题的解决方法,有可能你并不适用,仅供参考。
1、配置文件放在resources下面
2、下载hadoop-common-2.2.0-bin-master 这个文件,里面有hadoop必要的bin文件,以此当做hadoop的家目录
配置如下:
3.启动的时候,你可能还会遇到mysql数据元字符的问题:
Specified key was too long; max key length is 767 bytes
我就直接贴了,把hive的元数据库改成latin1字符集,记住库和里面所有表
alter database hive character set latin1; alter table BUCKETING_COLS convert to character set latin1; alter table CDS convert to character set latin1; alter table COLUMNS_V2 convert to character set latin1; alter table DATABASE_PARAMS convert to character set latin1; alter table DBS convert to character set latin1; alter table FUNC_RU convert to character set latin1; alter table FUNCS convert to character set latin1; alter table GLOBAL_PRIVS convert to character set latin1; alter table PART_COL_STATS convert to character set latin1; alter table PARTITION_KEY_VALS convert to character set latin1; alter table PARTITIONS convert to character set latin1; alter table ROLES convert to character set latin1; alter table SDS convert to character set latin1; alter table SEQUENCE_TABLE convert to character set latin1; alter table SERDES convert to character set latin1; alter table SKEWED_STRING_LIST convert to character set latin1; alter table SKEWED_STRING_LIST_VALUES convert to character set latin1; alter table SORT_COLS convert to character set latin1; alter table TAB_COL_STATS convert to character set latin1; alter table TBLS convert to character set latin1; alter table VERSION convert to character set latin1;
4.测试
object test { def main(args: Array[String]): Unit = { val conf = new SparkConf() conf.setAppName(s"TestHive") conf.setMaster("local[4]") val spark = SparkSession.builder.config(conf).enableHiveSupport().getOrCreate() spark.sql("show databases").show } }
相关推荐
zlsdmx 2020-07-05
tomson 2020-07-05
archive 2020-07-30
成长之路 2020-07-28
eternityzzy 2020-07-19
taisenki 2020-07-05
tugangkai 2020-07-05
SignalDu 2020-07-05
tugangkai 2020-07-04
tomson 2020-07-05
Zhangdragonfly 2020-06-28
genshengxiao 2020-06-26
成长之路 2020-06-26
tomson 2020-06-26
蜗牛之窝 2020-06-26
成长之路 2020-06-25