记录一下window idea 如何直连sparksql 使用hive数据元数据

换了几个开发环境,每次都会遇到sparksql连不上元数据的情况,整理一下脱坑吧。。。。。

进入主题:

首先说一下几个文件吧,这些是我遇到的几个问题的解决方法,有可能你并不适用,仅供参考。

1、配置文件放在resources下面

记录一下window idea 如何直连sparksql 使用hive数据元数据

2、下载hadoop-common-2.2.0-bin-master 这个文件,里面有hadoop必要的bin文件,以此当做hadoop的家目录

配置如下:

记录一下window idea 如何直连sparksql 使用hive数据元数据

3.启动的时候,你可能还会遇到mysql数据元字符的问题:

  Specified key was too long; max key length is 767 bytes

我就直接贴了,把hive的元数据库改成latin1字符集,记住库和里面所有表

alter database hive character set latin1;  

 alter table BUCKETING_COLS                 convert to character set latin1;
 alter table CDS                            convert to character set latin1;
 alter table COLUMNS_V2                     convert to character set latin1;
 alter table DATABASE_PARAMS                convert to character set latin1;
 alter table DBS                            convert to character set latin1;
 alter table FUNC_RU                        convert to character set latin1;
 alter table FUNCS                          convert to character set latin1;
 alter table GLOBAL_PRIVS                   convert to character set latin1;
 alter table PART_COL_STATS                 convert to character set latin1;
 alter table PARTITION_KEY_VALS             convert to character set latin1;
 alter table PARTITIONS                     convert to character set latin1;
 alter table ROLES                          convert to character set latin1;
 alter table SDS                            convert to character set latin1;
 alter table SEQUENCE_TABLE                 convert to character set latin1;
 alter table SERDES                         convert to character set latin1;
 alter table SKEWED_STRING_LIST             convert to character set latin1;
 alter table SKEWED_STRING_LIST_VALUES      convert to character set latin1;
 alter table SORT_COLS                      convert to character set latin1;
 alter table TAB_COL_STATS                  convert to character set latin1;
 alter table TBLS                           convert to character set latin1;
 alter table VERSION                        convert to character set latin1;

 4.测试

object test {
    def main(args: Array[String]): Unit = {

      val conf = new SparkConf()
      conf.setAppName(s"TestHive")
      conf.setMaster("local[4]")
      val spark = SparkSession.builder.config(conf).enableHiveSupport().getOrCreate()

      spark.sql("show databases").show
    }

}

记录一下window idea 如何直连sparksql 使用hive数据元数据

相关推荐