[原]关于hive升级到0.11的版本问题2

标签: | 发表时间:2013-08-14 18:27 | 作者:chenyi8888
出处:http://blog.csdn.net/chenyi8888

通过网友的帮助,打印加载路径地址,发现果然有个类在其他地方里存在,如下:

[hadoop@master ~]$ hive
file:/home/hadoop/hadoop-1.0.3/conf/
file:/home/hadoop/jrockit-jdk1.6.0_29/lib/tools.jar
file:/home/hadoop/hadoop-1.0.3/
file:/home/hadoop/hadoop-1.0.3/hadoop-core-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/asm-3.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/aspectjrt-1.6.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/aspectjtools-1.6.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-beanutils-1.7.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-beanutils-core-1.8.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-cli-1.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-codec-1.4.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-collections-3.2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-configuration-1.6.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-daemon-1.0.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-digester-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-el-1.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-httpclient-3.0.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-io-2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-lang-2.4.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-logging-1.1.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-logging-api-1.0.4.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-math-2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-net-1.4.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/core-3.1.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/hadoop-capacity-scheduler-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/hadoop-fairscheduler-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/hadoop-thriftfs-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/hbase-0.94.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/hive-shims-0.9.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/hsqldb-1.8.0.10.jar
file:/home/hadoop/hadoop-1.0.3/lib/jackson-core-asl-1.8.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jackson-mapper-asl-1.8.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jasper-compiler-5.5.12.jar
file:/home/hadoop/hadoop-1.0.3/lib/jasper-runtime-5.5.12.jar
file:/home/hadoop/hadoop-1.0.3/lib/jdeb-0.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jersey-core-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jersey-json-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jersey-server-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jets3t-0.6.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/jetty-6.1.26.jar
file:/home/hadoop/hadoop-1.0.3/lib/jetty-util-6.1.26.jar
file:/home/hadoop/hadoop-1.0.3/lib/jsch-0.1.42.jar
file:/home/hadoop/hadoop-1.0.3/lib/junit-4.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/kfs-0.2.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/log4j-1.2.15.jar
file:/home/hadoop/hadoop-1.0.3/lib/mockito-all-1.8.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/oro-2.0.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/protobuf-java-2.4.0a.jar
file:/home/hadoop/hadoop-1.0.3/lib/servlet-api-2.5-20081211.jar
file:/home/hadoop/hadoop-1.0.3/lib/slf4j-api-1.4.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/slf4j-log4j12-1.4.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/xmlenc-0.52.jar
file:/home/hadoop/hadoop-1.0.3/lib/zookeeper-3.3.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/jsp-2.1/jsp-2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/jsp-2.1/jsp-api-2.1.jar
file:/home/hadoop/
file:/home/hadoop/hive-0.11.0/conf/
file:/home/hadoop/hive-0.11.0/lib/antlr-runtime-3.4.jar
file:/home/hadoop/hive-0.11.0/lib/avro-1.7.1.jar
file:/home/hadoop/hive-0.11.0/lib/avro-mapred-1.7.1.jar
file:/home/hadoop/hive-0.11.0/lib/commons-cli-1.2.jar
file:/home/hadoop/hive-0.11.0/lib/commons-codec-1.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-collections-3.2.1.jar
file:/home/hadoop/hive-0.11.0/lib/commons-compress-1.4.1.jar
file:/home/hadoop/hive-0.11.0/lib/commons-configuration-1.6.jar
file:/home/hadoop/hive-0.11.0/lib/commons-dbcp-1.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-io-2.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-lang-2.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-logging-1.0.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-logging-api-1.0.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-pool-1.5.4.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-connectionpool-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-core-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-enhancer-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-rdbms-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/derby-10.4.2.0.jar
file:/home/hadoop/hive-0.11.0/lib/guava-11.0.2.jar
file:/home/hadoop/hive-0.11.0/lib/hbase-0.94.6.1.jar
file:/home/hadoop/hive-0.11.0/lib/hbase-0.94.6.1-tests.jar
file:/home/hadoop/hive-0.11.0/lib/hive-beeline-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-cli-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-common-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-contrib-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib[hadoop@master ~]$ hive
file:/home/hadoop/hadoop-1.0.3/conf/
file:/home/hadoop/jrockit-jdk1.6.0_29/lib/tools.jar
file:/home/hadoop/hadoop-1.0.3/
file:/home/hadoop/hadoop-1.0.3/hadoop-core-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/asm-3.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/aspectjrt-1.6.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/aspectjtools-1.6.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-beanutils-1.7.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-beanutils-core-1.8.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-cli-1.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-codec-1.4.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-collections-3.2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-configuration-1.6.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-daemon-1.0.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-digester-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-el-1.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-httpclient-3.0.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-io-2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-lang-2.4.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-logging-1.1.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-logging-api-1.0.4.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-math-2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/commons-net-1.4.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/core-3.1.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/hadoop-capacity-scheduler-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/hadoop-fairscheduler-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/hadoop-thriftfs-1.0.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/hbase-0.94.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/hive-shims-0.9.0.jar
file:/home/hadoop/hadoop-1.0.3/lib/hsqldb-1.8.0.10.jar
file:/home/hadoop/hadoop-1.0.3/lib/jackson-core-asl-1.8.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jackson-mapper-asl-1.8.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jasper-compiler-5.5.12.jar
file:/home/hadoop/hadoop-1.0.3/lib/jasper-runtime-5.5.12.jar
file:/home/hadoop/hadoop-1.0.3/lib/jdeb-0.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jersey-core-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jersey-json-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jersey-server-1.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/jets3t-0.6.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/jetty-6.1.26.jar
file:/home/hadoop/hadoop-1.0.3/lib/jetty-util-6.1.26.jar
file:/home/hadoop/hadoop-1.0.3/lib/jsch-0.1.42.jar
file:/home/hadoop/hadoop-1.0.3/lib/junit-4.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/kfs-0.2.2.jar
file:/home/hadoop/hadoop-1.0.3/lib/log4j-1.2.15.jar
file:/home/hadoop/hadoop-1.0.3/lib/mockito-all-1.8.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/oro-2.0.8.jar
file:/home/hadoop/hadoop-1.0.3/lib/protobuf-java-2.4.0a.jar
file:/home/hadoop/hadoop-1.0.3/lib/servlet-api-2.5-20081211.jar
file:/home/hadoop/hadoop-1.0.3/lib/slf4j-api-1.4.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/slf4j-log4j12-1.4.3.jar
file:/home/hadoop/hadoop-1.0.3/lib/xmlenc-0.52.jar
file:/home/hadoop/hadoop-1.0.3/lib/zookeeper-3.3.5.jar
file:/home/hadoop/hadoop-1.0.3/lib/jsp-2.1/jsp-2.1.jar
file:/home/hadoop/hadoop-1.0.3/lib/jsp-2.1/jsp-api-2.1.jar
file:/home/hadoop/
file:/home/hadoop/hive-0.11.0/conf/
file:/home/hadoop/hive-0.11.0/lib/antlr-runtime-3.4.jar
file:/home/hadoop/hive-0.11.0/lib/avro-1.7.1.jar
file:/home/hadoop/hive-0.11.0/lib/avro-mapred-1.7.1.jar
file:/home/hadoop/hive-0.11.0/lib/commons-cli-1.2.jar
file:/home/hadoop/hive-0.11.0/lib/commons-codec-1.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-collections-3.2.1.jar
file:/home/hadoop/hive-0.11.0/lib/commons-compress-1.4.1.jar
file:/home/hadoop/hive-0.11.0/lib/commons-configuration-1.6.jar
file:/home/hadoop/hive-0.11.0/lib/commons-dbcp-1.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-io-2.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-lang-2.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-logging-1.0.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-logging-api-1.0.4.jar
file:/home/hadoop/hive-0.11.0/lib/commons-pool-1.5.4.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-connectionpool-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-core-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-enhancer-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/datanucleus-rdbms-2.0.3.jar
file:/home/hadoop/hive-0.11.0/lib/derby-10.4.2.0.jar
file:/home/hadoop/hive-0.11.0/lib/guava-11.0.2.jar
file:/home/hadoop/hive-0.11.0/lib/hbase-0.94.6.1.jar
file:/home/hadoop/hive-0.11.0/lib/hbase-0.94.6.1-tests.jar
file:/home/hadoop/hive-0.11.0/lib/hive-beeline-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-cli-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-common-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-contrib-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-exec-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-hbase-handler-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-hwi-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-jdbc-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-metastore-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-serde-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-service-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/hive-shims-0.11.0.jar
file:/home/hadoop/hive-0.11.0/lib/jackson-core-asl-1.8.8.jar
file:/home/hadoop/hive-0.11.0/lib/jackson-jaxrs-1.8.8.jar
file:/home/hadoop/hive-0.11.0/lib/jackson-mapper-asl-1.8.8.jar
file:/home/hadoop/hive-0.11.0/lib/jackson-xc-1.8.8.jar
file:/home/hadoop/hive-0.11.0/lib/JavaEWAH-0.3.2.jar
file:/home/hadoop/hive-0.11.0/lib/javolution-5.5.1.jar
file:/home/hadoop/hive-0.11.0/lib/jdo2-api-2.3-ec.jar
file:/home/hadoop/hive-0.11.0/lib/jetty-6.1.26.jar
file:/home/hadoop/hive-0.11.0/lib/jetty-util-6.1.26.jar
file:/home/hadoop/hive-0.11.0/lib/jline-0.9.94.jar
file:/home/hadoop/hive-0.11.0/lib/json-20090211.jar
file:/home/hadoop/hive-0.11.0/lib/libfb303-0.9.0.jar
file:/home/hadoop/hive-0.11.0/lib/libthrift-0.9.0.jar
file:/home/hadoop/hive-0.11.0/lib/log4j-1.2.16.jar
file:/home/hadoop/hive-0.11.0/lib/maven-ant-tasks-2.1.3.jar
file:/home/hadoop/hive-0.11.0/lib/metrics-core-2.1.2.jar
file:/home/hadoop/hive-0.11.0/lib/protobuf-java-2.4.1.jar
file:/home/hadoop/hive-0.11.0/lib/servlet-api-2.5-20081211.jar
file:/home/hadoop/hive-0.11.0/lib/slf4j-api-1.6.1.jar
file:/home/hadoop/hive-0.11.0/lib/slf4j-log4j12-1.6.1.jar
file:/home/hadoop/hive-0.11.0/lib/snappy-0.2.jar
file:/home/hadoop/hive-0.11.0/lib/ST4-4.0.4.jar
file:/home/hadoop/hive-0.11.0/lib/tempus-fugit-1.1.jar
file:/home/hadoop/hive-0.11.0/lib/xz-1.0.jar
file:/home/hadoop/hive-0.11.0/lib/zookeeper-3.4.3.jar
file:/home/hadoop/
file:/home/hadoop/hadoop-1.0.3/lib/hadoop-lzo-0.4.15.jar

注意红色字体部分,说明加载的顺序导致,不知道谁之前把hive-shims-0.9.0.jar拷贝到hadoop的lib目录下去了,去掉此jar,问题得到解决。

终于可以不在0.9的版本上做修改了,坑爹啊!!!!

附上打印类加载路径代码:

URL[] URLs=((URLClassLoader)CliDriver.class.getClassLoader()).getURLs();

for(URL url:URLs)

      System.out.println(url);


 

作者:chenyi8888 发表于2013-8-14 18:27:35 原文链接
阅读:11 评论:0 查看评论

相关 [hive 升级 版本] 推荐:

[原]关于hive升级到0.11的版本问题2

- - 分布式应用与服务器架构专栏
通过网友的帮助,打印加载路径地址,发现果然有个类在其他地方里存在,如下:. 注意红色字体部分,说明加载的顺序导致,不知道谁之前把hive-shims-0.9.0.jar拷贝到hadoop的lib目录下去了,去掉此jar,问题得到解决. 终于可以不在0.9的版本上做修改了,坑爹啊. 作者:chenyi8888 发表于2013-8-14 18:27:35 原文链接.

Chromium 升级到 V17 版本

- 龍渊冭子 - cnBeta.COM
尽管Chrome和Chromium的版本号没有任何意义,但我们还是不得不提一句,Chromium已经升级到了V17版本. 另外,Chrome Beta分支for Windows/Mac OS X/Linux和Chrome Frame则升级到了15.0.874.100,主要新特性:.

hive调优

- - 互联网 - ITeye博客
一、    控制hive任务中的map数: . 1.    通常情况下,作业会通过input的目录产生一个或者多个map任务. 主要的决定因素有: input的文件总个数,input的文件大小,集群设置的文件块大小(目前为128M, 可在hive中通过set dfs.block.size;命令查看到,该参数不能自定义修改);.

hive 优化 tips

- - CSDN博客推荐文章
一、     Hive join优化. 也可以显示声明进行map join:特别适用于小表join大表的时候,SELECT /*+ MAPJOIN(b) */ a.key, a.value FROM a join b on a.key = b.key. 2.     注意带表分区的join, 如:.

Hive中的join

- - CSDN博客云计算推荐文章
select a.* from a join b on a.id = b.id select a.* from a join b on (a.id = b.id and a.department = b.department). 在使用join写查询的时候有一个原则:应该将条目少的表或者子查询放在join操作符的左边.

hive优化(2)

- - 开源软件 - ITeye博客
Hive是将符合SQL语法的字符串解析生成可以在Hadoop上执行的MapReduce的工具. 使用Hive尽量按照分布式计算的一些特点来设计sql,和传统关系型数据库有区别,. 所以需要去掉原有关系型数据库下开发的一些固有思维. 1:尽量尽早地过滤数据,减少每个阶段的数据量,对于分区表要加分区,同时只选择需要使用到的字段.

hive优化

- - 开源软件 - ITeye博客
hive.optimize.cp=true:列裁剪. hive.optimize.prunner:分区裁剪. hive.limit.optimize.enable=true:优化LIMIT n语句. hive.limit.optimize.limit.file=10:最大文件数.   1.job的输入数据大小必须小于参数:hive.exec.mode.local.auto.inputbytes.max(默认128MB).

Hive优化

- - 互联网 - ITeye博客
     使用Hive有一段时间了,目前发现需要进行优化的较多出现在出现join、distinct的情况下,而且一般都是reduce过程较慢.      Reduce过程比较慢的现象又可以分为两类:. 情形一:map已经达到100%,而reduce阶段一直是99%,属于数据倾斜. 情形二:使用了count(distinct)或者group by的操作,现象是reduce有进度但是进度缓慢,31%-32%-34%...一个附带的提示是使用reduce个数很可能是1.

hive bucket 桶

- - CSDN博客推荐文章
对于每一个表(table)或者分区,Hive可以进一步组织成桶. Hive也是针对某一列进行桶的组织. Hive采用对列值哈希,然后除以桶的个数求余的方式决定该条记录存放在哪个桶当中. 采用桶能够带来一些好处,比如JOIN操作. 对于JOIN操作两个表有一个相同的列,如果对这两个表都进行了桶操作. 那么将保存相同列值的桶进行JOIN操作就可以,可以大大较少JOIN的数据量.

hive mapjoin使用

- - 淘剑笑的博客
今天遇到一个hive的问题,如下hive sql:. 该语句中B表有30亿行记录,A表只有100行记录,而且B表中数据倾斜特别严重,有一个key上有15亿行记录,在运行过程中特别的慢,而且在reduece的过程中遇有内存不够而报错. 为了解决用户的这个问题,考虑使用mapjoin,mapjoin的原理:.