HIVE报错记录

执行sql时报错信息

Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)

MR任务报错

Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

解决方法：

[hadoop@n4 hadoop]$ hadoop classpath
/home/hadoop/hadoop-3.2.3/etc/hadoop:/home/hadoop/hadoop-3.2.3/share/hadoop/common/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/common/*:/home/hadoop/hadoop-3.2.3/share/hadoop/hdfs:/home/hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/hdfs/*:/home/hadoop/hadoop-3.2.3/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/mapreduce/*:/home/hadoop/hadoop-3.2.3/share/hadoop/yarn:/home/hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/yarn/*

将获取的classpath配置到yarn-site.xml中并重启yarn和hive服务

<property>
    <name>yarn.application.classpath</name>
    <value>
/home/hadoop/hadoop-3.2.3/etc/hadoop:/home/hadoop/hadoop-3.2.3/share/hadoop/common/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/common/*:/home/hadoop/hadoop-3.2.3/share/hadoop/hdfs:/home/hadoop/hadoop-3.2.3/share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/hdfs/*:/home/hadoop/hadoop-3.2.3/share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/mapreduce/*:/home/hadoop/hadoop-3.2.3/share/hadoop/yarn:/home/hadoop/hadoop-3.2.3/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-3.2.3/share/hadoop/yarn/*
    </value>
  </property>

==========================================================================

堆内存溢出报错

环境信息为测试环境

2022-07-21 11:09:48,808 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space

解决方法：

通过报错看到是在map阶段OOM，所以调整map阶段内存参数

查看hive map任务堆内存资源设置

堆内存为512m,这里调大该参数，设置为800m,再次尝试

# map 任务内存总大小
SET mapreduce.map.memory.mb=1024;
# map 堆大小
SET mapreduce.map.java.opts=-Xms800m -Xmx800m;
# reduce 任务内存总大小
SET mapreduce.reduce.memory.mb=1024;
# reduce 堆大小
SET mapreduce.reduce.java.opts=-Xmx1024m;

成功跑出数据

==========================================================================

TEZ引擎报错：

java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

报错原因：

该类来自于guava.jar

hive和hadoop中该jar包版本不一致导致

解决方法：

把hadoop中该jar包复制到hive的lib中

hadoop位置：

$HADOOP_HOME/shard/hadoop/hdfs/lib/

报错信息：

Column length too big for column 'PARAM_VALUE' (max = 21845);

原因是编码问题，重新把hive与mysql的关联的数据库编码改成latin1就行了

alter database hive character set latin1;