mapreduce报错:java.io.IOException: Split metadata size exceeded 10000000.
Failure Info:Job initialization failed: java.io.IOException: Split metadata size exceeded 10000000. Aborting job job_201205162059_1073852 at org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:48) at org.apache.hadoop.mapred.JobInProgress.createSplits(JobInProgress.java:817) at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:711) at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4028) at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
出错原因:该job的job.splitmetainfo文件大小超过限制;
1. job.splitmetainfo,该文件记录split的元数据信息,job split ----> HDFS block && slave node
存放路径位于:${hadoop.tmp.dir}/mapred/staging/${user.name}/.staging/jobId/
2. 参数mapreduce.jobtracker.split.metainfo.maxsize控制该文件的最大大小,默认为:10000000(10M)
解决方法:
1. 修改mapreduce.jobtracker.split.metainfo.maxsize参数值,但需要重启jobtracker才能生效;
2. 该问题的根本原因还是因为input小文件或者目录太多导致的,所以建议合并小文件;
相关推荐
zhangll00 2020-05-07
银角大王 2020-06-25
Hhanwen 2020-05-12
sui 2020-04-29
GeorgeTH 2020-03-02
Aggressivesnail 2020-02-28
laohyx 2020-01-30
清风徐来水波不兴 2020-01-05
sdwylry 2020-01-05
LuoXinLoves 2019-12-24
budding0 2019-12-17
iphilo 2019-12-16
jiangtie 2019-12-09
嗡汤圆 2019-11-10
LiHansiyuan 2019-11-03
程松 2019-09-05
上方科技技术交流 2016-07-11