MapReduce作业在Reduce端的timeout问题

最近在测试对象存储的fs的时候，在跑TeraGen时碰到一个问题，数据量在50GB以下的时候没有什么问题，一旦数据量增大到上百GB后，则会出现问题，在Driver端，也就是APP Master处看到的错误是：

23/03/03 13:46:31 INFO mapreduce.Job: Task Id : attempt_1677815879112_0001_m_000001_0, Status : FAILED
AttemptID:attempt_1677815879112_0001_m_000001_0 task timeout set: 300s, taskTimedOut: true; task stuck timeout set: 600s, taskStuck: false
[2023-03-03 13:46:30.483]Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 2583362 as user root for container container_e12_1677815879112_0001_01_000003, result=success
[2023-03-03 13:46:30.487]Container killed by the ApplicationMaster.
[2023-03-03 13:46:30.501]Container killed on request. Exit code is 143
[2023-03-03 13:46:30.503]Container exited with a non-zero exit code 143. 

23/03/03 13:46:31 INFO mapreduce.Job: Task Id : attempt_1677815879112_0001_m_000000_0, Status : FAILED
AttemptID:attempt_1677815879112_0001_m_000000_0 task timeout set: 300s, taskTimedOut: true; task stuck timeout set: 600s, taskStuck: false
[2023-03-03 13:46:30.483]Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 2732317 as user root for container container_e12_1677815879112_0001_01_000002, result=success
[2023-03-03 13:46:30.487]Container killed by the ApplicationMaster.
[2023-03-03 13:46:30.497]Container killed on request. Exit code is 143
[2023-03-03 13:46:30.498]Container exited with a non-zero exit code 143.

核心是：

task timeout set: 300s, taskTimedOut: true; task stuck timeout set: 600s, taskStuck: false 。

在具体的Node上看到的问题是：

这个问题看起来是因为timeout，翻一下hadoop的代码可以看到：

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java

在这个文件的run方法中：

boolean taskTimedOut = (taskTimeOut > 0) && 
      (currentTime > (entry.getValue().getLastProgress() + taskTimeOut));
   
  if(taskTimedOut) {
    // task is lost, remove from the list and raise lost event
    iterator.remove();
    eventHandler.handle(new TaskAttemptDiagnosticsUpdateEvent(entry
        .getKey(), "AttemptID:" + entry.getKey().toString()
        + " Timed out after " + taskTimeOut / 1000 + " secs"));
    eventHandler.handle(new TaskAttemptEvent(entry.getKey(),
        TaskAttemptEventType.TA_TIMED_OUT));
  }

有对taskTimedOut的判断，taskTimedOut的值来自于：

@Override
protected void serviceInit(Configuration conf) throws Exception {
    super.serviceInit(conf);
    taskTimeOut = conf.getInt(MRJobConfig.TASK_TIMEOUT, 5 * 60 * 1000);
    taskTimeOutCheckInterval =
        conf.getInt(MRJobConfig.TASK_TIMEOUT_CHECK_INTERVAL_MS, 30 * 1000);
}

也就是在mapred-site.xml里面通过mapreduce.task.timeout来配置。

如果要解决timeouT的话，只需要修改mapreduce.task.timeout 和 mapreduce.task.stuck.timeout-ms，更多参数可以看：

https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

对于一个TeraGen的MR的作业，如果使用FileOutputCommitter，默认的逻辑是会生成2个task，因为TeraGen 100GB的时候是会生成2个50GB的文件，所以具体的执行步骤是：

TeraGen100GB->启动2个Node->每一个Node生成50GB文件到tmp目录下->执行rename动作移动数据到真正的目录下

这个环节里面生成50GB数据是在map环节，rename是在reduce环节。

而每一个环节都有一个timeout，也就是app master如果在指定的时间收不到心跳，就会kill这个node，重新拉起一个node。

---

扫码手机观看或分享：