当前位置： 文江博客话题详情

Hadoop wordcount 无法运行 - 需要解码 hadoop 错误消息的帮助

发布于 2024-11-07 20:31:21 字数 14886 浏览 3 评论 0原文

我需要一些帮助来找出我的工作失败的原因。我建了一个单 节点集群只是为了尝试一下。我按照这里的示例。

一切似乎都工作正常。我格式化了 namenode，并且能够通过 Web 界面连接到 jobtracker、datanode 和 namenode。我能够启动和停止所有 hadoop 服务。

但是，当我尝试运行 wordcount 示例时，我得到以下信息：

Error initializing attempt_201105161023_0002_m_000011_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

我在本地文件系统上创建了目录。

$ sudo mkdir /app/hadoop/tmp 
$ sudo chown hadoop:hadoop /app/hadoop/tmp

我还修改了 core-site.xml：

<!-- In: conf/core-site.xml --> 
<property> 
  <name>hadoop.tmp.dir</name> 
  <value>/app/hadoop/tmp</value> 
  <description>A base for other temporary directories.</description> 
</property>

当我格式化名称节点时，它会在本地和本地节点上创建子目录 和 HDFS 成功。

当我查看失败的 wordcount 输出结果时，出现错误 消息抱怨 I/O 错误，已存档 /app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/ job_201105161023_0002/jobToken

我做了一些故障排除，可以浏览到此 jobToken 文件 本地文件系统，没问题。内容类似于 HDTS MapReduce.job 201105161023_0002

那么这是权限问题吗？我使 hadoop 进程用户能够写入所有子目录，并且它能够创建文件。那么还有什么可能是错误的呢？

命令行的一些更详细的输出：

hadoop@jonathan-VirtualBox:/usr/local/hadoop/hadoop-0.20.203.0$ bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount app/download app/output4
11/05/16 13:38:56 INFO input.FileInputFormat: Total input paths to process : 3
11/05/16 13:39:05 INFO mapred.JobClient: Running job: job_201105161222_0003
11/05/16 13:39:06 INFO mapred.JobClient:  map 0% reduce 0%
11/05/16 13:39:17 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stdout
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stderr
11/05/16 13:39:22 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stdout
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stderr
11/05/16 13:39:23 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stdout
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stderr
11/05/16 13:39:30 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stdout
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stderr
11/05/16 13:39:33 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stdout
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stderr
11/05/16 13:39:36 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stdout
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stderr
11/05/16 13:39:39 INFO mapred.JobClient: Job complete: job_201105161222_0003
11/05/16 13:39:39 INFO mapred.JobClient: Counters: 4
11/05/16 13:39:39 INFO mapred.JobClient:   Job Counters 
11/05/16 13:39:39 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=0
11/05/16 13:39:39 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
原文

I need some help on figuring out why my job failed. I built a single
node cluster just to try it out. I followed the example here.

Everything seems to be working correctly. I formatted the namenode and am able to connect to the jobtracker, datanode, and namenode via the web interface. I am able to start and stop all the hadoop services.

However, when I try to run the wordcount example, I get this:

Error initializing attempt_201105161023_0002_m_000011_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

I created the directory on local file system.

$ sudo mkdir /app/hadoop/tmp 
$ sudo chown hadoop:hadoop /app/hadoop/tmp

I also modified core-site.xml:

<!-- In: conf/core-site.xml --> 
<property> 
  <name>hadoop.tmp.dir</name> 
  <value>/app/hadoop/tmp</value> 
  <description>A base for other temporary directories.</description> 
</property>

When I format the namenode, it creates the subdirectory on both local
and HDFS successfully.

When I look at the result of the failed wordcount output, the error
message complains about an I/O error, on file
/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/
job_201105161023_0002/jobToken

I did some troubleshooting and can browse to this jobToken file on the
local file system, no problem. The content is something like HDTS MapReduce.job 201105161023_0002

So is it a permission issue? I made hadoop process user able to write to all the subdirectories and it was able to create the file. So what else can be wrong?

Some more detailed output from the command line:

hadoop@jonathan-VirtualBox:/usr/local/hadoop/hadoop-0.20.203.0$ bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount app/download app/output4
11/05/16 13:38:56 INFO input.FileInputFormat: Total input paths to process : 3
11/05/16 13:39:05 INFO mapred.JobClient: Running job: job_201105161222_0003
11/05/16 13:39:06 INFO mapred.JobClient:  map 0% reduce 0%
11/05/16 13:39:17 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stdout
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stderr
11/05/16 13:39:22 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stdout
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stderr
11/05/16 13:39:23 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stdout
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stderr
11/05/16 13:39:30 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stdout
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stderr
11/05/16 13:39:33 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stdout
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stderr
11/05/16 13:39:36 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stdout
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stderr
11/05/16 13:39:39 INFO mapred.JobClient: Job complete: job_201105161222_0003
11/05/16 13:39:39 INFO mapred.JobClient: Counters: 4
11/05/16 13:39:39 INFO mapred.JobClient:   Job Counters 
11/05/16 13:39:39 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=0
11/05/16 13:39:39 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0

如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论， 你可以免费 注册 一个本站的账号。

评论（2）

桜花祭 2024-11-14 20:31:21

检查您的 /etc/hosts 配置。
您必须提及主域名，从属域名也是如此。

例如：
127.0.0.1 localhost.localdomain 本地主机
107.108.206.64主ABC-ubuntu
107.108.208.24从XYZ-ubuntu
107.108.87.81slave1 QWE-ubuntu

因为在reduce处理过程中，它搜索的是XYZ-ubuntu主机名而不是slave。

另外，您的主机配置应该全部相同。

Check out your /etc/hosts configuration.
You have to mention master domain name and same for slave.

For eg:
127.0.0.1 localhost.localdomain localhost
107.108.206.64 master ABC-ubuntu
107.108.208.24 slave XYZ-ubuntu
107.108.87.81 slave1 QWE-ubuntu

Because during reduce processing , it search for XYZ-ubuntu host name instead of slave.

Also, your hosts config should be same in all.

回复 原文
多孤肩上扛 2024-11-14 20:31:21

不要创建 /app/hadoop/tmp 文件夹
默认情况下它将在 /tmp/hadoop 下创建...

因此从 conf 文件中删除该属性。
会起作用的

Dont Create /app/hadoop/tmp folder
It will create by default under /tmp/hadoop...

So remove that property from conf file .
It will work

回复 原文
~没有更多了~

关于作者

软糖

暂无简介

0 文章
0 评论
24 人气
发私信

相关话题

更多

热门标签

操作系统 程序设计 IT运维 Linux系统管理 JavaScript 服务器应用 solaris C/C++ PHP Shell BSD Vue.js aix Oracle Python HTML 系统管理 HTML5 CSS 前端
更多

推荐作者

胡图图

文章 0 评论 0

zt006

文章 0 评论 0

z祗昰~

文章 0 评论 0

冰葑

文章 0 评论 0

野の

文章 0 评论 0

天空

文章 0 评论 0

更多

友情链接

文江博客
    我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站，即表示您同意使用 Cookies 和您的相关数据。
    原文