hadoop streaming 运行时 subprocess failed with code 126
在运行hadoop streaming时,报错,异常内容如下:
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 126
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
hadoop的启动shell脚本如下:
hdfs_input="/test/testLine.log"
hdfs_ouput="/out"
python_lib="${fs_default_name}/python/python.tar.gz"
WORK_DIR=$(cd `dirname $0`; pwd)
${hadoop_path}/bin/hadoop jar ${hadoop_path}/libexec/share/hadoop/tools/lib/hadoop-streaming-2.8.2.jar \
-archives ${python_lib}\#lib \
-D mapreduce.job.name="didp_log" \
-D mapreduce.job.reduces=1 \
-input ${hdfs_input} \
-output ${hdfs_ouput} \
-mapper "lib/python/bin/python mapper.py" \
-reducer "lib/python/bin/python reducer.py" \
-file ${WORK_DIR}/mapper.py \
-file ${WORK_DIR}/reducer.py \
-inputformat org.apache.hadoop.mapred.TextInputFormat \
-outputformat org.apache.hadoop.mapred.TextOutputFormat \
求大神告知如何处理? 多谢
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论