对于复杂类型，Reducer 中的 ReadFields() 中的空缓冲区

发布于 2024-11-05 05:18:54 字数 1845 浏览 2 评论 0原文

我试图在映射器和化简器之间传递一个复杂的可写对象，更具体地说是 ObjectWritables 的 ArrayWritable。

public class ObjectArrayWritable extends ArrayWritable {
    public ObjectArrayWritable() {
        super(ObjectWritable.class);
    }
}

我的映射器：Mapper

我的减速器：Reducer

减速器在 readfields( ) 可写接口的方法并抛出以下异常：

java.lang.NullPointerException
    at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:183)
    at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
    at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)

当我尝试实现自己的自定义可写并对其进行调试时，我注意到readfields(DataInput data) 中 DataInput 的缓冲区收到 null。

请注意，我的错误类似于： hadoop + Writable接口+ readFields在reducer中抛出异常但是物体的清洁没有帮助..

有人可以帮忙吗？

原文

I am trying to pass a complex writable between mapper and reducer, more specifically ArrayWritable of ObjectWritables.

public class ObjectArrayWritable extends ArrayWritable {
    public ObjectArrayWritable() {
        super(ObjectWritable.class);
    }
}

My mapper: Mapper<LongWritable, Text, Text, ObjectArrayWritable>

My reducer: Reducer<Text,ObjectArrayWritable,Text,ObjectArrayWritable>

The reducer crashes on the readfields() method of the writable interface and throws the following exception:

java.lang.NullPointerException
    at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:183)
    at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
    at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
    at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
    at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)

When i tried to implement my own custom writable and debugged it i noticed that the buffer of the DataInput in the readfields(DataInput data) is received null..

Note that my error is similar to:
hadoop + Writable interface + readFields throws an exception in reducer but the cleaning of the object didn't help..

Can anyone assist?

分享到QQ

分享到微博