Casbah返回大量对象的问题

Question

Casbah（或 mongodb 的 java 驱动程序）似乎在处理大量返回对象时遇到问题。例如，以下代码段将产生 IllegalArgumentException 并且不会返回单个结果（下面是完整的堆栈跟踪）。但是，如果我将“限制（...）”减少到 1994，一切似乎都工作正常。

for (link <- links; query = link $exists true) {
    val group = new HashMap[String, Set[(String, String)]] with MultiMap[String, (String, String)]
    log.find(query, fieldsToGet.result).limit(1996) foreach {

      x => {
        group.addBinding(x.get(link).toString, (x.get("_id").toString(), x.get("eventType").toString))
      }

    }
    allGroups += link -> group
  }

Apr 26, 2011 8:23:40 PM com.mongodb.DBTCPConnector$MyPort error
SEVERE: MyPort.error called
java.lang.IllegalArgumentException: response too long: 1278031173
    at com.mongodb.Response.<init>(Response.java:40)
    at com.mongodb.DBPort.go(DBPort.java:101)
    at com.mongodb.DBPort.go(DBPort.java:66)
    at com.mongodb.DBPort.call(DBPort.java:56)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
    at com.mongodb.DBCursor._check(DBCursor.java:309)
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72)
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at Sequencer$.<init>(Sequencer.scala:20)
    at Sequencer$.<clinit>(Sequencer.scala)
    at Sequencer.main(Sequencer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Exception in thread "main" java.lang.ExceptionInInitializerError
    at Sequencer.main(Sequencer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Caused by: java.lang.IllegalArgumentException: response too long: 1278031173
    at com.mongodb.Response.<init>(Response.java:40)
    at com.mongodb.DBPort.go(DBPort.java:101)
    at com.mongodb.DBPort.go(DBPort.java:66)
    at com.mongodb.DBPort.call(DBPort.java:56)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266)
    at com.mongodb.DBCursor._check(DBCursor.java:309)
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431)
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456)
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72)
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23)
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at Sequencer$.<init>(Sequencer.scala:20)
    at Sequencer$.<clinit>(Sequencer.scala)
    ... 6 more

似乎异常是由 java 驱动程序中的“Response.java”中的以下检查产生的。

ByteArrayInputStream bin = new ByteArrayInputStream( b );
_len = Bits.readInt( bin );
if ( _len > ( 32 * 1024 * 1024 ) )
 throw new IllegalArgumentException( "response too long: " + _len );

是否是由返回的特定对象引起的？或者这可能是关于casbah？

谢谢， 德里克

Answer 1

看起来 Java 驱动程序正在检查当前响应块是否大于 32 MB，然后抛出异常。

如果在游标上设置batchSize(FEWER_NUMBER_OF_DOCS)，这将减少数据库中的锁定时间并返回小于32 MB 的数据。

我会尝试使用batchSize 来看看什么最适合您的应用程序。

http://api.mongodb.org/scala/casbah/2.1.2/ scaladoc/

Java 驱动程序中的最大值可能会增加。

您的回复的奇怪部分是它说它返回了约 1.19 GB 的数据。

如果您的响应没有那么多数据，则可能表明集合已损坏。