由于我们在应用程序中收集的数据的性质，有必要允许用户能够搜索 1,000,000 多行数据，并尽快在客户端接收结果。
我们尝试过各种技术，目前正在尝试文档存储引擎 MongoDB。我们本质上是按照我们想要在客户端显示的方式将报告数据存储在 Mongo 中。
但是，这就是我想借鉴你的经验的地方。我们使用了一些报告表插件，例如 datatables.net，但是当客户端请求可能匹配 100,000 多个记录的数据时，将该数据传输到客户端的延迟可能需要一分多钟才能将 JSON 从服务器导出到客户。
我尝试通过一次仅向客户端发送 1000 个结果来过滤结果集，但如果用户决定对列进行排序怎么办？整个请求必须再次执行，并将前 1000 个结果加载到客户端。
但是，如果客户端选择在列内搜索，则必须再次重新获取结果，该怎么办？当所有数据都加载到客户端时，所有这些问题都不存在。
所以本质上我的问题是，如何高效、快速地允许客户端搜索和操作大型数据集，但以一种不需要一次性发送每个结果的方式将结果传递到客户端，从而保持客户端尽可能轻量？
我确信 MongoDB 一定有一些前端数据表类型插件，可能通过 node.js 进行直接 JSON 通信，但我找不到任何东西！
感谢您的帮助。
Due to the nature of the data that we collect in our application, its necessary to allow user to be able to search through 1,000,000+ rows of data, and receive results client side as fast as possible.
We have played around with various techniques, and currently experimenting with the document storage engine MongoDB. We are essentially storing the report data in Mongo in the way we want to show it on the client side.
But, this is where i wanted to lean on your experience. We have used some report table plugins such as datatables.net, but when a client requests data which may match 100,000+ records, the latency in delivering that data to the client side, can take well over a minute to export the JSON from server to client.
I have tried filtering the resultset by only sending 1000 result at a time to the client, but what if the user decides to sort a column? The entire request has to be performed again, and the 1st 1000 results loaded into the client.
But what if the client chooses to search within a column, the results have to be re-fetched again. All of these problems dont exist when all of the data has been loaded into the client side.
So essentially my question is, how can you efficiently and speedily, allow clients to search and manipulate large datasets, but have the results delivered to the client side in a manner which doesnt require every result to be sent through in one go, therefore keeping the client side as lightweight as possible?
Im sure there must be some front-end datatable type plugins for MongoDB, potentially via node.js for direct JSON communication, but i cant find anything!
Thanks for your help.
发布评论
评论（2）
听起来您的问题在于您尝试传递客户端的数据量巨大。什么屏幕显示1000行数据？
使用起始索引、排序顺序和页面大小发出请求，并按该顺序仅获取那么多数据并将其返回客户端。
以 jqGrid 为例，它是一个客户端网格，它将分页和排序信息传递回服务器，并且一次只检索一页。 jqGrid 与 MongoDB 以及正确的索引的速度快得令人难以置信。
It sounds like your problem lies with the sheer volume of data you are trying to pass client-side. What screen displays 1,000 rows of data??
Make a request with a start index, a sort order and a page size and fetch only that much data in that order and return it client-side.
Take a look at jqGrid for example as a client-side grid that passes paging and sorting information back to the server and only retrieves one page-worth at a time. jqGrid with MongoDB and the correct indexes in place is blindingly fast.
MongoDB 很棒，但是您可能还希望在 MongoDB 之外对此内容建立索引，以提供您所描述的性能和灵活性。查看 Apache Solr 或 弹性搜索。这两个都是基于Lucene，它具有非常强大的搜索功能。特别是，您可以以非常高效的方式执行分页和排序查询。这两个工具都可以返回 JSON 格式的结果集。有了这么多记录，并使用基于 Lucene 的东西，您还可以非常轻松地提供更高级的过滤功能，例如事实浏览和词干分析。
MongoDB is great, however you may want to also index this content outside of MongoDB to provide the performance and flexibility you describe. Take a look at Apache Solr or Elastic Search. Both of these are based on Lucene, which has very robust searching capabilities. In particular you can execute a query with paging and sorting in a very performant manner. Both of these tools can return the resultset in JSON. With that many records, and using a something based on Lucene you will also be able to provide more advanced filtering capabilities very easily such as facted browsing and stemming.