Solr索引-主/从复制，如何处理巨大的索引和高流量？

发布于 2024-10-02 09:01:53 字数 8533 浏览 2 评论 0原文

我目前面临 SOLR 的问题（更确切地说是从属复制），在花了相当多的时间在线阅读之后，我发现自己必须寻求一些启发。

- Solr 的索引大小有限制吗？

当处理单个 master 时，什么时候是决定使用多核或多索引的合适时机？是否有任何迹象表明当索引达到一定大小时，建议进行分区？

- 将段从主服务器复制到从服务器时是否有最大大小？

复制时，当从服务器无法下载内容并为其建立索引时，是否存在分段大小限制？当有大量流量需要检索信息以及大量新文档需要复制时，从属服务器无法复制的阈值是多少。

更实际地说，以下是导致我提出这些问题的背景：我们想要索引相当数量的文档，但是当数量达到一千万以上时，从属服务器无法处理它并开始因 SnapPull 错误而导致复制失败。这些文档由一些文本字段组成（名称、类型、描述……大约 10 个其他字段，假设最多 20 个字符）。

我们有 1 个主服务器和 2 个从主服务器复制数据的从服务器。

这是我第一次使用 Solr（我通常使用 spring、hibernate...但不使用 Solr 在 web 应用程序上工作），所以我不知道如何解决这个问题。

目前我们的想法是向主服务器添加多个核心，并从每个核心中创建一个从服务器进行复制。这是正确的方法吗？

如果是，我们如何确定所需的核心数量？现在我们只是尝试看看它的行为方式，并在必要时进行调整，但我想知道是否有针对这个特定主题的最佳实践或一些基准测试。

对于如此数量的平均大小的文档，需要 x 个核心或索引...

感谢您对我如何处理大量平均大小的文档的帮助！

这是从属设备尝试复制时抛出的错误的副本：

ERROR [org.apache.solr.handler.ReplicationHandler] - <SnapPull failed >
org.apache.solr.common.SolrException: Index fetch failed :
        at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
        at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.RuntimeException: java.io.IOException: read past EOF
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
        at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:467)
        at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
        ... 11 more
Caused by: java.io.IOException: read past EOF
        at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
        at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
        at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:70)
        at org.apache.lucene.index.SegmentInfos$2.doBody(SegmentInfos.java:410)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:704)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:538)
        at org.apache.lucene.index.SegmentInfos.readCurrentVersion(SegmentInfos.java:402)
        at org.apache.lucene.index.DirectoryReader.isCurrent(DirectoryReader.java:791)
        at org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:404)
        at org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:352)
        at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:413)
        at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:424)
        at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:35)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1049)
        ... 14 more

编辑：在Mauricio的回答之后，solr库已更新到1.4.1，但仍然出现此错误。我增加了 commitReserveDuration，即使“SnapPull Failed”错误似乎已经消失，又开始出现另一个错误，不确定为什么，因为我似乎在网络上找不到太多答案：

ERROR [org.apache.solr.servlet.SolrDispatchFilter] - <ClientAbortException:  java.io.IOException
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:370)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:323)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
        at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:183)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89)
        at org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:48)
        at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:322)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:837)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:640)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
        at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException
        at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:703)
        at org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:733)
        at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:124)
        at org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(InternalAprOutputBuffer.java:539)
        at org.apache.coyote.Response.doWrite(Response.java:560)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:365)
        ... 22 more
>
ERROR [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/].[SolrServer]] - <Servlet.service() for servlet SolrServer threw exception>
java.lang.IllegalStateException
        at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:405)
        at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:837)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:640)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
        at java.lang.Thread.run(Thread.java:595)

我仍然想知道最佳实践是什么使用 solr 处理包含大量文档的大索引（超过 20G）。我是否在某处遗漏了一些明显的链接？教程、文档？

原文

I'm currently facing an issue with SOLR (more exactly with the slaves replication) and after having spent quite a few time reading online I find myself having to ask for some enlightenment.

- Does Solr have some limitation in size for its index?

When dealing with a single master, when is it the right moment to decide to use multi cores or multi indexes?
Is there any indications on when reaching a certain size of index, partitioning is recommended?

- Is there any max size when replicating segments from master to slave?

When replicating, is there a segment size limit when the slave won't be able to download the content and index it? What is the threshold to which a slave won't be able to replicate when there's a lot of traffic to retrieve info and lots of new documents to replicate.

To be more factual, here is the context that led me to these questions:
We want to index a fair amount of documents, but when the amount reaches more than a dozen millions, the slaves can't handle it and start failing replicating with a SnapPull error.
The documents are composed with a few text fields (name, type, description, ... about 10 other fields of let's say 20 characters max).

We have one master, and 2 slaves which replicate data from the master.

This is my first time working with Solr (I work usually on webapps using spring, hibernate... but no use of Solr), so I'm not sure how to tackle this issue.

Our idea is for the moment to add multiple cores to the master, and have a slave replicating from each of this core.
Is it the right way to go?

If it is, how can we determine the number of cores needed? Right now we're just going to try and see how it behaves and adjust if necessary, but I was wondering if there was any best practices or some benchmarks that have been done on this specific topic.

For this amount of documents with this average size, x cores or indexes are needed ...

Thanks for any help in how I could deal with a huge amount of documents of average size!

Here is a copy of the error that is being thrown when a slave is trying to replicate:

ERROR [org.apache.solr.handler.ReplicationHandler] - <SnapPull failed >
org.apache.solr.common.SolrException: Index fetch failed :
        at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
        at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
        at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.RuntimeException: java.io.IOException: read past EOF
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
        at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:467)
        at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
        ... 11 more
Caused by: java.io.IOException: read past EOF
        at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:151)
        at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
        at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:70)
        at org.apache.lucene.index.SegmentInfos$2.doBody(SegmentInfos.java:410)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:704)
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:538)
        at org.apache.lucene.index.SegmentInfos.readCurrentVersion(SegmentInfos.java:402)
        at org.apache.lucene.index.DirectoryReader.isCurrent(DirectoryReader.java:791)
        at org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:404)
        at org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:352)
        at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:413)
        at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:424)
        at org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:35)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1049)
        ... 14 more

EDIT:
After Mauricio's answer, the solr libraries have been updated to 1.4.1 but this error was still raised.
I increased the commitReserveDuration and even if the "SnapPull Failed" error seems to have disappeared, another one started being raised, not sure about why as I can't seem to find much answer on the web:

ERROR [org.apache.solr.servlet.SolrDispatchFilter] - <ClientAbortException:  java.io.IOException
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:370)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:323)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
        at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:183)
        at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:89)
        at org.apache.solr.request.BinaryResponseWriter.write(BinaryResponseWriter.java:48)
        at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:322)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:837)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:640)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
        at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException
        at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:703)
        at org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:733)
        at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:124)
        at org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(InternalAprOutputBuffer.java:539)
        at org.apache.coyote.Response.doWrite(Response.java:560)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:365)
        ... 22 more
>
ERROR [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/].[SolrServer]] - <Servlet.service() for servlet SolrServer threw exception>
java.lang.IllegalStateException
        at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:405)
        at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:272)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:837)
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:640)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
        at java.lang.Thread.run(Thread.java:595)

I still wonder what are the best practices to handle a big index (more than 20G) containing a lot of documents with solr. Am I missing some obvious links somewhere? Tutorials, documentations?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

清泪尽 2024-10-09 09:01:53

核心是一种主要用于在单个 Solr 实例中具有不同模式的工具。也用作甲板索引。分片和复制是正交问题。
你提到“流量很大”。这是一个非常主观的衡量标准。相反，尝试确定您需要 Solr 的 QPS（每秒查询数）。另外，单个 Solr 实例是否足够快地回答您的查询？只有这样您才能确定是否需要横向扩展。单个 Solr 实例可以处理大量流量，也许您甚至不需要扩展。
确保在具有足够内存的服务器上运行 Solr（并确保 Java 可以访问它）。 Solr 非常消耗内存，如果将其放在内存受限的服务器上，性能将会受到影响。
正如 Solr wiki 所解释的，如果单个查询运行时间太长，请使用分片，如果单个 Solr 实例无法处理流量，请使用复制。 “太长”和“流量”取决于您的特定应用程序。测量它们。
Solr 有很多影响性能的设置：缓存自动预热、存储字段、合并因子等。查看 SolrPerformanceFactors 。
这里没有硬性规定。每个应用程序都有不同的搜索需求。针对您的特定场景进行模拟和测量。
关于复制错误，请确保您运行的是 1.4.1，因为 1.4.0 存在复制错误。