当前位置： 文江博客 话题详情

无法查询链接电影数据库的本地版本

发布于 2024-10-31 14:27:00 字数 6823 浏览 0 评论 0 原文

我正在尝试使用 SPARQL 查询链接电影数据库的本地版本。该文件为 N-Triples 格式，大小约为 450mb。我正在使用 servlet 来实现。现在，当我传递查询时，servlet 大约需要五分钟多的时间来处理它，最后我得到以下异常：

type Exception report

message 

description The server encountered an internal error () that prevented it from fulfilling this request.

exception 

javax.servlet.ServletException: Servlet execution threw an exception


root cause 

java.lang.OutOfMemoryError: Java heap space
    java.util.Arrays.copyOfRange(Arrays.java:3209)
    java.lang.String.<init>(String.java:215)
    java.lang.StringBuilder.toString(StringBuilder.java:430)
    org.openjena.riot.tokens.TokenizerText.allBetween(TokenizerText.java:732)
    org.openjena.riot.tokens.TokenizerText.parseToken(TokenizerText.java:152)
    org.openjena.riot.tokens.TokenizerText.hasNext(TokenizerText.java:69)
    org.openjena.atlas.iterator.PeekIterator.fill(PeekIterator.java:37)
    org.openjena.atlas.iterator.PeekIterator.next(PeekIterator.java:77)
    org.openjena.riot.lang.LangBase.nextToken(LangBase.java:145)
    org.openjena.riot.lang.LangNTriples.parseOne(LangNTriples.java:59)
    org.openjena.riot.lang.LangNTriples.parseOne(LangNTriples.java:21)
    org.openjena.riot.lang.LangNTuple.runParser(LangNTuple.java:58)
    org.openjena.riot.lang.LangBase.parse(LangBase.java:75)
    org.openjena.riot.system.JenaReaderNTriples2.readWorker(JenaReaderNTriples2.java:28)
    org.openjena.riot.system.JenaReaderRIOT.readImpl(JenaReaderRIOT.java:124)
    org.openjena.riot.system.JenaReaderRIOT.read(JenaReaderRIOT.java:79)
    com.hp.hpl.jena.rdf.model.impl.ModelCom.read(ModelCom.java:226)
    com.hp.hpl.jena.util.FileManager.readModelWorker(FileManager.java:395)
    com.hp.hpl.jena.util.FileManager.loadModelWorker(FileManager.java:299)
    com.hp.hpl.jena.util.FileManager.loadModel(FileManager.java:250)
    ServletExample.runQuery(ServletExample.java:92)
    ServletExample.doGet(ServletExample.java:62)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:729)


note The full stack trace of the root cause is available in the Apache Tomcat/5.5.31 logs.

我的代码是：

import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.ServletException;
import javax.servlet.http.*;

import com.hp.hpl.jena.query.*;
import com.hp.hpl.jena.rdf.model.*;
import com.hp.hpl.jena.util.FileManager;

public class ServletExample
    extends HttpServlet
{
    /***********************************/
    /* Constants                       */
    /***********************************/

    private static final long serialVersionUID = 1L;

    public static final String SPARQL_ENDPOINT = "http://data.linkedmdb.org/sparql";

    public static final String QUERY ="PREFIX m: <http://data.linkedmdb.org/resource/movie/>"
+"SELECT DISTINCT ?actorName WHERE {"+
"?dir1 m:director_name \"Sofia Coppola\"."+
"?dir2 m:director_name \"Francis Ford Coppola\"."+
"?dir1film m:director ?dir1;"+
"m:actor ?actor."+
"?dir2film m:director ?dir2;"+
"m:actor ?actor."+
"?actor m:actor_name ?actorName."+
"}";
        /*"PREFIX m: <http://data.linkedmdb.org/resource/movie/>\n" +
            "SELECT DISTINCT ?actorName WHERE {\n" +
            "  ?dir1     m:director_name %dir_name_1%.\n" +
            "  ?dir2     m:director_name %dir_name_2%.\n" +
            "  ?dir1film m:director ?dir1;\n" +
            "            m:actor ?actor.\n" +
            "  ?dir2film m:director ?dir2;\n" +
            "            m:actor ?actor.\n" +
            "  ?actor    m:actor_name ?actorName.\n" +
            "}\n" +
            "";*/

    private static final String HEADER = "<html>\n" +
            "      <head>\n" +
            "        <title>results</title>\n" +
            "          <link href=\"simple.css\" type=\"text/css\" rel=\"stylesheet\" />\n" +
            "      </head>\n" +
            "      <body>\n" +
            "";

    private static final String FOOTER = "</body></html>";

    /**
     * Respond to HTTP GET request. Will need to be mounted against some URL
     * pattern in web.xml
     */
    @Override
    protected void doGet( HttpServletRequest req, HttpServletResponse resp )
        throws ServletException, IOException
    {
        String dir1 = req.getParameter( "dir1" );//"Sofia";
       String dir2 = req.getParameter( "dir2" );//"Francis Ford Coppola";
        //String dir1 = "Sofia";
        //String dir2 = "Francis Ford Coppola"; 
        if (dir1 == null || dir2 == null || dir1.isEmpty() || dir2.isEmpty()) {
            noInput( resp );
        }
        else {
            runQuery( resp, dir1, dir2 );
        }
    }

    protected void noInput( HttpServletResponse resp )
        throws IOException
    {
        header( resp );
        resp.getWriter().println( "<p>Please select director names as query params <code>dir1</code> and <code>dir2</code></p>" );
        footer( resp );
    }

    protected void footer( HttpServletResponse resp ) throws IOException {
        resp.getWriter().println( FOOTER );
    }

    protected void header( HttpServletResponse resp ) throws IOException {
        resp.getWriter().println( HEADER );
    }

    protected void runQuery( HttpServletResponse resp, String dir1, String dir2 )
        throws IOException
    {
        PrintWriter out = resp.getWriter();

        // Set up the query
   //     String q = QUERY.replace( "%dir_name_1%", "\"" + dir1 + "\"" )
     //                   .replace( "%dir_name_2%", "\"" + dir2 + "\"" );
        String q=QUERY;
        Query query = QueryFactory.create( q ) ;
        Model model = FileManager.get().loadModel( "e:\\applications\\linkedmdb-18-05-2009-dump\\dump.nt" );
       // QueryExecution qexec = QueryExecutionFactory.sparqlService( SPARQL_ENDPOINT, query );

       //com.hp.hpl.jena.query.Query query = QueryFactory.create(QUERY);
      QueryExecution qexec = QueryExecutionFactory.create(query, model);
        // perform the query
        ResultSet results = qexec.execSelect();

        // generate the output
        header( resp );
        if (!results.hasNext()) {
            out.println( "<p>No results, sorry.</p>" );
        }
        else {
            out.println( "<h1>Results</h1>" );
            while (results.hasNext()) {
                QuerySolution qs = results.next();
                String actorName = qs.getLiteral( "actorName" ).getLexicalForm();
                out.println( String.format( "<div>Actor named: %s</div>", actorName ) );
            }
        }
        footer( resp );
    }
}

有什么方法可以解决此异常吗？

原文

I am trying to query a local version of Linked Movie Database using SPARQL. The file is in N-Triples format and its size is approximately 450mb. I am using servlets for implementation. Now when I pass the query, it takes about more than five minutes for the servlet to process it and at the end I get the following exception:

type Exception report

message 

description The server encountered an internal error () that prevented it from fulfilling this request.

exception 

javax.servlet.ServletException: Servlet execution threw an exception


root cause 

java.lang.OutOfMemoryError: Java heap space
    java.util.Arrays.copyOfRange(Arrays.java:3209)
    java.lang.String.<init>(String.java:215)
    java.lang.StringBuilder.toString(StringBuilder.java:430)
    org.openjena.riot.tokens.TokenizerText.allBetween(TokenizerText.java:732)
    org.openjena.riot.tokens.TokenizerText.parseToken(TokenizerText.java:152)
    org.openjena.riot.tokens.TokenizerText.hasNext(TokenizerText.java:69)
    org.openjena.atlas.iterator.PeekIterator.fill(PeekIterator.java:37)
    org.openjena.atlas.iterator.PeekIterator.next(PeekIterator.java:77)
    org.openjena.riot.lang.LangBase.nextToken(LangBase.java:145)
    org.openjena.riot.lang.LangNTriples.parseOne(LangNTriples.java:59)
    org.openjena.riot.lang.LangNTriples.parseOne(LangNTriples.java:21)
    org.openjena.riot.lang.LangNTuple.runParser(LangNTuple.java:58)
    org.openjena.riot.lang.LangBase.parse(LangBase.java:75)
    org.openjena.riot.system.JenaReaderNTriples2.readWorker(JenaReaderNTriples2.java:28)
    org.openjena.riot.system.JenaReaderRIOT.readImpl(JenaReaderRIOT.java:124)
    org.openjena.riot.system.JenaReaderRIOT.read(JenaReaderRIOT.java:79)
    com.hp.hpl.jena.rdf.model.impl.ModelCom.read(ModelCom.java:226)
    com.hp.hpl.jena.util.FileManager.readModelWorker(FileManager.java:395)
    com.hp.hpl.jena.util.FileManager.loadModelWorker(FileManager.java:299)
    com.hp.hpl.jena.util.FileManager.loadModel(FileManager.java:250)
    ServletExample.runQuery(ServletExample.java:92)
    ServletExample.doGet(ServletExample.java:62)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:729)


note The full stack trace of the root cause is available in the Apache Tomcat/5.5.31 logs.

My code is:

import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.ServletException;
import javax.servlet.http.*;

import com.hp.hpl.jena.query.*;
import com.hp.hpl.jena.rdf.model.*;
import com.hp.hpl.jena.util.FileManager;

public class ServletExample
    extends HttpServlet
{
    /***********************************/
    /* Constants                       */
    /***********************************/

    private static final long serialVersionUID = 1L;

    public static final String SPARQL_ENDPOINT = "http://data.linkedmdb.org/sparql";

    public static final String QUERY ="PREFIX m: <http://data.linkedmdb.org/resource/movie/>"
+"SELECT DISTINCT ?actorName WHERE {"+
"?dir1 m:director_name \"Sofia Coppola\"."+
"?dir2 m:director_name \"Francis Ford Coppola\"."+
"?dir1film m:director ?dir1;"+
"m:actor ?actor."+
"?dir2film m:director ?dir2;"+
"m:actor ?actor."+
"?actor m:actor_name ?actorName."+
"}";
        /*"PREFIX m: <http://data.linkedmdb.org/resource/movie/>\n" +
            "SELECT DISTINCT ?actorName WHERE {\n" +
            "  ?dir1     m:director_name %dir_name_1%.\n" +
            "  ?dir2     m:director_name %dir_name_2%.\n" +
            "  ?dir1film m:director ?dir1;\n" +
            "            m:actor ?actor.\n" +
            "  ?dir2film m:director ?dir2;\n" +
            "            m:actor ?actor.\n" +
            "  ?actor    m:actor_name ?actorName.\n" +
            "}\n" +
            "";*/

    private static final String HEADER = "<html>\n" +
            "      <head>\n" +
            "        <title>results</title>\n" +
            "          <link href=\"simple.css\" type=\"text/css\" rel=\"stylesheet\" />\n" +
            "      </head>\n" +
            "      <body>\n" +
            "";

    private static final String FOOTER = "</body></html>";

    /**
     * Respond to HTTP GET request. Will need to be mounted against some URL
     * pattern in web.xml
     */
    @Override
    protected void doGet( HttpServletRequest req, HttpServletResponse resp )
        throws ServletException, IOException
    {
        String dir1 = req.getParameter( "dir1" );//"Sofia";
       String dir2 = req.getParameter( "dir2" );//"Francis Ford Coppola";
        //String dir1 = "Sofia";
        //String dir2 = "Francis Ford Coppola"; 
        if (dir1 == null || dir2 == null || dir1.isEmpty() || dir2.isEmpty()) {
            noInput( resp );
        }
        else {
            runQuery( resp, dir1, dir2 );
        }
    }

    protected void noInput( HttpServletResponse resp )
        throws IOException
    {
        header( resp );
        resp.getWriter().println( "<p>Please select director names as query params <code>dir1</code> and <code>dir2</code></p>" );
        footer( resp );
    }

    protected void footer( HttpServletResponse resp ) throws IOException {
        resp.getWriter().println( FOOTER );
    }

    protected void header( HttpServletResponse resp ) throws IOException {
        resp.getWriter().println( HEADER );
    }

    protected void runQuery( HttpServletResponse resp, String dir1, String dir2 )
        throws IOException
    {
        PrintWriter out = resp.getWriter();

        // Set up the query
   //     String q = QUERY.replace( "%dir_name_1%", "\"" + dir1 + "\"" )
     //                   .replace( "%dir_name_2%", "\"" + dir2 + "\"" );
        String q=QUERY;
        Query query = QueryFactory.create( q ) ;
        Model model = FileManager.get().loadModel( "e:\\applications\\linkedmdb-18-05-2009-dump\\dump.nt" );
       // QueryExecution qexec = QueryExecutionFactory.sparqlService( SPARQL_ENDPOINT, query );

       //com.hp.hpl.jena.query.Query query = QueryFactory.create(QUERY);
      QueryExecution qexec = QueryExecutionFactory.create(query, model);
        // perform the query
        ResultSet results = qexec.execSelect();

        // generate the output
        header( resp );
        if (!results.hasNext()) {
            out.println( "<p>No results, sorry.</p>" );
        }
        else {
            out.println( "<h1>Results</h1>" );
            while (results.hasNext()) {
                QuerySolution qs = results.next();
                String actorName = qs.getLiteral( "actorName" ).getLexicalForm();
                out.println( String.format( "<div>Actor named: %s</div>", actorName ) );
            }
        }
        footer( resp );
    }
}

Is there any way to resolve this exception?

如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论， 你可以免费 注册 一个本站的账号。

评论（2）

寂寞清仓 2024-11-07 14:27:00

看来您正在使用 Jena/RIOT 将所有数据加载到内存中。据我所知，LinkedIMDB 足够大，足以给您带来这种方法的问题。您正在做的是将所有数据库调入内存。

增加 JVM 堆可能是一种可能的解决方案，但如果数据持续增长，它就无法扩展。

正确的解决方案是采用专为这种大小的数据集设计的其他 Jena 配置。它们是：

  1. Jena SDB，它使用关系数据库作为后端。
  2. Jena TDB，它使用基于 B 树索引的本机 Java 存储来加快查询速度。它的扩展性比 (1) 更好。

或者，您可以选择可扩展的 RDF 数据库，例如 4store 并通过 耶拿 ARQ。该解决方案是迄今为止可扩展且性能更好的解决方案。

It seems you're loading all your data in memory using Jena/RIOT. As far as I know, LinkedIMDB is large enough to give you problems with this approach. What you're doing is bringing all your database to memory.

Increasing the heap of your JVM could be one possible solution but it won't scale if your data keeps growing.

The right solution is to go for other configurations of Jena that are designed for this size of datasets. These are:

  1. Jena SDB, which uses relational databases as backend.
  2. Jena TDB, which uses a native Java storage based on B-trees indexes to speed up queries. It scales better than (1).

Optionally you could go for scalable RDF databases such 4store and query your data via Jena ARQ. This solution is by far the one that will scale and perform better.

回复 原文
我一向站在原地 2024-11-07 14:27:00

Java 虚拟机 (JVM) 中的堆内存不足。要么增加 JVM 可用的堆内存量，要么设计您的软件以使用更少的内存，例如以较小的块处理内容。

要增加堆内存，请将这些参数添加到 servlet 容器或应用程序服务器的启动脚本中，即执行 java 二进制文件的位置。这告诉 JVM 它可能使用最多 512 MB 的内存，如果这还不够，请尝试使用更大的值：

-Xmx512m

在不查看实际代码的情况下，很难说如何改进您的软件以使用更少的内存。

You are running out of heap memory in Java Virtual Machine (JVM). Either increase amount of heap memory that is available to JVM or design your software to use less memory, for example process the stuff in smaller chunks.

To increase heap memory, add these parameters to your servlet container's or application server's startup script, somewhere your java binary is executed. This tells JVM that it may use up to 512 megabytes of memory, if that is not enough, try with larger values:

-Xmx512m

It is hard to say how to improve your software to use less memory without seeing the actual code.

回复 原文
~没有更多了~

关于作者

胡大本事

暂无简介

0 文章
0 评论
22 人气
发私信

相关话题

更多

热门标签

操作系统 程序设计 IT运维 Linux系统管理 JavaScript 服务器应用 solaris C/C++ PHP Shell BSD Vue.js aix Oracle Python HTML 系统管理 HTML5 CSS 前端
更多

推荐作者

1CH1MKgiKxn9p

文章 0 评论 0

ゞ记忆︶ㄣ

文章 0 评论 0

JackDx

文章 0 评论 0

信远

文章 0 评论 0

yaoduoduo1995

文章 0 评论 0

霞映澄塘

文章 0 评论 0

更多

友情链接

文江博客
    我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站，即表示您同意使用 Cookies 和您的相关数据。
    原文