在 Java/MySQL 中调用 Statement.cancel() 强制关闭大型结果集的后果

发布于 2024-11-26 09:57:34 字数 2749 浏览 0 评论 0原文

我正在开发一个网站,允许注册用户搜索 10-2000 万条记录,并且这些记录将与许多内存缓存(在 Java 中)交叉引​​用。其中一些缓存只有 5000 条左右的记录,其中之一将接近 1.8 亿条记录。

因此,我在每个 SQL 查询中只设置一个索引。有时用户不需要索引,因此 Java 端将获得迭代数据库中每条记录的语句的 Integer.MIN_VALUE 获取大小。

JSP 页面每 2 秒查询一次进度并更新 JavaScript 进度条等。

一切都运行良好,速度非常快,但我允许用户取消正在进行的查询,方法是从另一个线程设置一个易失性布尔值,然后在rs.next() 检查它是否被取消并跳出 while 循环,并正确关闭数据库资源。

在 1000 万条记录中取消 ResultSet 需要花费近 45 秒的时间。

经过一些测试后,ResultSet.close() 方法延迟了快速取消,以允许用户返回编辑其参数。但是,如果我在关闭 ResultSet 之前取消语句 (Statement.cancel()),它将立即关闭并退出搜索。

要点如下:

public void search() throws Exception {
    total = getSize(where); // where clause created dynamically and depending on options user chooses may be null
current = 0;
Connection c = getConnection();

    Statement s = c.createStatement();
s.setFetchSize(Integer.MIN_VALUE);
ResultSet rs = s.executeQuery(queryBuffer.toString()); // potentially a query returning a lot of stuff

    logger.info("Full Query " + queryBuffer.toString());

    long stime = System.currentTimeMillis();

    while (rs.next()) {

        Record rec = helper.deriveLightweightRecord(rs);

        current++;

        if (stack.accept(rec)) { // stack of filters to check records against, one has 180 million records in cache
            valid++;
        }

        if (requestCancel) {
            logger.info("Cancel Detected at Search Thread Breaking Now " + System.currentTimeMillis());
            break;
        }

        if (current % 20000 == 0) {

            long etime = System.currentTimeMillis();
            logger.info("Iterated 20000 Records in " + ((etime - stime) / 1000.0) + " Seconds");
            stime = etime;
        }
    }


    try {
        if (requestCancel)
            s.cancel(); // if i don't call cancel, the result set close takes a LONG time >30secs, calling cancel the ResultSet close takes all of a couple millis
            s.close();
            rs.close();
    } catch (Exception e) {
        logger.error("Ignorable for now", e); // occassionally get a SQLException because of cancel
    } finally {
        returnConnection(c);
}
    logger.info("Closed Connection " + System.currentTimeMillis());

    if (requestCancel) {
        logger.info("Checking Request Cancel Now " + System.currentTimeMillis());
        cancelled = true;
        searching = false;
        cancelNotifier.interrupt();
        resetStatus();
    } else {
        current = total;
        searching = false;
    }

 }

回答这个问题,取消声明会产生什么后果,因为我以前从未使用过此功能。 Mysql Connector 和 Java API 上的 javadoc 并没有过多提及 Statement.cancel(),除了如果底层 RDMS 不支持(MySQL 似乎支持)的话,它可能会抛出不支持的操作异常。

这在专用网络服务器上会降低性能或容易出错吗?似乎运行良好,除了偶尔抛出有关尝试关闭已取消语句上的 ResultSet 的 SQLException 之外。流式传输结果集时这是否会对数据库造成潜在损害?

提前致谢。

I'm developing a website that will allow registered users to search through anywhere from 10-20 million records, and those records will be cross referenced against a number of in memory caches (in Java). Some of these caches will only be 5000 or so records one of them will be close to 180 million records.

Therefore, I only have one index that I set in each SQL query. Sometimes users don't need the index, so the Java end will get a Integer.MIN_VALUE fetch size of a statement that iterates through every record in the database.

Every 2 seconds the JSP page queries the progress and updates a JavaScript progress bar, etc.

Everything works great, and it's blazing fast, but I offer users to cancel the query in progress, by setting a volatile boolean from another thread, and then in the rs.next() check if it's cancelled and break out of the while loop, and properly close DB resources.

It took nearly 45 seconds to cancel on a ResultSet halfway through 10 mil records.

After doing some testing it is the ResultSet.close() method that is delaying a quick cancel to allow the user to go back to editing their parameters. However if I cancel the statement (Statement.cancel()) before closing the ResultSet it closes and exits the search immediately.

Here's the gist of it:

public void search() throws Exception {
    total = getSize(where); // where clause created dynamically and depending on options user chooses may be null
current = 0;
Connection c = getConnection();

    Statement s = c.createStatement();
s.setFetchSize(Integer.MIN_VALUE);
ResultSet rs = s.executeQuery(queryBuffer.toString()); // potentially a query returning a lot of stuff

    logger.info("Full Query " + queryBuffer.toString());

    long stime = System.currentTimeMillis();

    while (rs.next()) {

        Record rec = helper.deriveLightweightRecord(rs);

        current++;

        if (stack.accept(rec)) { // stack of filters to check records against, one has 180 million records in cache
            valid++;
        }

        if (requestCancel) {
            logger.info("Cancel Detected at Search Thread Breaking Now " + System.currentTimeMillis());
            break;
        }

        if (current % 20000 == 0) {

            long etime = System.currentTimeMillis();
            logger.info("Iterated 20000 Records in " + ((etime - stime) / 1000.0) + " Seconds");
            stime = etime;
        }
    }


    try {
        if (requestCancel)
            s.cancel(); // if i don't call cancel, the result set close takes a LONG time >30secs, calling cancel the ResultSet close takes all of a couple millis
            s.close();
            rs.close();
    } catch (Exception e) {
        logger.error("Ignorable for now", e); // occassionally get a SQLException because of cancel
    } finally {
        returnConnection(c);
}
    logger.info("Closed Connection " + System.currentTimeMillis());

    if (requestCancel) {
        logger.info("Checking Request Cancel Now " + System.currentTimeMillis());
        cancelled = true;
        searching = false;
        cancelNotifier.interrupt();
        resetStatus();
    } else {
        current = total;
        searching = false;
    }

 }

Getting to the question, what are the ramifications of cancelling a Statement, since I've never had to use this function before. The javadocs on Mysql Connector and Java API don't really mention too much about Statement.cancel(), save for the fact that it could throw a unsupported operation exception if the underlying RDMS doesn't support it (which MySQL seems to).

Will this be degrading or error prone on a dedicated webserver? Seems to run fine, besides very occassionally throwing a SQLException about trying to close a ResultSet on a cancelled statement. Is this potentially damaging to the the database when streaming ResultSets?

Thanks in advance.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文