使用psycopg2时,如何获得copy_from()函数的执行时间?

发布于 2025-02-11 17:50:02 字数 1772 浏览 0 评论 0原文

现在,我正在使用psycopg2连接到PostgreSQL,并使用复制功能将本地TXT数据加载到数据库中。

目前,我的代码如下:

#!/usr/bin/env python
import time
import psycopg2
import psycopg2.extensions
from psycopg2.extras import LoggingConnection, LoggingCursor
import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

class MyLoggingCursor(LoggingCursor):
    def execute(self, query, vars=None):
        self.timestamp = time.time()
        return super(MyLoggingCursor, self).execute(query, vars)

    def callproc(self, procname, vars=None):
        self.timestamp = time.time()
        return super(MyLoggingCursor, self).callproc(procname, vars)

class MyLoggingConnection(LoggingConnection):
    def filter(self, msg, curs):
        return msg + "   %d ms" % int((time.time() - curs.timestamp) * 1000)

    def cursor(self, *args, **kwargs):
        kwargs.setdefault('cursor_factory', MyLoggingCursor)
        return LoggingConnection.cursor(self, *args, **kwargs)

db_settings = {
    "database": "gps_stat",
    "user": "user",
    "password": "test@123",
    "host": "localhost",
    "port": 26000
}

target_txt = "test.txt"
table_name = 'gps_base'

start_time=time.time()
conn = psycopg2.connect(connection_factory=MyLoggingConnection, **db_settings)
conn.initialize(logger)
cur = conn.cursor('cursor_unique_name', cursor_factory=psycopg2.extras.DictCursor)
f = open(target_txt, 'r')
cur.copy_from(f, table_name, sep=',')
conn.commit()
conn.close()
f.close()
print("Time:{}ms".format(time.time()-start_time))

但是myloggingConnection不满足我的需求。

Test_TXT中大约有2亿个数据。上述程序运行后,它打印时间:2000ms,但实际上我等待了大约20分钟才能获得时间的结果:2000ms

我该怎么做才能获得Copy_from()函数的真实执行时间?

Now I am using psycopg2 to connect to postgresql and use the copy function to load the local txt data into the database.

Currently my code is as follows:

#!/usr/bin/env python
import time
import psycopg2
import psycopg2.extensions
from psycopg2.extras import LoggingConnection, LoggingCursor
import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

class MyLoggingCursor(LoggingCursor):
    def execute(self, query, vars=None):
        self.timestamp = time.time()
        return super(MyLoggingCursor, self).execute(query, vars)

    def callproc(self, procname, vars=None):
        self.timestamp = time.time()
        return super(MyLoggingCursor, self).callproc(procname, vars)

class MyLoggingConnection(LoggingConnection):
    def filter(self, msg, curs):
        return msg + "   %d ms" % int((time.time() - curs.timestamp) * 1000)

    def cursor(self, *args, **kwargs):
        kwargs.setdefault('cursor_factory', MyLoggingCursor)
        return LoggingConnection.cursor(self, *args, **kwargs)

db_settings = {
    "database": "gps_stat",
    "user": "user",
    "password": "test@123",
    "host": "localhost",
    "port": 26000
}

target_txt = "test.txt"
table_name = 'gps_base'

start_time=time.time()
conn = psycopg2.connect(connection_factory=MyLoggingConnection, **db_settings)
conn.initialize(logger)
cur = conn.cursor('cursor_unique_name', cursor_factory=psycopg2.extras.DictCursor)
f = open(target_txt, 'r')
cur.copy_from(f, table_name, sep=',')
conn.commit()
conn.close()
f.close()
print("Time:{}ms".format(time.time()-start_time))

But the MyLoggingConnection does not meet my needs.

There are about 200 million pieces of data in the test_txt. After the above program runs, it prints Time: 2000ms, but in fact I waited about 20 minutes to get the result of Time: 2000ms.

What can I do to get the real execution time of the copy_from() function?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

场罚期间 2025-02-18 17:50:02

最初我有同样的问题,然后我写了小程序来验证这一点,并知道这一点

start_time=time.time()

time.sleep(3)

print("Total Time Taken :: {} ms".format(time.time()-start_time))

,输出是

总时间:: 3.010446071624756 MS

这意味着,您要获得的时间在几秒钟内,因为我将时间放在3秒钟内。如果是在MS中,则结果应该为3000 ms,而结果为3.010,只是SEC。我相信您的执行时间为2000秒。

愉快的编码!

Initially I had same issue and then I have written small program to verify this and came to know this

start_time=time.time()

time.sleep(3)

print("Total Time Taken :: {} ms".format(time.time()-start_time))

and output is

Total Time Taken :: 3.010446071624756 ms

It means, time which you are getting is in seconds as I am putting time on sleep for 3 sec. If it is in ms then, result should have been 3000+ ms whereas result is 3.010 which is nothing but sec. I believe your execution is taking total 2000 sec.

happy coding!

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文