使用psycopg2时,如何获得copy_from()函数的执行时间?
现在,我正在使用psycopg2连接到PostgreSQL,并使用复制功能将本地TXT数据加载到数据库中。
目前,我的代码如下:
#!/usr/bin/env python
import time
import psycopg2
import psycopg2.extensions
from psycopg2.extras import LoggingConnection, LoggingCursor
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class MyLoggingCursor(LoggingCursor):
def execute(self, query, vars=None):
self.timestamp = time.time()
return super(MyLoggingCursor, self).execute(query, vars)
def callproc(self, procname, vars=None):
self.timestamp = time.time()
return super(MyLoggingCursor, self).callproc(procname, vars)
class MyLoggingConnection(LoggingConnection):
def filter(self, msg, curs):
return msg + " %d ms" % int((time.time() - curs.timestamp) * 1000)
def cursor(self, *args, **kwargs):
kwargs.setdefault('cursor_factory', MyLoggingCursor)
return LoggingConnection.cursor(self, *args, **kwargs)
db_settings = {
"database": "gps_stat",
"user": "user",
"password": "test@123",
"host": "localhost",
"port": 26000
}
target_txt = "test.txt"
table_name = 'gps_base'
start_time=time.time()
conn = psycopg2.connect(connection_factory=MyLoggingConnection, **db_settings)
conn.initialize(logger)
cur = conn.cursor('cursor_unique_name', cursor_factory=psycopg2.extras.DictCursor)
f = open(target_txt, 'r')
cur.copy_from(f, table_name, sep=',')
conn.commit()
conn.close()
f.close()
print("Time:{}ms".format(time.time()-start_time))
但是myloggingConnection
不满足我的需求。
Test_TXT中大约有2亿个数据。上述程序运行后,它打印时间:2000ms
,但实际上我等待了大约20分钟才能获得时间的结果:2000ms
。
我该怎么做才能获得Copy_from()函数的真实执行时间?
Now I am using psycopg2 to connect to postgresql and use the copy function to load the local txt data into the database.
Currently my code is as follows:
#!/usr/bin/env python
import time
import psycopg2
import psycopg2.extensions
from psycopg2.extras import LoggingConnection, LoggingCursor
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class MyLoggingCursor(LoggingCursor):
def execute(self, query, vars=None):
self.timestamp = time.time()
return super(MyLoggingCursor, self).execute(query, vars)
def callproc(self, procname, vars=None):
self.timestamp = time.time()
return super(MyLoggingCursor, self).callproc(procname, vars)
class MyLoggingConnection(LoggingConnection):
def filter(self, msg, curs):
return msg + " %d ms" % int((time.time() - curs.timestamp) * 1000)
def cursor(self, *args, **kwargs):
kwargs.setdefault('cursor_factory', MyLoggingCursor)
return LoggingConnection.cursor(self, *args, **kwargs)
db_settings = {
"database": "gps_stat",
"user": "user",
"password": "test@123",
"host": "localhost",
"port": 26000
}
target_txt = "test.txt"
table_name = 'gps_base'
start_time=time.time()
conn = psycopg2.connect(connection_factory=MyLoggingConnection, **db_settings)
conn.initialize(logger)
cur = conn.cursor('cursor_unique_name', cursor_factory=psycopg2.extras.DictCursor)
f = open(target_txt, 'r')
cur.copy_from(f, table_name, sep=',')
conn.commit()
conn.close()
f.close()
print("Time:{}ms".format(time.time()-start_time))
But the MyLoggingConnection
does not meet my needs.
There are about 200 million pieces of data in the test_txt. After the above program runs, it prints Time: 2000ms
, but in fact I waited about 20 minutes to get the result of Time: 2000ms
.
What can I do to get the real execution time of the copy_from() function?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
最初我有同样的问题,然后我写了小程序来验证这一点,并知道这一点
,输出是
这意味着,您要获得的时间在几秒钟内,因为我将时间放在3秒钟内。如果是在MS中,则结果应该为3000 ms,而结果为3.010,只是SEC。我相信您的执行时间为2000秒。
愉快的编码!
Initially I had same issue and then I have written small program to verify this and came to know this
and output is
It means, time which you are getting is in seconds as I am putting time on sleep for 3 sec. If it is in ms then, result should have been 3000+ ms whereas result is 3.010 which is nothing but sec. I believe your execution is taking total 2000 sec.
happy coding!