Python 中的多线程 Pickling

发布于 12-05 23:54 字数 182 浏览 2 评论 0原文

我有一个带有多个线程的 python 程序。每个线程都会检测事件,我想将其存储在某个地方,以便我可以再次读取它们(用于测试)。现在,我使用 Pickle 输出事件,每个线程输出到不同的文件。理想情况下,我只会使用一个输出文件,所有线程都会写入该文件,但是当我尝试这样做时,看起来各个线程尝试同时写入其输出,并且它们没有被正确腌制。有办法做到这一点吗?

I have a python program with multiple threads. Each thread detects events, which I would like to store somewhere so that I can read them in again (for testing). Right now, I'm using Pickle to output the events, and each thread outputs to a different file. Ideally, I would only use one output file, and all the threads would write to it, but when I try this, it looks like the various threads try to write their output at the same time, and they don't get pickled properly. Is there a way to do this?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(5

一片旧的回忆2024-12-12 23:54:55

似乎是使用 Queue 的好地方。

  • 让所有事件检测线程将项目放入共享队列中。
  • 创建另一个线程以从队列中获取项目,并从此线程中写入/腌制/任何内容。

来自队列文档:

“Queue模块实现了多生产者、多消费者队列。它
当信息必须被传递时,在线程编程中特别有用
在多个线程之间安全地交换。其中的 Queue 类
模块实现所有必需的锁定语义。这取决于
Python 中线程支持的可用性;看线程
模块。”

seems like a good place to use a Queue.

  • Have all your event detection threads put items on a shared Queue.
  • Create another thread to get items from the queue, and write/pickle/whatever from this thread.

from the Queue docs:

"The Queue module implements multi-producer, multi-consumer queues. It
is especially useful in threaded programming when information must be
exchanged safely between multiple threads. The Queue class in this
module implements all the required locking semantics. It depends on
the availability of thread support in Python; see the threading
module."

戏蝶舞2024-12-12 23:54:55

是的,使用 threading.Lock() 对象。
您在创建所有线程之前创建一个锁,将其交给负责保存/腌制项目的方法,并且该方法应该在写入文件之前获取锁并在之后释放它。

Yes, with threading.Lock() objects.
You create a lock before creating all your threads, you give it to the method that is responsible of saving/pickling items, and this method should acquire the lock before writing into the file and releasing it after.

吻风2024-12-12 23:54:55

您可以创建一个 并在每次调用 < 时获取/释放它代码>pickle.dump()。

You could create a lock and acquire/release it around every call to pickle.dump().

温柔女人霸气范2024-12-12 23:54:55

logging 模块内置了一个 Rlock它的处理程序。因此,您可以像平常一样记录(只需创建一个处理程序来记录到文件。)

The logging module has a Rlock built into its Handlers. So you could logging as normal (just create a handler to log to a file.)

白馒头2024-12-12 23:54:55

以下是使用 threading.Lock() 的示例:

import threading
import pickle
picke_lock = threading.Lock()
   def do(s):
       picke_lock.acquire()
       try:
           ps = pickle.dumps(s)
       finally:
           picke_lock.release()
       return ps

t1 = threading.Thread(target=do, args =("foo",))
t2 = threading.Thread(target=do, args =("bar",))
p1 = t1.start()
p2 = t2.start()

inpt = raw_input('type anything and click enter... ')

Here is an example using threading.Lock():

import threading
import pickle
picke_lock = threading.Lock()
   def do(s):
       picke_lock.acquire()
       try:
           ps = pickle.dumps(s)
       finally:
           picke_lock.release()
       return ps

t1 = threading.Thread(target=do, args =("foo",))
t2 = threading.Thread(target=do, args =("bar",))
p1 = t1.start()
p2 = t2.start()

inpt = raw_input('type anything and click enter... ')
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文