Python 生成器预取?

发布于 2024-12-03 08:41:27 字数 287 浏览 2 评论 0原文

我有一个生成器,每次迭代都需要很长时间才能运行。是否有一种标准方法让它产生一个值,然后在等待再次调用时生成下一个值?

每次在 GUI 中按下按钮时都会调用生成器,并且用户将在每次按下按钮后考虑结果。

编辑:解决方法可能是:

def initialize():
    res = next.gen()

def btn_callback()
    display(res)
    res = next.gen()
    if not res:
       return

I have a generator that takes a long time for each iteration to run. Is there a standard way to have it yield a value, then generate the next value while waiting to be called again?

The generator would be called each time a button is pressed in a gui and the user would be expected to consider the result after each button press.

EDIT: a workaround might be:

def initialize():
    res = next.gen()

def btn_callback()
    display(res)
    res = next.gen()
    if not res:
       return

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(4

甲如呢乙后呢 2024-12-10 08:41:27

如果我想做像你的解决方法这样的事情,我会编写一个这样的类:

class PrefetchedGenerator(object):
    def __init__(self, generator):
         self._data = generator.next()
         self._generator = generator
         self._ready = True

    def next(self):
        if not self._ready:
            self.prefetch()
        self._ready = False
        return self._data

    def prefetch(self):
        if not self._ready:
            self._data = self._generator.next()
            self._ready = True

它比你的版本更复杂,因为我这样做是为了让它处理不调用预取或调用预取太多次的情况。基本思想是当您想要下一个项目时调用 .next() 。当你有“时间”可以消磨时,你可以调用预取。

您的另一个选择是线程。

class BackgroundGenerator(threading.Thread):
    def __init__(self, generator):
        threading.Thread.__init__(self)
        self.queue = Queue.Queue(1)
        self.generator = generator
        self.daemon = True
        self.start()

    def run(self):
        for item in self.generator:
            self.queue.put(item)
        self.queue.put(None)

    def next(self):
            next_item = self.queue.get()
            if next_item is None:
                 raise StopIteration
            return next_item

这将与您的主应用程序分开运行。无论获取每次迭代需要多长时间,您的 GUI 都应该保持响应。

If I wanted to do something like your workaround, I'd write a class like this:

class PrefetchedGenerator(object):
    def __init__(self, generator):
         self._data = generator.next()
         self._generator = generator
         self._ready = True

    def next(self):
        if not self._ready:
            self.prefetch()
        self._ready = False
        return self._data

    def prefetch(self):
        if not self._ready:
            self._data = self._generator.next()
            self._ready = True

It is more complicated than your version, because I made it so that it handles not calling prefetch or calling prefetch too many times. The basic idea is that you call .next() when you want the next item. You call prefetch when you have "time" to kill.

Your other option is a thread..

class BackgroundGenerator(threading.Thread):
    def __init__(self, generator):
        threading.Thread.__init__(self)
        self.queue = Queue.Queue(1)
        self.generator = generator
        self.daemon = True
        self.start()

    def run(self):
        for item in self.generator:
            self.queue.put(item)
        self.queue.put(None)

    def next(self):
            next_item = self.queue.get()
            if next_item is None:
                 raise StopIteration
            return next_item

This will run separately from your main application. Your GUI should remain responsive no matter how long it takes to fetch each iteration.

友欢 2024-12-10 08:41:27

不。生成器不是异步的。这不是多处理。

如果您想避免等待计算,您应该使用multiprocessing包,以便独立的进程可以完成昂贵的计算。

您需要一个单独的过程来计算和排队结果。

然后,您的“生成器”可以简单地将可用结果出列。

No. A generator is not asynchronous. This isn't multiprocessing.

If you want to avoid waiting for the calculation, you should use the multiprocessing package so that an independent process can do your expensive calculation.

You want a separate process which is calculating and enqueueing results.

Your "generator" can then simply dequeue the available results.

节枝 2024-12-10 08:41:27

您绝对可以使用生成器来做到这一点,只需创建生成器,以便每个 next 调用在获取下一个值和通过放入多个 yield 语句返回它之间交替。下面是一个示例:

import itertools, time

def quick_gen():
    counter = itertools.count().next
    def long_running_func():
        time.sleep(2)
        return counter()
    while True:
        x = long_running_func()
        yield
        yield x

>>> itr = quick_gen()
>>> itr.next()   # setup call, takes two seconds
>>> itr.next()   # returns immediately
0
>>> itr.next()   # setup call, takes two seconds
>>> itr.next()   # returns immediately
1

请注意,生成器不会自动执行获取下一个值的处理,而是由调用者为每个值调用 next 两次。对于您的用例,您将调用 next 一次作为设置,然后每次用户单击按钮时,您将显示生成的下一个值,然后再次调用 next预取。

You can definitely do this with generators, just create your generator so that each next call alternates between getting the next value and returning it by putting in multiple yield statements. Here is an example:

import itertools, time

def quick_gen():
    counter = itertools.count().next
    def long_running_func():
        time.sleep(2)
        return counter()
    while True:
        x = long_running_func()
        yield
        yield x

>>> itr = quick_gen()
>>> itr.next()   # setup call, takes two seconds
>>> itr.next()   # returns immediately
0
>>> itr.next()   # setup call, takes two seconds
>>> itr.next()   # returns immediately
1

Note that the generator does not automatically do the processing to get the next value, it is up to the caller to call next twice for each value. For your use case you would call next once as a setup up, and then each time the user clicks the button you would display the next value generated, then call next again for the pre-fetch.

爱人如己 2024-12-10 08:41:27

我在追求类似的东西。我希望当后台线程处理下一个时,yield 能够快速返回一个值(如果可以的话)。

import Queue
import time
import threading

class MyGen():
    def __init__(self):
        self.queue = Queue.Queue()
        # Put a first element into the queue, and initialize our thread
        self.i = 1
        self.t = threading.Thread(target=self.worker, args=(self.queue, self.i))
        self.t.start()

    def __iter__(self):
        return self

    def worker(self, queue, i):
        time.sleep(1) # Take a while to process
        queue.put(i**2)

    def __del__(self):
        self.stop()

    def stop(self):
        while True: # Flush the queue
            try:
                self.queue.get(False)
            except Queue.Empty:
                break
        self.t.join()

    def next(self):
        # Start a thread to compute the next next.
        self.t.join()
        self.i += 1
        self.t = threading.Thread(target=self.worker, args=(self.queue, self.i))
        self.t.start()

        # Now deliver the already-queued element
        while True:
            try:
                print "request at", time.time()
                obj = self.queue.get(False)
                self.queue.task_done()
                return obj
            except Queue.Empty:
                pass
            time.sleep(.001)

if __name__ == '__main__':
    f = MyGen()
    for i in range(5):
#        time.sleep(2) # Comment out to get items as they are ready
        print "*********"
        print f.next()
        print "returned at", time.time()

上面的代码给出了以下结果:

*********
request at 1342462505.96
1
returned at 1342462505.96
*********
request at 1342462506.96
4
returned at 1342462506.96
*********
request at 1342462507.96
9
returned at 1342462507.96
*********
request at 1342462508.96
16
returned at 1342462508.96
*********
request at 1342462509.96
25
returned at 1342462509.96

I was after something similar. I wanted yield to quickly return a value (if it could) while a background thread processed the next, next.

import Queue
import time
import threading

class MyGen():
    def __init__(self):
        self.queue = Queue.Queue()
        # Put a first element into the queue, and initialize our thread
        self.i = 1
        self.t = threading.Thread(target=self.worker, args=(self.queue, self.i))
        self.t.start()

    def __iter__(self):
        return self

    def worker(self, queue, i):
        time.sleep(1) # Take a while to process
        queue.put(i**2)

    def __del__(self):
        self.stop()

    def stop(self):
        while True: # Flush the queue
            try:
                self.queue.get(False)
            except Queue.Empty:
                break
        self.t.join()

    def next(self):
        # Start a thread to compute the next next.
        self.t.join()
        self.i += 1
        self.t = threading.Thread(target=self.worker, args=(self.queue, self.i))
        self.t.start()

        # Now deliver the already-queued element
        while True:
            try:
                print "request at", time.time()
                obj = self.queue.get(False)
                self.queue.task_done()
                return obj
            except Queue.Empty:
                pass
            time.sleep(.001)

if __name__ == '__main__':
    f = MyGen()
    for i in range(5):
#        time.sleep(2) # Comment out to get items as they are ready
        print "*********"
        print f.next()
        print "returned at", time.time()

The code above gave the following results:

*********
request at 1342462505.96
1
returned at 1342462505.96
*********
request at 1342462506.96
4
returned at 1342462506.96
*********
request at 1342462507.96
9
returned at 1342462507.96
*********
request at 1342462508.96
16
returned at 1342462508.96
*********
request at 1342462509.96
25
returned at 1342462509.96
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文