Python:在运行一定数量或记录后,寻找一种在异步信号量期间使迭代器休眠 N 秒/分钟的方法

发布于 2025-01-14 21:54:23 字数 1955 浏览 2 评论 0原文

基于下面的代码,在运行循环时触发 N 条记录(即 5 条记录)后(顺序触发(首选)或随机触发顺序都可以),需要在 asyncio 信号量进程方式中休眠一段时间,即 10 秒。我设计的方式是有限并行异步,其中在给定时间处理 n(即 2)条记录。 处理大量数组数据(1000 万)

或者是否有更好的具有可配置值的解决方案,因为我将使用 python 3.6

from time import time
import asyncio

async def square(x, a):
    print(f'start {x}', a)
    await asyncio.sleep(a)
    print(f'end {x}', a)
    return a * a

async def main(limit_number):
    s = asyncio.Semaphore(limit_number)
    async def func(x, a):
        with await s:
            return await square(x, a)
    counter = 0
    tasks = []
    list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    for index, a in enumerate(list):
            if index % 5 == 0 and index > 1:
                print("sleep >>>>>>>>>>> 10s")
                counter +=1
                await asyncio.sleep(10)   ## "need help to correctly pass the sleep to generate below expected output"
            tasks.append(func(counter, a))
    completed, pending = await asyncio.wait(tasks)
    print([t.result() for t in completed])

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(2))
    loop.close()

当前输出

sleep >>>>>>>>>>> 10s
sleep >>>>>>>>>>> 10s
start 6 6
start 1 1
end 6 6
end 1 1
start 7 7
start 2 2
end 7 7
end 2 2
start 8 8
start 3 3
end 8 8
end 3 3
start 9 9
start 0 0
end 9 9
end 0 0
start 4 4
start 10 10
end 4 4
end 10 10
start 5 5
end 5 5
[36, 64, 16, 1, 9, 100, 49, 81, 25, 4, 0]

预期输出

start 0 0
start 0 1
end 0 0
start 0 2
end 0 1
start 0 3
end 0 2
start 0 4
sleep >>>>>>>>>>> 10s
end 0 3
start 1 5
end 0 4
start 1 6
end 1 5
start 1 7
end 1 6
start 1 8
end 1 7
start 1 9
sleep >>>>>>>>>>> 10s
end 1 8
start 2 10
end 1 9
end 2 10
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Based on below code, need help within asyncio Semaphore process way to sleep for certain time i.e. 10 sec after triggering N records (i.e. 5 records) while running the loop (sequential trigger (preferred) or random trigger order is okay). Way I have designed is a limited parallel asyncio where n (i.e. 2) records are process at a given time.
Or is there a better solution with configurable value as I will be processing huge volume of array data (10mil)

using python 3.6

from time import time
import asyncio

async def square(x, a):
    print(f'start {x}', a)
    await asyncio.sleep(a)
    print(f'end {x}', a)
    return a * a

async def main(limit_number):
    s = asyncio.Semaphore(limit_number)
    async def func(x, a):
        with await s:
            return await square(x, a)
    counter = 0
    tasks = []
    list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    for index, a in enumerate(list):
            if index % 5 == 0 and index > 1:
                print("sleep >>>>>>>>>>> 10s")
                counter +=1
                await asyncio.sleep(10)   ## "need help to correctly pass the sleep to generate below expected output"
            tasks.append(func(counter, a))
    completed, pending = await asyncio.wait(tasks)
    print([t.result() for t in completed])

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main(2))
    loop.close()

Current output

sleep >>>>>>>>>>> 10s
sleep >>>>>>>>>>> 10s
start 6 6
start 1 1
end 6 6
end 1 1
start 7 7
start 2 2
end 7 7
end 2 2
start 8 8
start 3 3
end 8 8
end 3 3
start 9 9
start 0 0
end 9 9
end 0 0
start 4 4
start 10 10
end 4 4
end 10 10
start 5 5
end 5 5
[36, 64, 16, 1, 9, 100, 49, 81, 25, 4, 0]

Expected output

start 0 0
start 0 1
end 0 0
start 0 2
end 0 1
start 0 3
end 0 2
start 0 4
sleep >>>>>>>>>>> 10s
end 0 3
start 1 5
end 0 4
start 1 6
end 1 5
start 1 7
end 1 6
start 1 8
end 1 7
start 1 9
sleep >>>>>>>>>>> 10s
end 1 8
start 2 10
end 1 9
end 2 10
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文