Python:在运行一定数量或记录后,寻找一种在异步信号量期间使迭代器休眠 N 秒/分钟的方法
基于下面的代码,在运行循环时触发 N 条记录(即 5 条记录)后(顺序触发(首选)或随机触发顺序都可以),需要在 asyncio 信号量进程方式中休眠一段时间,即 10 秒。我设计的方式是有限并行异步,其中在给定时间处理 n(即 2)条记录。 处理大量数组数据(1000 万)
或者是否有更好的具有可配置值的解决方案,因为我将使用 python 3.6
from time import time
import asyncio
async def square(x, a):
print(f'start {x}', a)
await asyncio.sleep(a)
print(f'end {x}', a)
return a * a
async def main(limit_number):
s = asyncio.Semaphore(limit_number)
async def func(x, a):
with await s:
return await square(x, a)
counter = 0
tasks = []
list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for index, a in enumerate(list):
if index % 5 == 0 and index > 1:
print("sleep >>>>>>>>>>> 10s")
counter +=1
await asyncio.sleep(10) ## "need help to correctly pass the sleep to generate below expected output"
tasks.append(func(counter, a))
completed, pending = await asyncio.wait(tasks)
print([t.result() for t in completed])
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main(2))
loop.close()
当前输出
sleep >>>>>>>>>>> 10s
sleep >>>>>>>>>>> 10s
start 6 6
start 1 1
end 6 6
end 1 1
start 7 7
start 2 2
end 7 7
end 2 2
start 8 8
start 3 3
end 8 8
end 3 3
start 9 9
start 0 0
end 9 9
end 0 0
start 4 4
start 10 10
end 4 4
end 10 10
start 5 5
end 5 5
[36, 64, 16, 1, 9, 100, 49, 81, 25, 4, 0]
预期输出
start 0 0
start 0 1
end 0 0
start 0 2
end 0 1
start 0 3
end 0 2
start 0 4
sleep >>>>>>>>>>> 10s
end 0 3
start 1 5
end 0 4
start 1 6
end 1 5
start 1 7
end 1 6
start 1 8
end 1 7
start 1 9
sleep >>>>>>>>>>> 10s
end 1 8
start 2 10
end 1 9
end 2 10
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Based on below code, need help within asyncio Semaphore process way to sleep for certain time i.e. 10 sec after triggering N records (i.e. 5 records) while running the loop (sequential trigger (preferred) or random trigger order is okay). Way I have designed is a limited parallel asyncio where n (i.e. 2) records are process at a given time.
Or is there a better solution with configurable value as I will be processing huge volume of array data (10mil)
using python 3.6
from time import time
import asyncio
async def square(x, a):
print(f'start {x}', a)
await asyncio.sleep(a)
print(f'end {x}', a)
return a * a
async def main(limit_number):
s = asyncio.Semaphore(limit_number)
async def func(x, a):
with await s:
return await square(x, a)
counter = 0
tasks = []
list = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
for index, a in enumerate(list):
if index % 5 == 0 and index > 1:
print("sleep >>>>>>>>>>> 10s")
counter +=1
await asyncio.sleep(10) ## "need help to correctly pass the sleep to generate below expected output"
tasks.append(func(counter, a))
completed, pending = await asyncio.wait(tasks)
print([t.result() for t in completed])
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main(2))
loop.close()
Current output
sleep >>>>>>>>>>> 10s
sleep >>>>>>>>>>> 10s
start 6 6
start 1 1
end 6 6
end 1 1
start 7 7
start 2 2
end 7 7
end 2 2
start 8 8
start 3 3
end 8 8
end 3 3
start 9 9
start 0 0
end 9 9
end 0 0
start 4 4
start 10 10
end 4 4
end 10 10
start 5 5
end 5 5
[36, 64, 16, 1, 9, 100, 49, 81, 25, 4, 0]
Expected output
start 0 0
start 0 1
end 0 0
start 0 2
end 0 1
start 0 3
end 0 2
start 0 4
sleep >>>>>>>>>>> 10s
end 0 3
start 1 5
end 0 4
start 1 6
end 1 5
start 1 7
end 1 6
start 1 8
end 1 7
start 1 9
sleep >>>>>>>>>>> 10s
end 1 8
start 2 10
end 1 9
end 2 10
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论