Python：使用子进程流数据而不会出现死锁？
我正在编写一个小脚本来整理大量数据。事情是这样的：
outproc = None
for input in input_files:
p = Popen('process_input "%s" | more_input_processing' %(input, ),
shell=True, stdout=PIPE)
for line in p.stdout.xreadlines():
if linecount % 1000000 == 0:
outfile = "output%03d" %(linecount // 1000000, )
if outproc:
outproc.stdin.close()
result = outproc.wait() # <-- deadlock here
assert result == 0, "outproc exited with %s" %(result, )
outproc = Popen('handle_output "%s"' %(outfile, ),
shell=True, stdin=PIPE)
linecount += 1
outproc.stdin.write(line)
p.stdout.close()
result = p.wait()
assert result == 0, "p exited with %s" %(result, )
不过，正如文档警告的那样，当我尝试等待 outproc 时，我遇到了僵局（请参阅评论）。
文档提出的“解决方案”是使用 .communicate()...但是这样做会涉及在刷新之前将所有输入读取到内存中，这是不可取的。
那么，如何在子进程之间传输数据而不出现死锁呢？
I'm writing a little script for shuffling a lot of data around. It's something like this:
outproc = None
for input in input_files:
p = Popen('process_input "%s" | more_input_processing' %(input, ),
shell=True, stdout=PIPE)
for line in p.stdout.xreadlines():
if linecount % 1000000 == 0:
outfile = "output%03d" %(linecount // 1000000, )
if outproc:
outproc.stdin.close()
result = outproc.wait() # <-- deadlock here
assert result == 0, "outproc exited with %s" %(result, )
outproc = Popen('handle_output "%s"' %(outfile, ),
shell=True, stdin=PIPE)
linecount += 1
outproc.stdin.write(line)
p.stdout.close()
result = p.wait()
assert result == 0, "p exited with %s" %(result, )
As the documentation warns, though, I'm hitting a deadlock when I try to wait for outproc (see comment).
The “solution” proposed by the documentation is to use .communicate()… But doing that would involve reading all the input into memory before flushing it, which is undesirable.
So, how can I stream data between subprocesses without deadlocks?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
您没有在子进程实际读取的管道上使用
close，因此它不会收到 SIGPIPE 或任何导致其退出的内容。当您有足够的数据时，只需终止该进程即可。或者，通过管道输入和输出并使用 select 来知道何时应该读取或写入。You're not using
closeon a pipe the subprocess actually reads, so it won't receive SIGPIPE or anything to cause it to exit. Just kill the process when you have enough data. Alternatively, pipe both input and output and use select to know when you should read or write.