Python 将 std(in/out/err) 流式传输到另一个进程

发布于 2024-11-30 17:54:01 字数 1112 浏览 1 评论 0原文

我有一个“在后台”运行的 python 进程。这是一个以完全不同的方式启动的 subprocess.Popen() 调用过程。正如我到目前为止所实现的那样，这两个进程正在使用多处理.connections.(Listener|Client) 相互通信。但我愿意以任何方式来完成以下任务。

我想做的是让“监听器”后台进程使用 std(in|out|err) ，就好像它们来自“客户端”进程一样。流式处理很重要，这样 io 就不会被读入内存，而是可以高效地读写，同时还能避免侦听器进程的阻塞。我想这意味着我需要较低级别的访问或某种异步/线程/多处理解决方案。这就是我不清楚如何做。例如：

while True:  # TODO
    conn = listener.accept()
    try:
        callable_, args, kw = conn.recv()
        callable_(*args, **kw)

问题是如何允许 callable_ 高效且无阻塞地执行到 std(in|out|err) 的流式传输。监听器也是长时间运行的，我试图支持任意 python 代码，在监听器模块导入时 from sys import std(in|out|err) 仍然会看到 std(in|err) 。 out|err) 就像来自客户端一样。我可以在导入任何其他内容之前在侦听器启动时进行设置（修改/替换 sys.std(in|out|err)？）。

有没有一种好的方法可以透明地传输此 io，以便在初始设置后，侦听器进程运行的代码可以是任意的，并且可以正常处理 std(in|out|err) 而不会阻塞等？最好不要创建新进程，因为目标是减少进程开销，但在客户端使用多个进程更好，因为客户端很瘦。同样，如果我必须循环读取/写入 std(in|out|err) 中的字节段以实现有效的流处理，那么最好在客户端中执行此操作。

原文

I have a python process running "in the background". It's a subprocess.Popen() call started in a totally different process. As I've implemented it so far, the two processes are using multiprocessing.connections.(Listener|Client) to communicate with eachother. But I'm open to any way to accomplish the following.

What I want to do is have the "listener" background process use std(in|out|err) as if they've come from the "client" process. Streaming is important so that io isn't read into memory but is read and written efficiently but also to avoid blocking on the listener process. I guess that means I need a lower-level access or some sort of async/threading/multiprocessing solution. That's what I'm unclear on how do to. For example:

while True:  # TODO
    conn = listener.accept()
    try:
        callable_, args, kw = conn.recv()
        callable_(*args, **kw)

The question is how to allow callable_ to execute streaming to/from std(in|out|err) efficiently and without blocking. The listener is also long-running, and I'm trying to support arbitrary python code that does from sys import std(in|out|err) at listener module import time will still see std(in|out|err) as if from the client. I can do setup (modifying/replacing sys.std(in|out|err)?) at listener startup before anything else is impored.

Is there a good way to stream this io transparently such that after initial setup the code being run by the listener process can be arbitrary and can treat std(in|out|err) normally without blocking, etc.? Preferably without creating new processes since the goal is to reduce process overhead, though using multiple processes on the client end is preferable since the client is thin. Similarly, if I have to loop over reads/writes of segments of bytes from std(in|out|err) to accomplish effective streaming it would be best to do it in the client.

分享到QQ

分享到微博