将 Python Twisted 与多处理混合？
我需要用Python编写一个类似代理的程序，工作流程与Web代理非常相似。该程序位于客户端和服务器之间，接收客户端发送到服务器的请求，处理该请求，然后将其发送到原始服务器。当然使用的协议是使用TCP的私有协议。
为了尽量减少工作量，我想使用Python Twisted来处理请求接收（该部分充当服务器）和重新发送（该部分充当客户端）。
为了最大限度地提高性能，我想使用 python 多处理（线程有 GIL 限制）将程序分成三个部分（进程）。第一个进程运行 Twisted 来接收请求，将请求放入队列中，并立即将成功返回给原始客户端。第二个进程从队列中获取请求，进一步处理该请求并将其放入另一个队列。第三个进程从第二个队列中获取请求并将其发送到原始服务器。
我是 Python Twisted 的新手，我知道它是事件驱动的，我还听说最好不要将 Twisted 与线程或多处理混合在一起。所以我不知道这种方式是否合适，或者是否有更优雅的方式，仅使用 Twisted ？
I need to write a proxy like program in Python, the work flow is very similar to a web proxy. The program sits in between the client and the server, incept requests sent by the client to the server, process the request, then send it to the original server. Of course the protocol used is a private protocol uses TCP.
To minimize the effort, I want to use Python Twisted to handle the request receiving (the part acts as a server) and resending (the part acts as a client).
To maximum the performance, I want to use python multiprocessing (threading has the GIL limit) to separate the program into three parts (processes). The first process runs Twisted to receive requests, put the request in a queue, and return success immediately to the original client. The second process take request from the queue, process the request further and put it to another queue. The 3rd process take request from the 2nd queue and send it to the original server.
I was a new comer to Python Twisted, I know it is event driven, I also heard it's better to not mix Twisted with threading or multiprocessing. So I don't know whether this way is appropriate or is there a more elegant way by just using Twisted?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
Twisted 有自己的事件驱动方式来运行子进程，这（以我谦虚但正确的观点）比
multiprocessing模块更好。核心 API 是 spawnProcess，但是诸如 ampoule 提供了更高级别的包装器。如果您使用
spawnProcess，您将能够以与处理 Twisted 中任何其他事件相同的方式处理子流程的输出；如果您使用多处理，您将需要开发自己的基于队列的方式，以某种方式将子进程的输出获取到Twisted主循环中，因为正常的callFromThread线程可能使用的 API 无法在另一个进程中工作。根据您的调用方式，它会尝试腌制反应器，或者仅在子流程中使用不同的非工作反应器；无论哪种方式，它都会永远失去你的呼叫。Twisted has its own event-driven way of running subprocesses which is (in my humble, but correct, opinion) better than the
multiprocessingmodule. The core API is spawnProcess, but tools like ampoule provide higher-level wrappers over it.If you use
spawnProcess, you will be able to handle output from subprocesses in the same way you'd handle any other event in Twisted; if you usemultiprocessing, you'll need to develop your own queue-based way of getting output from a subprocess into the Twisted mainloop somehow, since the normalcallFromThreadAPI that a thread might use won't work from another process. Depending on how you call it, it will either try to pickle the reactor, or just use a different non-working reactor in the subprocess; either way it will lose your call forever.安瓿是我在阅读您的问题时首先想到的。它是一个简单的进程池实现，使用 AMP 协议 进行通信。您可以使用 deferToAMPProcess 函数，它非常容易使用。
ampouleis the first thing I think when reading your question.It is a simple process pool implementation which uses the AMP protocol to communicate. You can use the
deferToAMPProcessfunction, it's very easy to use.您可以尝试像 http://us 中所述的协作多任务技术。 pycon.org/2010/conference/schedule/event/73/。这与 Glyph 提到的技术类似，值得一试。
您可以尝试将 ZeroMQ 与 Twisted 结合使用，但目前它非常困难且处于实验阶段:)
You can try something like Cooperative Multitasking technique as it's described there http://us.pycon.org/2010/conference/schedule/event/73/ . It's simillar to technique as Glyph menitioned and it's worth a try.
You can try to use ZeroMQ with Twisted but it's really hard and experimental for now :)