stackless python 如何能够快速实现并发?

发布于 2024-08-13 05:06:52 字数 204 浏览 5 评论 0原文

stackless python 没有很好地利用多核,那么它​​应该比 python 线程/多处理更快的点在哪里?

所有基准测试都使用 stackless python tasklet 与 python 线程锁和队列进行比较,这是不公平的,因为锁总是效率很低。

如果使用单线程函数调用没有锁,它应该与 stackless python 一样高效

stackless python didn't take a good usage of multi-core, so where is the point it should be faster than python thread/multiprocessing ?

all the benchmark use stackless python tasklet to compare with python thread lock and queue, that's unfair, cause lock always has low efficiency

see, if use single thread function call without lock it should be as efficient as stackless python

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

看海 2024-08-20 05:06:52

首先关注功能,然后才是性能(除非您知道自己有需要)。

服务器上的大部分时间都花在 I/O 上,因此多核并没有多大帮助。如果您正在使用的主要 I/O,多线程 python 可能是最简单的答案。

如果服务器请求是 CPU 密集型的,那么拥有一个父进程(无论是否是多线程)和相应的子进程确实很有意义。

如果你真的想扩展,你可以考虑不同的平台,比如 Erlang。如果你确实想扩展并仍然使用 python,你可以考虑分布式 erlang,其中 Python 进程作为分布式集群上的 Erlang 端口进行管理。

有很多选择,但除非您正在处理一些的事情,否则您很可能会采取一种简单的方法。

尽早发布,经常发布。

Focus on functionality first, and performance second (unless you know you have the need).

Most of the time on a server is spent with I/O, so multi-cores do not help so much. If it is mostly I/O that you are working with, multi-threading python may be the simplest answer.

If the server requests are CPU intensive, then having a parent process (be it multi-threaded or not), and respective child processes does make a good bit of sense.

If you really want to scale, you could look at a different platform, like Erlang. If you really want to scale and still use python, you could look at distributed erlang with Python processes managed as Erlang ports on a distributed cluster.

Lots of options, but unless you are dealing with someting big big, you could most likely take a simple approach.

release early, release often.

烟雨扶苏 2024-08-20 05:06:52

有一种新的流行的东西,称为异步 IO 循环和消息传递并发以及其他一些流行术语。嗯,它一点也不新鲜,但只是最近 5 年才被主流发现。

Stackless Python 是 Python 的一个版本,其中 VM 本身已被修改以更好地支持这些消息传递和 IO 循环,其技巧是绿色线程/协程。

还有其他库可以使用不同的工具执行相同的操作,例如 Python 上的 Twisted 和 Tornado。您甚至可以在 Stackless Python 等上运行混合 Twisted。

IO 循环位直接映射到 Berkley 套接字如何执行异步 IO,并且只需付出一些努力就可以扩展为主动而非被动,并与文件系统和网络套接字一起使用,例如最新的 libevent。

要横向扩展以利用多个核心,您有两种方法:多线程;共享状态,例如线程或进程之间 - 多处理例如消息队列。当前架构的一个普遍限制是,线程方法对于本地的大量核心来说效果很好,而当核心数量变得巨大或者这些核心位于不同的机器上时,消息传递就会在性能方面超过性能。您可以采取混合方法。

由于 Python VM 中的内部设计选择,它在多线程方面通常不如多处理那么高效,因此您比在其他平台上更快地进入多个进程并进行消息传递。

但通常消息传递方法是一个更干净、容易正确的版本。

还有其他语言基于相同的方法构建,具有不同的附加目标和约束,例如 Erlang、node.js、Clojure、Go。

其中,Clojure 可能是信息最丰富的。当您了解 Clojure 的运作方式并思考原因时,其他系统的整体目标和约束就会落实到位......

There is this new and trendy thing called asynchronous-IO-loops and message-passing-concurrency and a few other trendy terms. Well, its not at all new, but it is only just these last 5 years being discovered by the mainstream.

Stackless Python is a version of Python where the VM has itself been modified to better support these message passing and IO loops, and its trick is green threading / coroutines.

There are other libraries for doing the same with different tools, e.g. Twisted and Tornado, on Python. You can even run hybrid Twisted on Stackless Python and so on.

The IO loop bit maps directly to how Berkley sockets do asynchronous IO, and with a bit of effort can be extended to be proactive rather than reactive and work with file systems as well as network sockets, e.g. the newest libevent.

To scale sideways to utilise more than one core is where you have two approaches - multithreading; shared state e.g. threads or between processes - multiprocessing e.g. message queues. It is a general limitation of current architectures that the threads approach works well for a large number of cores locally, whereas message passing overtakes performance-wise as the number of cores becomes massive or if those cores are on different machines. And you can make a hybrid approach.

Because of internal design choices in the Python VM, it is generally not as efficient at multi-threading as multi-processing, so you go to multiple processes with message passing sooner than you might on other platforms.

But generally the message passing approach is a cleaner, easily correct version.

And there are other languages that build on this same approach with different additional aims and constraints e.g. Erlang, node.js, Clojure, Go.

Of these, Clojure is perhaps the most informative. When you understand how Clojure ticks, and think through the whys, the whole aims and constraints of the other systems will fall into place...

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文