带有后备实施的包装Python绑定的最佳实践(PEP 518)
让我们考虑以下示例。 有一个静态的C ++库libfoo,分布成CMake子项目。 现在,有一个称为pyfoo的python实现,该实现应为libfoo提供python绑定(使用pybind11), 但是,如果无法构建libfoo 在目标平台上。
这样一个项目的存储库看起来就是这样:
+--+ libFoo
| +--- CMakeLists.txt (export a static library target foo::foo)
| +--- src/..., inc/... (c++ sources for libFoo)
|
+--+ pyFoo
+--- pyproject.toml
+--- ... (other metadata/package files, tests, etc)
+--+ foo (main package dir)
+--+ _native
| +--- CMakeLists.txt (imports libFoo/CMakeLists.txt as a subproject, configures, adds pybind11 module)
| +--- foo_bindings.cpp (literally pybind11 bindings using imported foo::foo)
|
+--- _fallback/... (pure Python implementation of libFoo for compatibility)
+--- __init__.py (essentially "try: from ._native.foo_bindings import *; except ImportError: from ._fallback import *")
现在,这种结构对我来说是一个开发环境(pyfoo/foo/_native
cmake cmake cmake cmake commake complate complate complate foo_bindings to >忍者安装
), 为了创建分布和车轮,这似乎不是处理构建本机(C ++)部分的正确方法。
我对其他软件包如何处理此类情况进行了一些研究,但是我只能找到以下方案:
- 软件包只是一个lib +绑定(没有纯Python实施),因此设置需要本机构建;
- 该软件包将使用CTYPES/CFFI导入外部库(例如Libssl.So),如果导入失败,则将其倒入Python实现,因此从构建系统的角度来看,它是一个纯粹的Python软件包。
我在这里要实现的目标是拥有一个可以在尽可能多的方案中使用的软件包,包括用户可能无法编译C ++源的情况(例如没有MSVC的Windows)和车轮无法用于其平台。
我将如何指定使用尽可能多的现代工具/解决方案(考虑使用PEP 518而不是将所有内容放入设置中), 在保持界面简单清洁的同时?
pip安装pyfoo
的所需决策树是:
- 如果当前平台有轮子,请使用它,
- 如果没有轮子,请尝试构建
foo._native.bindings.bindings
从源(使用CMAKE),包括libfoo
的直接依赖性。- 如果构建成功,一切都还好吧,请在最终设置中使用foo._native.bindings
- 如果构建失败,请不要断开设置,而是安装除
foo._native
package以外的所有内容。
奖励问题:如果foo._fallback
具有附加的软件包依赖项(超出foo
/foo.___native
requience quients ,在pyproject.toml
中列出它们是一个好习惯吗?如果默认实现是foo._native
中的默认实现?
一般指针会没事的。我已经介绍了有关此事的几篇文章,但它们似乎是不完整或基本的。 例如[https://packaging.python.org/en/latest/guides/packaging-binary-extensions/]甚至给出以下示例:
示例:导入DateTime时,python返回DateTime.py模块,如果C实现(_DateTeTimeModule.c)可用。
但是随后无法解决体内的这种情况。
Let's consider the following example.
There's a static C++ library libFoo, distributed as a CMake sub-project.
Now, there's a Python implementation of Foo, called pyFoo, which should provide Python bindings for libFoo (using pybind11),
but does also provide a pure Python implementation of Foo's capabilities, in case libFoo cannot be build
on the target platform.
The repository for such a project would look like this:
+--+ libFoo
| +--- CMakeLists.txt (export a static library target foo::foo)
| +--- src/..., inc/... (c++ sources for libFoo)
|
+--+ pyFoo
+--- pyproject.toml
+--- ... (other metadata/package files, tests, etc)
+--+ foo (main package dir)
+--+ _native
| +--- CMakeLists.txt (imports libFoo/CMakeLists.txt as a subproject, configures, adds pybind11 module)
| +--- foo_bindings.cpp (literally pybind11 bindings using imported foo::foo)
|
+--- _fallback/... (pure Python implementation of libFoo for compatibility)
+--- __init__.py (essentially "try: from ._native.foo_bindings import *; except ImportError: from ._fallback import *")
Now, while this kind of structure worked for me as a development environment (pyFoo/foo/_native
CMake moves compiled foo_bindings into an appropriate place upon ninja install
),
this doesn't seem like a proper way to handle building the native (C++) part for the sake of creating distributions and wheels.
I did some research on how other packages handle these kinds of situations, but I could only find the following scenarios:
- The package is just a lib + binding, (no pure Python implementation), so native build is required for setup;
- The package imports an external library (e.g. libssl.so) using ctypes/cffi and falls back to Python implementation if the import fails, so from the point of view of the build system, it's a pure Python package.
What I'm trying to achieve here, is to have a package that will work in as many scenarios as possible, including cases where users may not be able to compile C++ sources (e.g. Windows without MSVC) and wheels are not available for their platforms.
How would I go about specifying that using as many modern tools/solutions (think using PEP 518 instead of putting everything into setup.py) as possible,
while keeping the interface simple and clean?
A desired decision tree for a pip install pyfoo
would be:
- If there is a wheel for the current platform, use it
- If there is no wheel, try building
foo._native.bindings
from source (using CMake), including direct dependency oflibfoo
.- If the build succeeds, everything's all right, use foo._native.bindings in the final setup
- If the build fails, don't break setup, instead install everything except for the
foo._native
package.
Bonus question: If foo._fallback
has an additional package dependencies (beyond what foo
/foo._native
require), is it a good practice to list them in pyproject.toml
if the default implementation is that in foo._native
?
General pointers would be OK. I've looked through several articles on the matter, but they seem to be incomplete, or rather basic.
For instance [https://packaging.python.org/en/latest/guides/packaging-binary-extensions/] even gives the following example:
Example: When importing datetime, Python falls back to the datetime.py module if the C implementation ( _datetimemodule.c) is not available.
but then fails to address this scenario in the body.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论