如何使用 Google App Engine 管理第三方 Python 库？（virtualenv？pip？）

发布于 2024-10-15 15:10:39 字数 1237 浏览 1 评论 0原文

使用 Google App Engine 管理第三方 Python 库的最佳策略是什么？

假设我想使用 Flask，一个 Web 应用程序框架。博客条目表示这样做，这似乎不对：

$ cd /tmp/
$ wget http://pypi.python.org/packages/source/F/Flask/Flask-0.6.1.tar.gz
$ tar zxf Flask-0.6.1.tar.gz
$ cp -r Flask-0.6.1/flask ~/path/to/project/
(... repeat for other packages ...)

必须有更好的方法来管理第三方代码，特别是如果我想跟踪版本、测试升级或者两个库共享一个子目录。我知道 Python 可以从 zip 文件导入模块，并且 pip 可以处理出色的 REQUIREMENTS 文件，而且我已经看到 pip 有一个 zip 与 GAE 一起使用的命令。

（注意：有一些类似的问题 - 1、2、3，4，5 — 但它们是针对具体情况的，并不能真正回答我的问题。）

原文

What's the best strategy for managing third-party Python libraries with Google App Engine?

Say I want to use Flask, a webapp framework. A blog entry says to do this, which doesn't seem right:

$ cd /tmp/
$ wget http://pypi.python.org/packages/source/F/Flask/Flask-0.6.1.tar.gz
$ tar zxf Flask-0.6.1.tar.gz
$ cp -r Flask-0.6.1/flask ~/path/to/project/
(... repeat for other packages ...)

There must be a better way to manage third-party code, especially if I want to track versions, test upgrades or if two libraries share a subdirectory. I know that Python can import modules from zipfiles and that pip can work with a wonderful REQUIREMENTS file, and I've seen that pip has a zip command for use with GAE.

(Note: There's a handful of similar questions — 1, 2, 3, 4, 5 — but they're case-specific and don't really answer my question.)

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

雨后彩虹 2024-10-22 15:10:39

我是这样做的：

项目
- .Python
- 垃圾箱
- 库
  - python2.5
    - 站点包
      - < pip 在此处安装软件包 >
- 包括
- 源代码
  - app.yaml
  - index.yaml
  - main.yaml
  - <符号链接 ../lib/python2.5/site-packages 中的 pip 安装包

project 目录是 virtualenv 所在的顶级目录。我使用以下命令获取 virtualenv：

cd project
virtualenv -p /usr/bin/python2.5 --no-site-packages --distribute .

src 目录是所有代码所在的位置。当您将代码部署到 GAE 时，*仅*将代码部署在 src 目录中，而不是其他任何内容。 appcfg.py 将解析符号链接并将库文件复制到 GAE。

我不会将库安装为 zip 文件，主要是为了方便，以防我需要阅读源代码，而出于好奇，我经常这样做。但是，如果您确实想压缩库，请将以下代码片段放入 main.py 中，

import sys
for p in ['librarie.zip', 'package.egg'...]:
    sys.path.insert(0, p)

之后您可以像往常一样导入压缩包。

需要注意的一件事是 setuptools 的 pkg_resources.py。我将其直接复制到我的 src 目录中，以便我的其他符号链接包可以使用它。留意任何使用 entry_point 的内容。就我而言，我使用的是 Toscawidgets2，我必须深入研究源代码来手动连接各个部分。如果您有很多依赖 entry_point 的库，这可能会变得很烦人。

Here's how I do it:

project
- .Python
- bin
- lib
  - python2.5
    - site-packages
      - < pip install packages here >
- include
- src
  - app.yaml
  - index.yaml
  - main.yaml
  - < symlink the pip installed packages in ../lib/python2.5/site-packages

The project directory is the top level directory where the virtualenv sits. I get the virtualenv using the following commands:

cd project
virtualenv -p /usr/bin/python2.5 --no-site-packages --distribute .

The src directory is where all your code goes. When you deploy your code to GAE, *only* deploy those in the src directory and nothing else. The appcfg.py will resolve the symlinks and copy the library files to GAE for you.

I don't install my libraries as zip files mainly for convenience in case I need to read the source code, which I happen to do a lot just out of curiosity. However, if you really want to zip the libraries, put the following code snippet into your main.py

import sys
for p in ['librarie.zip', 'package.egg'...]:
    sys.path.insert(0, p)

After this you can import your zipped up packages as usual.

One thing to watch out for is setuptools' pkg_resources.py. I copied that directly into my src directory so my other symlinked packages can use it. Watch out for anything that uses entry_points. In my case I'm using Toscawidgets2 and I had to dig into the source code to manually wire up the pieces. It can become annoying if you had a lot of libraries that rely on entry_point.

回复收藏 0 原文

望笑 2024-10-22 15:10:39

简单地说：

$ pip install -r requirements.txt -t <your_app_directory/lib>

创建/编辑 /appengine_config.py：

"""This file is loaded when starting a new application instance."""
import sys
import os.path

# add `lib` subdirectory to `sys.path`, so our `main` module can load
# third-party libraries.
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))

更新：

Google 将其示例更新为 appengine_config.py，例如：

    from google.appengine.ext import vendor
    vendor.add('lib')

注意：即使他们的示例有 .gitignore 忽略 lib/ 目录，您仍然需要将该目录保留在源代码下控制是否使用 git-push 部署方法。

What about simply:

$ pip install -r requirements.txt -t <your_app_directory/lib>

Create/edit <your_app_directory>/appengine_config.py:

"""This file is loaded when starting a new application instance."""
import sys
import os.path

# add `lib` subdirectory to `sys.path`, so our `main` module can load
# third-party libraries.
sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))

UPDATE:

Google updated their sample to appengine_config.py, like:

    from google.appengine.ext import vendor
    vendor.add('lib')

Note: Even though their example has .gitignore ignoring lib/ directory you still need to keep that directory under source control if you use git-push deployment method.

回复收藏 0 原文

墨小沫ゞ 2024-10-22 15:10:39

我更喜欢构建。

您可以在项目中的 setup.py 或 buildout.cfg 中设置依赖项，固定 buildout.cfg 中的版本，并指定哪些包在 GAE 上不可用且应包含在 packages.zip 中。 rod.recipe.appengine会将所需的包复制到packages.zip中，只要将packages.zip插入sys.path中，就可以将它们导入到任何地方。

如果您需要的包不在 pypi 上

find-links =
    https://github.com/tesdal/pusher_client_python/tarball/rewrite#egg=pusher-2.0dev2

[versions]
pusher = 2.0dev2

，并且所有这些设置和依赖项都在 git 中进行版本控制，您也可以使用来自 github 的 forks。

您不必想知道您的源代码树中当前包含哪一个 Flask 副本，并且可能复制到您的版本控制中（或者需要新开发人员手动解压和升级），您只需检查 buildout.cfg 中的版本即可。如果您想要新版本，请更改 buildout.cfg 并重新运行 buildout。

您还可以使用它将变量插入到配置文件模板中，例如如果您有带有 staging.cfg 的临时服务器，则在 app.yaml 中设置 appspot id 和版本等。

I prefer buildout.

You set up dependencies in setup.py in your project or buildout.cfg, pin the versions in buildout.cfg, and specify which packages are not available on GAE and should be included in packages.zip. rod.recipe.appengine will copy required packages into packages.zip, and as long as you insert packages.zip into the sys.path, they can be imported anywhere.

You can also use forks from github if the package you need is not on pypi

find-links =
    https://github.com/tesdal/pusher_client_python/tarball/rewrite#egg=pusher-2.0dev2

[versions]
pusher = 2.0dev2

and all of these settings and dependencies are versioned in git.

Instead of wondering which copy of Flask is currently included in your source tree and perhaps copied into your version control (or requiring new developers to manually unpack and upgrade), you simply check the version in buildout.cfg. If you want a new version, change buildout.cfg and rerun buildout.

You can also use it to insert variables into config file templates, like setting the appspot id and version in app.yaml if you have staging server with staging.cfg and so on.

回复收藏 0 原文

紙鸢 2024-10-22 15:10:39

我最近为此创建了一个名为 gaenv 的工具。它遵循requirements.txt格式，但不安装它，您可以使用pip install -rrequirements.txt安装然后运行命令行工具gaenv。

$ pip install -r requirements.txt
$ gaenv

这会自动创建符号链接，您也可以在 virtualenv 中安装 gaenv 并从那里运行二进制文件。
这是一篇关于它的博客文章：

http:// /blog.altlimit.com/2013/06/google-app-engine-virtualenv-tool-that.html

也在 github 上

https://github.com/faisalraja/gaenv

I recently created a tool for this called gaenv. It follows a requirements.txt format, but doesn't install it, you can install with pip install -r requirements.txt then run the command line tool gaenv.

$ pip install -r requirements.txt
$ gaenv

This creates symlinks automatically, you could install gaenv in your virtualenv too and run the binary from there.
Here is a blog post about it:

http://blog.altlimit.com/2013/06/google-app-engine-virtualenv-tool-that.html

also on github

https://github.com/faisalraja/gaenv

回复收藏 0 原文

江南烟雨〆相思醉 2024-10-22 15:10:39

（2021 年 6 月） 这篇文章已有十多年的历史，因此现在有必要更新答案。

Python 3：在 requirements.txt 中列出 3P 库以及任何所需的版本号；它们将在部署时由 Google 自动安装。（如果您决定将应用迁移到 Google Cloud Functions 或 Cloud Run。）
Python 2 不带内置 3P 库（常规 3P 库）：

如上所述创建 requirements.txt
安装/自我捆绑/在本地复制它们，例如通过 pip install -t lib -rrequirements.txt< /code>
创建 appengine_config.py，如本页第 5 步

Python 2 与内置 3P 库（特殊的 3P 库集）：

上面链接的所有列出的 3P 库都是“内置的 - in，”意味着它们在 App Engine 服务器上可用，因此您不必将它们与您的应用程序一起复制/自行捆绑（如上面的#2 所示），
只需在 中列出它们并提供可用版本即可库：app.yaml 的 部分像这样
（不要将内置库放入 requirements.txt 中，也不要使用 pip install 在本地安装它们，除非您想自行捆绑，因为，例如您需要较新版本的内置库。）
像上面一样创建 appengine_config.py。

如果您的 Python 2 应用程序同时具有内置和非内置 3P 库，请使用上面 #2 和 #3 中的技术（内置库位于 app. yaml 和 requirements.txt 中的非内置库并运行上面的 pip install cmd）。第二代运行时（如 Python 3）的改进之一是所有这些带有 3P 库的游戏都会神奇地消失（参见上面的#1）。

示例：Flask

Flask 是一个第三方微型 Web 框架，对于这个特定问题来说，这是一个有趣的案例。对于 Python 3，它们都进入 requirements.txt 中，因此您只需将 flask 添加到该文件即可。（只需从那里部署即可。）

对于 Python 2，它甚至更有趣，因为它是一个内置库。不幸的是，App Engine 服务器上的版本是 0.12。当我们现在处于/超越2.0.3时，谁想使用那个？！？因此，您不必像其他内置库一样将其放入 app.yaml 中，而是假装内置版本不存在并将其放入 requirements.txt 中然后运行 pip2 install -t lib -rrequirements.txt 将其与您的应用程序代码捆绑/供应商。（但是，Python 2 的最终版本是 1.1.4，所以这就是安装的版本。）

(Jun 2021) This post is over a decade old, and so an updated answer is warranted now.

Python 3: list 3P libraries in requirements.txt along with any desired version#s; they'll be automatically installed by Google upon deployment. (This is the same technique used if you decide to migrate your app to Google Cloud Functions or Cloud Run.)
Python 2 without built-in 3P libraries (regular 3P libraries):

Create requirements.txt as above
Install/self-bundle/copy them locally, say to lib, via pip install -t lib -r requirements.txt
Create appengine_config.py as shown in step 5 on this page

Python 2 with built-in 3P libraries (special set of 3P libraries):

All listed 3P libraries linked above are "built-in," meaning they're available on App Engine servers so you don't have to copy/self-bundle them w/your app (like in #2 above)
It suffices to list them with an available version in the libraries: section of your app.yaml like this
(Don't put built-in libraries in requirements.txt nor use pip install to install them locally unless you want to self-bundle because, say if you need a newer version of the built-in library.)
Create appengine_config.py like the above.

If you have a Python 2 app with both built-in and non-built-in 3P libraries, use the techniques in both #2 and #3 above (built-in libraries in app.yaml and non-built-in libraries in requirements.txt and run the pip install cmd above). One of the improvements in the second generation runtimes like Python 3 is that all these games with 3P libraries go away magically (see #1 above).

Example: Flask

Flask is a 3rd-party micro web framework, and it's an interesting case for this specific question. For Python 3, they all go into requirements.txt, so you'd just add flask to that file, and you're done. (Just deploy from there.)

For Python 2, it's even more interesting because it's a built-in library. Unfortunately, the version on App Engine servers is 0.12. Who wants to use that when we're at/beyond 2.0.3 now?!? So instead of putting it in app.yaml like other built-in libraries, you'd pretend the built-in version doesn't exist and put it in requirements.txt then run pip2 install -t lib -r requirements.txt to bundle/vendor it with your application code. (However, the final version for Python 2 is 1.1.4, so that's what gets installed.)

回复收藏 0 原文

煮茶煮酒煮时光 2024-10-22 15:10:39

Wernight 的解决方案最接近官方 Flask 示例应用，我已经通过将 sys.path.insert() 调用更改为 site.addsitedir() 以允许命名空间包通过处理其附带的.pth文件（这对于像Pyramid这样的框架很重要）。

到目前为止一切顺利，但是将目录附加到路径中，因此失去了用新版本覆盖所包含的库（例如WebOb和requests）的机会。

那么 appengine_config.py 中需要的内容（我也试图让官方存储库接受此更改）如下：

"""This file is loaded when starting a new application instance."""
import os.path
import site.addsitedir
import sys.path

dirname = 'lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)

# split path after 1st element ('.') so local modules are always found first
sys.path, remainder = sys.path[:1], sys.path[1:]

# add `lib` subdirectory as a site directory, so our `main` module can load
# third-party libraries.
site.addsitedir(dirpath)

# append the rest of the path
sys.path.extend(remainder)

此代码的最终版本可能最终隐藏在 < code>vendor.py 模块，并像 insertsitedir(index, path) 或其他变体那样调用，如您所见在参加此拉取请求的讨论中，但逻辑或多或少是如何工作的，以允许简单的 pip install -rrequirements.txt -t lib/ 适用于所有包，包括命名空间包，并且仍然允许用新版本覆盖包含的库，正如我到目前为止所无法找到更简单的替代方案< /a>.

Wernight's solution is the closest to current practice in the official Flask example app, which I've already improved by changing the sys.path.insert() call to site.addsitedir() in order to allow for namespace packages by processing their attendant .pth files (which are important for frameworks like Pyramid).

So far so good, but that appends the directory to the path, and so loses the opportunity to override the included libraries (like WebOb and requests) with newer versions.

What is needed then in appengine_config.py (and I am trying to get this change accepted into the official repos as well) is the following:

"""This file is loaded when starting a new application instance."""
import os.path
import site.addsitedir
import sys.path

dirname = 'lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)

# split path after 1st element ('.') so local modules are always found first
sys.path, remainder = sys.path[:1], sys.path[1:]

# add `lib` subdirectory as a site directory, so our `main` module can load
# third-party libraries.
site.addsitedir(dirpath)

# append the rest of the path
sys.path.extend(remainder)

The final version of this code may end up hidden away in a vendor.py module and called like insertsitedir(index, path) or some other variation, as you can see in the discussion attending this pull request, but the logic is more or less how it will work regardless, to allow a simple pip install -r requirements.txt -t lib/ to work for all packages including namespace ones, and to still allow overriding the included libraries with new versions, as I have so far been unable to find a simpler alternative.

回复收藏 0 原文