当前位置：文江博客话题详情

如何在AWS胶水中连接和查询MySQL DB

发布于 2025-01-26 15:31:44 字数 106 浏览 1 评论 0原文

我正在使用sqlalchemy创建连接和查询mysql db，但是，胶水似乎不支持“ sqlalchemy”甚至“ pymysql”。有没有办法在胶水python shell Jobs上执行此操作？

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

醉生梦死 2025-02-02 15:31:44

我认为您需要安装sqlalchemy和pymysql。如果您使用的是Spark Runtime，则胶水使安装其他PY Libs非常容易，但是PY Shell运行时似乎有些不同。

我可以使用它的唯一方法是下载（或创建）WHL文件。 Luckily, you can download sqlalchemy 和 pymysql 。注意：如果您需要特定版本，则SQLachemy WHL文件有很多选择。

将这两个WHL文件放入S3存储桶中，您的胶合作业将可以访问。然后在工作示例中将两个路径（由逗号分隔）到python库路径

。

s3://my-bucket/SQLAlchemy-1.4.36-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl,s3://my-bucket/PyMySQL-1.0.2-py3-none-any .WHL

然后您应该能够像这样导入它们

import sqlalchemy
import pymysql


print('sqlalchemy', sqlalchemy.__version__)
print('pymysql', pymysql.__version__)

May 7, 2022, 9:38:03 AM Pending execution
Processing ./glue-python-libs-ox4yhv_1/SQLAlchemy-1.4.36-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Collecting greenlet!=0.4.17; python_version >= "3" and (platform_machine == "aarch64" or (platform_machine == "ppc64le" or (platform_machine == "x86_64" or (platform_machine == "amd64" or (platform_machine == "AMD64" or (platform_machine == "win32" or platform_machine == "WIN32"))))))
Downloading greenlet-1.1.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (147 kB)
Collecting importlib-metadata; python_version < "3.8"
Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB)
Collecting zipp>=0.5
Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Installing collected packages: greenlet, zipp, typing-extensions, importlib-metadata, SQLAlchemy
Successfully installed SQLAlchemy-1.4.36 greenlet-1.1.2 importlib-metadata-4.8.3 typing-extensions-4.1.1 zipp-3.6.0
Processing ./glue-python-libs-ox4yhv_1/PyMySQL-1.0.2-py3-none-any.whl
Installing collected packages: PyMySQL
Successfully installed PyMySQL-1.0.2
sqlalchemy 1.4.36 pymysql 1.0.2

I think you'll need to install sqlalchemy and pymysql. Glue makes it fairly easy to install additional py libs if you're using the Spark runtime, but the py shell runtime seems to be a little different.

The only way I've gotten it to work is to download(or create) whl files. Luckily, you can download sqlalchemy and pymysql from pypi. Note: there are many options for the sqlachemy whl file if you need a specific version.

Get those two whl files in a s3 bucket that your glue job will have access to. Then add both paths (separated by a comma) to the Python library path in your Job

example.

s3://my-bucket/SQLAlchemy-1.4.36-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl,s3://my-bucket/PyMySQL-1.0.2-py3-none-any.whl

Then you should be able to import them like so

import sqlalchemy
import pymysql


print('sqlalchemy', sqlalchemy.__version__)
print('pymysql', pymysql.__version__)

May 7, 2022, 9:38:03 AM Pending execution
Processing ./glue-python-libs-ox4yhv_1/SQLAlchemy-1.4.36-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Collecting greenlet!=0.4.17; python_version >= "3" and (platform_machine == "aarch64" or (platform_machine == "ppc64le" or (platform_machine == "x86_64" or (platform_machine == "amd64" or (platform_machine == "AMD64" or (platform_machine == "win32" or platform_machine == "WIN32"))))))
Downloading greenlet-1.1.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (147 kB)
Collecting importlib-metadata; python_version < "3.8"
Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB)
Collecting zipp>=0.5
Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB)
Collecting typing-extensions>=3.6.4; python_version < "3.8"
Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Installing collected packages: greenlet, zipp, typing-extensions, importlib-metadata, SQLAlchemy
Successfully installed SQLAlchemy-1.4.36 greenlet-1.1.2 importlib-metadata-4.8.3 typing-extensions-4.1.1 zipp-3.6.0
Processing ./glue-python-libs-ox4yhv_1/PyMySQL-1.0.2-py3-none-any.whl
Installing collected packages: PyMySQL
Successfully installed PyMySQL-1.0.2
sqlalchemy 1.4.36 pymysql 1.0.2

回复收藏 0 原文

~没有更多了~